How to Lessen the Usefulness of an Important City Survey

In 2020, the Sustainable Madison Committee was charged with developing an update to the 2011 Sustainable Madison Plan. Part of this work involved developing a survey intended to identify environmental priorities of Madison residents. An initial set of questions in an upcoming survey asks respondents to rate the relative importance of a number of environmental issues. Survey respondents will then be able to “pick their own adventures” and answer additional questions that drill down on any set of topics a respondent chooses. The survey is planned for release THIS WEEK.  City staff are shooting to gather over 5,000 responses which could yield a virtual treasure trove of information useful for policy development, city planning, budgeting, alders seeking to learn what their constituents care about most, as well as for residents themselves. However, given omissions and missed opportunities in the survey, it won't be nearly as useful as it could be.

A couple of weeks ago, I learned about this survey and attended a Sustainable Madison Committee meeting where it was discussed. (Note: given the amount of work associated with being an alder and my responsibilities to the committees to which I was assigned, I do not often have the luxury of attending extra meetings. However, I felt this one was particularly important to attend and I spent considerable time reviewing the survey and providing quick turnaround feedback to City staff.) With over 30 years of experience as a survey researcher, I offered suggested changes to the survey that align with best practices in survey research. Many were accepted and integrated, some were not. I also asked to separate topics that were conflated together, avoid leading questions, and add topics that I know are of intense interest to Madison residents. A few were included, most were not. (The compromise to some of my suggestions from the City staff's perspective was to include an “Other, please describe” response option. However, in survey research, respondents often don’t answer an “other” category because it takes more time, and because some things may not come to mind if unprompted, even if it is something of great interest or concern in a person’s life. Ensuring that response options include known and obvious items is just good survey science. Additionally, to the extent that this survey is relied upon to justify policy and budgetary decisions [which it surely will be, given the amount of investment the City is pouring into it], it is critical to ensure that known and obvious response options are included.)

As a public health researcher and alder with a passionate interest in the environment, conservation, and fighting climate change and equity related to all of these things, it is important for me to understand what residents of Madison are most concerned about with respect to environmental issues and how these concerns vary across socio-demographic groups (e.g., respondent answers to questions about race and ethnicity, age range, whether household with children < 18, sexual orientation, gender identity, student status, disability status, renter vs. homeowner, and a basic  income level question with a low, mid and high range) and across different areas of the City. Asking about a respondent’s zip code and a short list of characteristics can help inform an understanding of where inequities exist in terms of residents’ environmental concerns and needs. We can’t learn this critical insight about environmental justice (which the City includes as an area to prioritize) if we don’t ask about respondent characteristics and geographical locations.

I also suggest asking respondents if they are public transit riders, the extent to which they depend on public transportation, whether they would like to be public transit riders if they are not currently, whether they regularly ride a bicycle, whether they use an e-bike, whether they drive an electric car, and whether they frequently walk to their daily destinations, in order to see how environmental priorities are associated with travel modes that have implications for City planning. Yet, these types of questions are not part of the survey as of today. 

It is not often that City resources, time and energy are poured into a survey of this scope and with the intention of gathering input from thousands of residents. It literally makes me sick to my stomach to think of an opportunity like this being leveraged for only one purpose (updating the Sustainable Madison Plan), when it could easily serve multiple purposes and inform an understanding of disparities across residents in terms of their concerns as well as help alders who campaigned on environmental issues know what our own constituents care most about.

I was told that the changes I outline below or suggest above cannot be made because the Sustainable Madison Committee has not discussed them in their prior work. Yet, again, staff will not agree to hold off on releasing the survey until that committee has a chance to review the suggestions. The September 25 committee meeting that occurred after my suggested changes were made was canceled. It is also possible to add items to this survey without detracting from the prior work of that committee, especially if additions or wording changes make sense. It may require a budget amendment to have additional translation and survey design work, but I think those costs and a brief delay in launching the survey are well worth it to get the rich information that this survey could provide. 

Despite my frustration with, I still urge, in fact IMPLORE, you to take the survey and I will heavily promote it. I just want to be transparent about the process behind the scenes that I think has created a much less useful survey than it could be. If you would like more detail on what changes I requested, please read below.

Below are some of the more important suggested changes that I asked for and staff decisions related to each. The first set pertains to an initial list of environmental issues that respondents will be asked to rank or prioritize:

• Access to parks, green space, and street trees (replace "street trees" with "mature tree canopies" or "shade trees" because this is really about reducing heat sink. Changed.
• Separate out public transportation from biking and walking. Add: "Access to public transportation". Changed.
• Instead of "Environmental justice", say "Equitable access to safe and healthy environments". The suggested alternative defines the phrase more concretely. Changed.
• Missing from the list is an item about noise pollution. Add "Reducing noise pollution." Not changed. 
• Add an item on “Improving and expanding recycling services”. Not changed.

The next suggestions relate to the “drill-down” sections of the survey where respondents  “choose their own adventures”:

Quality, Affordable and Sustainable Housing  (Add "sustainable to the heading) Changed.

• Add:  Increase the availability of affordable home ownership opportunities developed with green building practices (or "built sustainably", or "that are climate sensitive") Not changed.
• Change an item to make the question about all housing, not just affordable rental housing: "Create and expand City policies and programs to support energy efficiency, sustainability, and removal of environmental toxins in new and existing (delete: "affordable rental") housing." Not changed.

Resilient City Design and Infrastructure

• Change Goal Wording: Protect and increase the access to infill, neighborhood-scale sustainable farming and community gardening. (Not everyone will  know what is meant by “infill”. We should also want to gauge interest in these things citywide, not just in “infill” areas, where the City is pursing more density.) Not changed.
• Change Goal Wording: Facilitate compact, transit-oriented development that supports walking, biking, and public transportation use to reduce traffic and greenhouse gas emissions; enhances livability; and improves environmental quality and public health.  I suggested deleting the part in italics because we don't yet have the evidence of these outcomes in the local context: "reducing traffic and greenhouse gas emissions" remains to be seen, especially given the large-scale removal of bus stops throughout the city; "enhanced livability" is a subjective assessment that shouldn't be presumed and conflated with the main question; "environmental quality" and "public health" are about more than public transit, and changes to transit must be weighed relative to other influences on the environment that may be negatively affected by the transit choices we make. From a survey science standpoint, the italicized portion represents a leading question--that is, the question is designed to make it harder for people to answer no. If this question is to see whether people agree that more density is beneficial for the environment, leave it at reducing traffic and greenhouse gas emissions. When you go further expounding on what you presume to be benefits, it just looks like a PR question and not an opportunity to learn what people really think. Not changed.  

Renewable Energy and Decarbonization

• Add: Original item: "Drive an electric vehicle"; Add “Drive a hybrid vehicle”. It is important for the city to learn about preferences related to hybrid vehicles, too. I think that these should be two separate questions. Electric car interest has implications for city infrastructure development and should be asked about separately for that reason, but interest in hybrid cars should not be ignored or conflated with electric car interest. Staff changed to “Drive a hybrid or fully-electric car.” Partially changed.  
• Add: "Learn more about nature-based temperature control sources, such as geothermal infrastructure, for heating and cooling." Not changed.
• Add: Learn more about electricity powered heat pumps. Changed.

Sustainable Transportation

• Add a goal: “Reduce the need for transportation by building livable neighborhoods with access to everyday services and resources” which reflects one of the existing goals of the current Comprehensive Plan. Not changed.
• Add item about e-bikes. Changed.

Clean, Abundant Water

• Add Goal on “Expand infrastructure for absorbing and capturing stormwater contaminants that can affect the health of drinking water, lakes and other water bodies.” Not changed
       

 Zero Waste

• Change “Participate in the City’s compost drop-off program during the summer” to “Participate in the City’s food scrap drop-off program during the summer.”I think people are more familiar with it being characterized as a food scrap program rather than a compost program.  Changed.

 Healthy Ecosystems

• Add an item about noise abatement. Not changed.

Vibrant, Green Economy

• Add “Work with homeowners, businesses, and developers to incentivize building practices that are “carbon-neutral”. Not changed.