Table of Contents:   

INTRODUCTION «   
ECONOMY «   
EDUCATION «   
ENVIRONMENT «   
GOVERNMENT «   
HEALTH «   
PUBLIC SAFETY «   
SOCIAL ISSUES «   
TRANSPORTATION «   
URBAN LIVABILITY «    

METHODOLOGY
Data Collection and Analysis



back  |  next

     

This section describes the process used to collect, organize and analyze the information presented in Voices from the Community: The visionPDX Input Report. It also provides an overview of the general methodological approach employed by visionPDX, along with some discussion of the strengths and limitations of this approach. It is divided into the following sections:

Our Process
Methodological Considerations
visionPDX Telephone Survey
How to Access the Raw Data

Our Process

From the beginning, one of the defining features of visionPDX was a commitment to community ownership of and participation in the visioning process. Soon after launching visionPDX, Mayor Tom Potter invited community members to apply to join a Vision Committee that would direct and oversee the project. Over 120 people from across the city responded, and 57 individuals representing a wide variety of backgrounds were ultimately selected to serve on the committee.

This committee was then divided into six subcommittees (Analysis, Communication, Engagement, Grants, Research and Speakers Bureau), each of which provided oversight and direction for major aspects of the project. The Analysis Subcommittee was charged with developing an overall project methodology, creating the questions to be used in the outreach phase, and monitoring the data collection and analysis process.

Modeled after community visioning projects across the globe, the method that was chosen was a qualitative survey methodology that employed four open-ended questions to solicit detailed, narrative responses from community members. Unlike closed-ended questionnaires, which require respondents to check pre-selected options, the open-ended survey format allowed respondents to write about whatever topics were of most concern to them, in as few or as many words as they desired. This design succeeded in producing rich and nuanced insight into community members’ thinking, as well as a wide array of concerns, strategies and ideas for change—some previously anticipated, but many unanticipated as well.

Because each answer had to be read and analyzed by individuals (as opposed to merely tallied by a computer), the use of this survey method created a challenging data analysis process. To analyze the over 13,000 completed surveys and notes from more than 75 small group discussions collected through visionPDX, the Analysis Subcommittee created a three-part data analysis process involving Portland State University, a team of 40 community volunteers and a smaller team of project staff.

As soon as survey responses were collected from the community, the City submitted these responses in electronic format to the Portland State University Survey Research Lab, which was contracted to perform initial data processing. The Survey Research Lab subsequently imported these responses into a qualitative analysis software program called N6. Once in N6, PSU student volunteers were able to assign codes to each response reflecting the major themes addressed by respondents. For example, a response such as this: “I love the park system, the transit system and the close proximity to the beach, mountains, Seattle,” would have received the following codes:

(6 8) Environment/Natural Environment
(7 2) Transportation/Public Transportation
(10 1) Urban Livability/Parks and Open Spaces
(10 7) Urban Livability/Access
(14 1) Question Type/Value Q1 (Q1 denotes that it was the first question asked on the questionnaire)

Once all responses were tagged with codes identifying the topics addressed as well as the survey question being answered, they were sorted into major categories and related sub-categories. This analysis process yielded nine major categories (Economy, Education, Environment, Government, Health, Public Safety, Social Issues, Transportation and Urban Livability), and sixty-seven subcategories, each of which contained anywhere from a few hundred to over a thousand responses, depending on the popularity of the subject.

These subject-specific reports were then printed, compiled in binders and submitted to the Data Analysis Work Group (DAWG), a volunteer committee charged with completing the second major analysis phase. Over a two-month period, 40 DAWG members read through each of the printed binders, sifting through thousands of pages of quotes to find common values, over-arching ideas, areas of agreement and disagreement as well as specific strategies and ideas for change. Working in teams of two or three, these volunteers prepared and submitted preliminary analysis reports, which directly informed Voices from the Community: The visionPDX Input Report.

In the final phase of analysis, a team of four project staff analyzed the reports prepared by the DAWG members and integrated the findings from these notes with additional analysis of the raw data. The findings and conclusions of this multi-phase analysis process are presented in full in Voices from the Community: The visionPDX Input Report.

Methodological Considerations

The qualitative analysis process used to interpret the survey results has strengths as well as some limitations, which merit a brief discussion. Standard quantitative survey methodologies are very useful for obtaining specific statements that can be generalized to a larger population. For instance, a quantitative survey could allow one to state with near certainty that “sixty percent of respondents want a new MAX line in SE Portland,” and, if the sample is representative of the larger population, one might even be able to conclude that roughly sixty percent of all Portlanders favor this idea, within the margin of error dictated by the sample size.

However, a quantitative survey may restrict the number of topics on which respondents can comment, or exclude certain perspectives from consideration. This type of survey also tends to produce less detailed answers, as respondents are often restricted in the amount or type of information they can provide. A qualitative methodology, such as the one chosen by visionPDX, allows respondents to address anything and everything that concerns them, producing significantly more detail and offering richer insights into respondents’ thinking than a quantitative survey.

This qualitative process does not yield the same type of hard, numerical conclusions that can often be obtained through a quantitative process. While the software program employed by visionPDX can group responses by general topic area, it cannot produce reports containing the exact number of people that favor specific ideas or strategies. This is because different people use different words and phrases to speak about similar concepts. Also, some people touch on concepts directly, while others use indirect language (including metaphor, allusion, sarcasm and other figures of speech) to make their point. Any attempt to tabulate responses using a key-word search of the raw data or similar function could be expected to yield grossly inaccurate results. Only through a careful reading of the data can accurate conclusions be drawn.

Therefore, what can be found in Voices from the Community: The visionPDX Input Report are not charts, graphs or percentages but rather narrative descriptions of recurring ideas, broad areas of agreement and disagreement, specific ideas and strategies that caught readers’ eyes and samples of quotes illustrating frequently-voiced opinions on issues. Words such as “few,” “some” or “many” are used to provide a general sense of the number of survey respondents that hold specific views or mention specific issues. Because some chapters contain more comments than others, the relative number of people described by these terms varies from chapter to chapter.

It is also important to note that the findings in this report should not be generalized beyond the survey participants to represent the views of the larger Portland population. This is because visionPDX survey participants were not selected through random sampling but were rather invited to participate through an intentionally open community engagement process (read about this process here). While visionPDX respondents generally mirrored the demographics of the larger population, some perspectives are sure to be over-represented as others are to be under-represented.

visionPDX Telephone Survey

To compare the results of visionPDX’s qualitative process against a more traditional quantitative process, the City contracted the survey research firm of Davis, Hibbits &  Midghall to conduct an independent telephone survey in September 2007. This 15-minute survey was administered over the phone to 500 Portlanders using random digit-dialing techniques. The results were analyzed using quantitative methods and have been compiled into a brief report which can be accessed in PDF format here

This random, quantitative survey confirms many of the key findings presented in Voices from the Community: The visionPDX Input Report, attesting to the quality and accuracy of the City’s community-led, qualitative process.

Note: Any sampling of opinion or attitudes is subject to a margin of error, which represents the difference between a sample of a given population and the total population (here, Portland residents age 18+).  For a sample size of 500, the margin-of-error would be +/- 4.4% at the 95% confidence level.

How to Access the Raw Data

Hard copies of the data reports printed by PSU are archived in binders at the Bureau of Planning Offices, located at 1900 SW 4th Ave., Suite 7100, Portland OR 97201. These reports are organized by topic area and contain all quotes submitted to the City by visionPDX participants. They provide a wealth of information and offer a fascinating glimpse into the thinking of thousands of everyday Portlanders. If you are interested in viewing these reports, please contact Stephanie D. Stephens, Vision into Action Program Manager, at 503-823-9588.

back  |  next


Economy | Education | Environment | Government | Health | Safety | Social Issues | Transportation | Urban Livability

 

   

Vision into Action / 1900 SW 4th, Suite 7100 / Portland, Oregon 97204 / Phone: (503) 823-9585