|
This section describes
the process used to collect, organize and analyze the information presented in Voices
from the Community: The visionPDX Input Report.
It also provides an overview of the general methodological
approach employed by visionPDX, along with some discussion of the
strengths and limitations of this approach. It is divided
into the following sections:
Our
Process
Methodological
Considerations
visionPDX
Telephone Survey
How to
Access the Raw Data
Our Process
From the beginning, one of the defining features
of visionPDX was a commitment to community ownership
of and participation in the visioning process. Soon
after launching visionPDX, Mayor Tom Potter invited
community members to apply to join a Vision Committee
that would direct and oversee the project. Over 120
people from across the city responded, and 57 individuals
representing a wide variety of backgrounds were ultimately
selected to serve on the committee.
This committee was then divided into six
subcommittees (Analysis, Communication, Engagement, Grants, Research
and Speakers Bureau), each of which provided oversight and direction
for major aspects of the project. The Analysis Subcommittee was charged
with developing an overall project methodology, creating the questions
to be used in the outreach phase, and monitoring the data collection
and analysis process.
Modeled after community visioning projects
across the globe, the method that was chosen was a qualitative survey
methodology that employed four open-ended questions to solicit
detailed, narrative responses from community members. Unlike
closed-ended questionnaires, which require respondents to check
pre-selected options, the open-ended survey format allowed respondents
to write about whatever topics were of most concern to them, in as few
or as many words as they desired. This design succeeded in producing
rich and nuanced insight into community members’ thinking, as well as a
wide array of concerns, strategies and ideas for change—some
previously anticipated, but many unanticipated as well.
Because each answer had to be read and
analyzed by individuals (as opposed to merely tallied by a computer),
the use of this survey method created a challenging data analysis
process. To analyze the over 13,000 completed surveys and
notes from more than 75 small group discussions collected through
visionPDX, the Analysis Subcommittee created a three-part data analysis
process involving Portland State University, a team of 40 community
volunteers and a smaller team of project staff.
As soon as survey responses were collected
from the community, the City submitted these responses in electronic
format to the Portland State University Survey Research Lab, which was
contracted to perform initial data processing. The Survey Research Lab
subsequently imported these responses into a qualitative analysis
software program called N6. Once in N6, PSU student volunteers were
able to assign codes to each response reflecting the major themes
addressed by respondents. For example, a response such as this: “I love
the park system, the transit system and the close proximity to the
beach, mountains, Seattle,” would have received the following codes:
(6 8) Environment/Natural Environment
(7 2) Transportation/Public Transportation
(10 1) Urban Livability/Parks and Open Spaces
(10 7) Urban Livability/Access
(14 1) Question Type/Value Q1 (Q1 denotes that it was the first
question asked on the questionnaire)
Once all responses were tagged with codes
identifying the topics addressed as well as the survey question being
answered, they were sorted into major categories and related
sub-categories. This analysis process yielded nine major categories
(Economy, Education, Environment, Government, Health, Public Safety,
Social Issues, Transportation and Urban Livability), and sixty-seven
subcategories, each of which contained anywhere from a few hundred to
over a thousand responses, depending on the popularity of the subject.
These subject-specific reports were then
printed, compiled in binders and submitted to the Data Analysis Work
Group (DAWG), a volunteer committee charged with completing the second
major analysis phase. Over a two-month period, 40 DAWG members read
through each of the printed binders, sifting through thousands of pages
of quotes to find common values, over-arching ideas, areas of agreement
and disagreement as well as specific strategies and ideas for change.
Working in teams of two or three, these volunteers prepared and submitted preliminary analysis
reports, which directly
informed Voices from the
Community: The visionPDX Input Report.
In the final phase of analysis, a team of
four project staff analyzed the reports prepared by the DAWG
members and integrated the findings from these notes with additional
analysis of the raw data. The findings and conclusions of this
multi-phase analysis process are presented in full in Voices from the
Community: The visionPDX Input Report.
Methodological
Considerations
The qualitative analysis process used to
interpret the survey results has strengths as well as some limitations,
which merit a brief discussion. Standard quantitative survey
methodologies are very useful for obtaining specific statements that
can be generalized to a larger population. For instance, a quantitative
survey could allow one to state with near certainty that “sixty percent
of respondents want a new MAX line in SE Portland,” and, if the sample
is representative of the larger population, one might even be able to
conclude that roughly sixty percent of all Portlanders favor this idea, within the margin of error dictated by the sample size.
However, a quantitative survey may restrict
the number of topics on which respondents can comment, or exclude
certain perspectives from consideration. This type of survey also tends
to produce less detailed answers, as respondents are often restricted
in the amount or type of information they can provide. A qualitative
methodology, such as the one chosen by visionPDX, allows respondents to
address anything and everything that concerns them, producing
significantly more detail and offering richer insights into
respondents’ thinking than a quantitative survey.
This qualitative process does not yield the
same type of hard, numerical conclusions that can often be obtained
through a quantitative process. While the software program employed by
visionPDX can group responses by general topic area, it cannot produce
reports containing the exact number of people that favor specific ideas
or strategies. This is because different people use different words and
phrases to speak about similar concepts. Also, some people touch on
concepts directly, while others use indirect language (including
metaphor, allusion, sarcasm and other figures of speech) to make their
point. Any attempt to tabulate responses using a key-word search of the raw data or
similar function could be expected to yield grossly inaccurate results.
Only through a careful reading of the data can accurate conclusions be
drawn.
Therefore, what can be found in Voices from the
Community: The visionPDX Input Report
are not charts, graphs or percentages but rather narrative
descriptions of recurring ideas, broad areas of agreement and
disagreement, specific ideas and strategies that caught readers’ eyes
and samples of quotes illustrating frequently-voiced opinions on
issues. Words such as “few,” “some” or “many” are used to provide a
general sense of the number of survey respondents that hold specific
views or mention specific issues. Because
some chapters contain more comments than others, the relative number of
people described by these terms varies from chapter to chapter.
It is also important to note that the
findings in this report should not be generalized beyond the survey
participants to represent the views of the larger Portland population.
This is because visionPDX survey participants were not selected through
random sampling but were rather invited to participate through an
intentionally open community engagement process (read about this
process here). While visionPDX respondents generally mirrored the
demographics of the larger population, some perspectives are sure to be over-represented
as others are to be under-represented.
visionPDX
Telephone Survey
To compare the results of
visionPDX’s qualitative process against a more traditional quantitative
process, the City contracted the survey research firm of Davis,
Hibbits & Midghall to conduct an independent telephone survey in
September 2007. This 15-minute survey was administered over the phone
to 500 Portlanders using random digit-dialing techniques. The results
were analyzed using quantitative methods and have been compiled into a
brief report which can be accessed in PDF format here. This random,
quantitative survey confirms many of the key findings presented in Voices from the
Community: The visionPDX Input Report,
attesting to the quality and accuracy of the City’s community-led,
qualitative process. Note: Any sampling of opinion or attitudes is subject to a margin of error,
which represents the difference between a sample of a given population
and the total population (here, Portland residents age 18+). For a
sample size of 500, the margin-of-error would be +/- 4.4% at the 95%
confidence level.
How
to Access the Raw Data
Hard copies of the data reports printed by
PSU are archived in binders at the Bureau of Planning Offices, located
at 1900 SW 4th Ave., Suite 7100, Portland OR 97201. These reports are
organized by topic area and contain all quotes submitted to the City by
visionPDX participants. They provide a wealth of information and offer
a fascinating glimpse into the thinking of thousands of everyday
Portlanders. If you are interested in viewing these reports, please
contact Stephanie D. Stephens, Vision into Action Program Manager, at
503-823-9588.
 | 
|
|
|
|
|
|
| |
|
