| This
section describes the process used to collect, organize
and analyze the information presented in Voices
from the Community: The visionPDX Input Report.
It also provides an overview of the general methodological
approach employed by visionPDX, along with some discussion
of the strengths and limitations of this approach. It
is divided into the following sections:
Our Process
Methodological Considerations
visionPDX Telephone Survey
How to Access the Raw Data
Our
Process
From the beginning, one of the defining features of
visionPDX was a commitment to community ownership of and
participation in the visioning process. Soon after launching
visionPDX, Mayor Tom Potter invited community members
to apply to join a Vision Committee that would direct
and oversee the project. Over 120 people from across the
city responded, and 57 individuals representing a wide
variety of backgrounds were ultimately selected to serve
on the committee.
This committee was then divided into six subcommittees
(Analysis, Communication, Engagement, Grants, Research
and Speakers Bureau), each of which provided oversight
and direction for major aspects of the project. The Analysis
Subcommittee was charged with developing an overall project
methodology, creating the questions to be used in the
outreach phase, and monitoring the data collection and
analysis process.
Modeled after community visioning projects across the
globe, the method that was chosen was a qualitative survey
methodology that employed four open-ended questions to
solicit detailed, narrative responses from community members.
Unlike closed-ended questionnaires, which require respondents
to check pre-selected options, the open-ended survey format
allowed respondents to write about whatever topics were
of most concern to them, in as few or as many words as
they desired. This design succeeded in producing rich
and nuanced insight into community members’ thinking,
as well as a wide array of concerns, strategies and ideas
for change—some previously anticipated, but many unanticipated
as well.
Because each answer had to be read and analyzed by individuals
(as opposed to merely tallied by a computer), the use
of this survey method created a challenging data analysis
process. To analyze the over 13,000 completed surveys
and notes from more than 75 small group discussions collected
through visionPDX, the Analysis Subcommittee created a
three-part data analysis process involving Portland State
University, a team of 40 community volunteers and a smaller
team of project staff.
As soon as survey responses were collected from the
community, the City submitted these responses in electronic
format to the Portland
State University Survey Research Lab, which was contracted
to perform initial data processing. The Survey Research
Lab subsequently imported these responses into a qualitative
analysis software program called N6. Once in N6, PSU student
volunteers were able to assign codes to each response
reflecting the major themes addressed by respondents.
For example, a response such as this: “I love the park
system, the transit system and the close proximity to
the beach, mountains, Seattle,” would have received the
following codes:
(6 8) Environment/Natural Environment
(7 2) Transportation/Public Transportation
(10 1) Urban Livability/Parks and Open Spaces
(10 7) Urban Livability/Access
(14 1) Question Type/Value Q1 (Q1 denotes that it was
the first question asked on the questionnaire)
Once all responses were tagged with codes identifying
the topics addressed as well as the survey question being
answered, they were sorted into major categories and related
sub-categories. This analysis process yielded nine major
categories (Economy, Education, Environment, Government,
Health, Public Safety, Social Issues, Transportation and
Urban Livability), and sixty-seven subcategories, each
of which contained anywhere from a few hundred to over
a thousand responses, depending on the popularity of the
subject.
These subject-specific reports were then printed, compiled
in binders and submitted to the Data Analysis Work Group
(DAWG), a volunteer committee charged with completing
the second major analysis phase. Over a two-month period,
40 DAWG members read through each of the printed binders,
sifting through thousands of pages of quotes to find common
values, over-arching ideas, areas of agreement and disagreement
as well as specific strategies and ideas for change. Working
in teams of two or three, these volunteers prepared and
submitted preliminary analysis reports, which directly
informed Voices from
the Community: The visionPDX Input Report.
In the final phase of analysis, a team of four project
staff analyzed the reports prepared by the DAWG members
and integrated the findings from these notes with additional
analysis of the raw data. The findings and conclusions
of this multi-phase analysis process are presented in
full in Voices from
the Community: The visionPDX Input Report.
Methodological
Considerations
The qualitative analysis process used to interpret the
survey results has strengths as well as some limitations,
which merit a brief discussion. Standard quantitative
survey methodologies are very useful for obtaining specific
statements that can be generalized to a larger population.
For instance, a quantitative survey could allow one to
state with near certainty that “sixty percent of respondents
want a new MAX line in SE Portland,” and, if the sample
is representative of the larger population, one might
even be able to conclude that roughly sixty percent of
all Portlanders favor this idea, within the margin of
error dictated by the sample size.
However, a quantitative survey may restrict the number
of topics on which respondents can comment, or exclude
certain perspectives from consideration. This type of
survey also tends to produce less detailed answers, as
respondents are often restricted in the amount or type
of information they can provide. A qualitative methodology,
such as the one chosen by visionPDX, allows respondents
to address anything and everything that concerns them,
producing significantly more detail and offering richer
insights into respondents’ thinking than a quantitative
survey.
This qualitative process does not yield the same type
of hard, numerical conclusions that can often be obtained
through a quantitative process. While the software
program employed by visionPDX can group responses by general
topic area, it cannot produce reports containing the exact
number of people that favor specific ideas or strategies.
This is because different people use different words and
phrases to speak about similar concepts. Also, some people
touch on concepts directly, while others use indirect
language (including metaphor, allusion, sarcasm and other
figures of speech) to make their point. Any attempt to
tabulate responses using a key-word search of the raw
data or similar function could be expected to yield grossly
inaccurate results. Only through a careful reading of
the data can accurate conclusions be drawn.
Therefore, what can be found in Voices
from the Community: The visionPDX Input Report
are not charts, graphs or percentages but rather narrative
descriptions of recurring ideas, broad areas of agreement
and disagreement, specific ideas and strategies that caught
readers’ eyes and samples of quotes illustrating frequently-voiced
opinions on issues. Words such as “few,” “some” or “many”
are used to provide a general sense of the number of survey
respondents that hold specific views or mention specific
issues. Because some chapters contain more comments than
others, the relative number of people described by these terms
varies from chapter to chapter.
It is also important to note that the findings in this
report should not be generalized beyond the survey participants
to represent the views of the larger Portland population.
This is because visionPDX survey participants were not
selected through random sampling but were rather invited
to participate through an intentionally open community
engagement process (read
about this process here). While visionPDX respondents
generally mirrored
the demographics of the larger population, some perspectives
are sure to be over-represented as others are to be under-represented.
visionPDX
Telephone Survey
To compare the results of visionPDX’s qualitative process
against a more traditional quantitative process, the City
contracted the survey research firm of Davis,
Hibbits & Midghall to conduct an independent
telephone survey in September 2007. This 15-minute survey
was administered over the phone to 500 Portlanders using
random digit-dialing techniques. The results were analyzed
using quantitative methods and have been compiled into
a brief report which can be accessed in
PDF format here.
This random, quantitative survey confirms many of the
key findings presented in Voices
from the Community: The visionPDX Input Report,
attesting to the quality and accuracy of the City’s community-led,
qualitative process.
Note:
Any sampling of opinion or attitudes is subject
to a margin of error, which represents the difference
between a sample of a given population and the total population
(here, Portland residents age 18+). For a sample
size of 500, the margin-of-error would be +/- 4.4% at
the 95% confidence level.
How
to Access the Raw Data
Hard copies of the data reports printed by PSU are archived
in binders at the Bureau of Planning Offices, located
at 1900 SW 4th Ave., Suite 7100, Portland OR 97201. These
reports are organized by topic area and contain all quotes
submitted to the City by visionPDX participants. They
provide a wealth of information and offer a fascinating
glimpse into the thinking of thousands of everyday Portlanders.
If you are interested in viewing these reports, please
contact Stephanie D. Stephens, Vision into Action Program
Manager, at 503-823-9588.
|
