Survey Design 101
Whether you’re designing a modest member survey or a large multi-year donor tracking study, applying a few basic principles will help you get valid, actionable results.
Lately I’ve been working on designing the Animal Tracker, a public opinion survey that will be repeated over the coming decades to provide a long-term barometer of important attitudes and behavior (go here to learn more about the collaborative project). As I have been designing the initial draft of the survey, it struck me that some animal advocates could benefit from a better understanding of planning and designing questionnaires.
Survey design is often as much art as it is science. All spoken or written language carries nuances, unintended or otherwise, and almost all survey questions contain some bias. A good design will mitigate the bias as much as possible to help produce valid and meaningful results. Note that I am assuming an objective survey meant to produce accurate results, not a “push poll” or a single question posted on a website where anyone can vote.
What can be done during the survey design process to help ensure valid results? Lots of things, actually, but here are a few of the most important to keep in mind:
The essence of survey validity is measuring what you intend to measure. This starts by identifying your research questions for the project; why are you conducting the survey and what answers do you expect it to provide? Make sure the questions are “actionable” and meaningful to your program or organization. Differentiate what you “need” to know from what you would just “like” to know. If a question doesn’t relate directly to one of your research questions, cut it from the survey to avoid wasting respondents’ time (and yours).
Stability (or Reliability)
Each survey question should have a stable meaning over time, also known as “reliability.” If asked of a similar population under similar circumstances, responses to your survey questions should be relatively stable over time. This is essential for longitudinal and/or tracking surveys where data are compared over multiple time periods. Even for one-off surveys, however, advocates should avoid asking questions for which the meaning is tied to current events or may otherwise “expire.” There are also some simple reliability tests that you can conduct; see this link for an overview.
Consistency is another essential element of survey design. If your questions are critical to the success of your group or campaign, consider pre-testing the survey to understand how people are interpreting your draft questions. Adjustments can then be made to ensure that these interpretations align with what you’re trying to measure and are consistent for different respondents. Pre-testing doesn’t need to be comprehensive, but subjects should include people from all important segments in your sample to understand differences by respondent type.
Here are some more considerations if you’re putting together a survey:
- Use standard response scales. These scales (like Likert scales, which ask respondents how much they agree or disagree with a statement) are tried and true and will let you more easily compare your findings with other surveys. Before you write a question, check to see how others have asked about the same thing. See the Faunalytics library or an external site like Roper’s iPoll database.
- Simplify the survey. Chances are your respondents won’t know as much about the issue as you do, so be sure to keep both your questions and answers fairly simple. For instance, don’t assume that everyone will define “meat” the same way; instead, give everyone a clear and comprehensive definition. Limit yourself to just one concept per question and break up more complex or nuanced questions. For instance, you can begin by asking someone if they “agree or disagree,” then follow up to ask if they agree/disagree “strongly” or only “somewhat.”
- Pre-test as much as possible. As noted, pre-testing a survey instrument is essential when the investment in research is large or the questions are critical to your group. However, even for smaller surveys, consider at least limited pre-testing with colleagues, family members, etc. (try to get at least one degree away from animal advocates).
- Balance the survey. Most quantitative surveys use a mix of closed- and open-end questions, with emphasis on the former. Don’t ask entirely open-ended questions unless your sample has time and interest in responding in depth (and you have time for all of the coding and analysis). Instead, focus on closed-end questions and use open-ends strategically to get rich answers to key questions.
That’s survey design, in a nutshell. Of course, I haven’t even touched on other aspects of survey research planning, such as probability sampling, maximizing respondent cooperation, etc. I’ll plan to cover those topics in future blog posts.
Have questions? Feel free to comment or ask a question below.