Questionnaire Design Tips: Some “Dos & Don’ts”
[For more research tips and guides for incorporating research into your advocacy, please be sure to check out our Research Advice section.]
Writing a good questionnaire, one that captures all you need to know and is “user-friendly” for your survey respondents, is both an art and a science. Questionnaire design has filled many text books and certainly can’t be covered in a few paragraphs. However, the tips below can help you get started, and the Faunalytics survey guidelines are available if you want to dive deeper.
If you’re writing survey questions, here are some helpful hints.
DO set your goals before you start
This applies to your research as much as to your communications efforts. To measure the success of either, you first have to clearly define what success looks like. What do you need to accomplish?
What do you need to learn from this survey? What information must your survey provide?
For animal advocates, this means first identifying your target audience and outlining what you want to persuade them to do (e.g., change attitudes and/or behavior). Then determine what you need to know about them to get their attention, engage them, and influence them. This is where a well-written survey can be especially useful.
DO prioritize questions and limit the length of your questionnaire
A lengthy or complicated survey leads to “respondent fatigue.” By the time they get to the later questions in the survey, people are tired and start answering questions as quickly as possible just to be done. Or they may just end the survey and their information will be lost to you. Either of these outcomes can damage the quality of your results. Glib answers often won’t be accurate, and drop-outs may cause a sampling bias. The answer is to keep surveys as short as possible.
Ask yourself how the research results will be used. Cut questions for which you don’t have an answer to, “What will we do with the results?” Don’t give in to the temptation to ask your respondents everything. If you are clear on your information goals, you will know what to cut. And if you really must have a longer questionnaire, put your most important questions near the beginning where respondents are most engaged.
DO pilot test your questionnaire
Testing your questionnaire before you field it will save you the frustration of getting off-target answers. Ideally, your “pilot” test will be done with a smaller sample of the respondents you plan to send your survey to. If you don’t have the time and resources to do that, it’s wise to at least test the questionnaire with a few people. Try to get some people who aren’t involved in animal issues and avoid asking people who are involved with the survey.
The pilot test will help reveal any wording in your questions that is confusing or ambiguous. Even experienced questionnaire designers can find that some respondents interpret a question differently than was intended. They may answer a different question than you thought you asked! You need to make this discovery before your survey is sent to hundreds of people.
DON’T write “double-barreled” questions
A common survey design error is to write a question that is really two questions in one. It’s impossible to get accurate answers from these questions because respondents only have the answer options to respond to one question, not both. Here is an example of a “double-barreled” question…
“How much has education about the treatment of animals on factory farms affected the meat and dairy industries?”
Some will answer this question regarding the meat industry, some regarding the dairy industry, and some will estimate the impact on both. You won’t know who answered in each way, which limits your analysis of the results. The “fix” is to ask two separate questions, one about the meat industry and one about the dairy industry.
DON’T write leading questions
“Leading” questions contain wording that may sway, or lead, respondents to one side of an argument. Leading questions can be harder to catch than you may think. Here is an example of a “leading” question…
“Do you think people should adopt an animal-friendly vegan diet?”
By using the phrase “animal-friendly,” you are leading respondents to equate a vegan diet with caring about animals. Even just using the word “should” is potentially leading. A better approach would be to ask respondents if they think people should adopt a vegan diet (without the “animal-friendly”). A follow-up question could ask those who say “yes” if their reason was due to animals, environment, health, etc.
DO write questions with enough answer options
Another error in question design springs from the difficulty of thinking through all the ways in which a question might be answered by someone very different from the question writer. This can force a response that isn’t accurate. Here is an example:
“On which day(s) of the week do you eat only vegetarian meals?”
This question is fine if you have already asked if they eat only vegetarian meals at least one day a week and they answered “yes,” and you ask this question only of those who said yes. Otherwise, you will force an answer that may not be true of their eating habits.
The same problem can arise if answer choices are overly simplified, such as asking for a “Yes/No” answer, or using absolutes like “always” and “ever,” when the actual answer is more complex. Here is an example…
“In the past 12 months, did you ever eat meat?” Yes/No
This question treats an avid meat eater the same as someone who eats meat once a year because turning down turkey at Thanksgiving would hurt the family’s feelings and/or lead to conflict. That nuance is lost when the respondent checks “yes.”
A better question will ask about the frequency of eating meat, whether over the past 12 months or – to make this easier for the respondent – over the past month or week.
It’s usually appropriate to offer “don’t know” responses and the option to select “neither” when asking people agree-disagree questions. While some respondents use these as easy ways to bypass answering a question, having these options can be important for accuracy. Animal issues are not “top of mind” for most people, so they may genuinely not have an opinion.
DON’T use jargon
To be clear, avoid jargon and acronyms. You may find that terms you think are common knowledge are not, when you test your questionnaire with someone outside of your organization. If you must use an acronym or a potentially unfamiliar term, provide a definition or example. For instance:
“Please indicate how much you agree or disagree with the statement, ‘Raising animals for food in CAFOs leads to animal abuse.”
Many people will not know that a CAFO is a “Concentrated Animal Feeding Operation.” You would need to provide the meaning of the abbreviation and probably a brief description as well. Or you may just want to use the more common term, “factory farm.” And even “factory farm” may need explaining, if you are fielding your survey to the general public.
The more you know about question design, the better your survey results will be! Be sure to start with clear goals and research priorities. Keep surveys as short and relevant to your audience as possible, and avoid leading questions and insider language. Make sure your questions stay “single barreled” and give people a full range of answer options if you want accurate information.
Do-it-yourself research is not only possible, but encouraged. As we all strive to evaluate our programs and improve our understanding of our impact, surveys are essential tools. It’s worth taking the time to learn the techniques to use – and the pitfalls to avoid – when writing questions.
For more reading, check out the research advice and survey guidelines on this website, or ask Faunalytics for assistance. You can also check out our older blogs on this topic including survey design basics, understanding survey bias, and understanding survey results.