Questions To Use In Survey Research And Experiments

If a simple program evaluation survey is all you’re looking for, you can find one possible template here.

There are many validated measures assessing different animal-related attitudes, beliefs, and more. If the options listed here don’t suit what you’re looking for, you have a few more options: You can look through Faunalytics’ past surveys on the Open Science Framework, have a look in the Faunalytics library for studies on a similar topic, or reach out to us for help using the contact form.

Animal Attitudes Scale (Herzog et al., 1991, 2015)

• Measures general attitudes toward animal protection
• 5- or 10-item versions (see Appendix of linked article for full scale)
• Example: “I sometimes get upset when I see wild animals in cages at zoos.”

Solidarity with Animals (Amiot & Bastian, 2017)

• Measures solidarity (belonginess, closeness, attachment) with animals
• 5 agreement items (see Study 1 Method section of linked article for scale details)
• Example: “I feel close to other animals.”

Speciesism Scale (Caviola, Everett, & Faber, 2018)

• Measures speciesism, the prejudice that humans are superior to other animals
• 6 agreement items (see linked summary for full scale)
• Example: “It is morally acceptable to trade animals like possessions.”

Individual Attitude Items (Farmed Animals):

• “Animals used for food have approximately the same ability to feel pain and discomfort as humans” (used in Faunalytics’ study of attitudes in BRIC countries)
• “Eating meat directly contributes to the suffering of animals” (used in Faunalytics’ study of attitudes in BRIC countries)
• “Low meat prices are more important than the well-being of animals used for food” (used in Faunalytics’ study of attitudes in BRIC countries)
• “It is important that animals used for food are well cared for” (used in Faunalytics’ study of attitudes in BRIC countries)

Finding out what people eat is not easy. There are many possible approaches, each with their own pros and cons. We break down the main approaches below, and provide sample questions. 

Self-Identification: How Do People Think of Themselves?

The simplest approach to measuring long-term diet is to ask people how they self-identify, in a question like this one:

Which of the following best describes your diet?

• Omnivore/meat-eater (no restrictions on eating animal products)
• Reducetarian, flexitarian, or semi-vegetarian (reducing meat consumption or only eating it occasionally)
• Pescetarian (eats plant-based foods, eggs, dairy, and fish)
• Vegetarian (eats plant-based foods, eggs, and dairy)
• Vegan (eats only plant-based foods)
• Other (please specify) _____________

This question is best if you want to know how people think of themselves. We recommend providing the extra details in parentheses because many people don’t know the definitions of each diet.

However, if you want to know what people actually eat, don’t assume that this question will give you the full story. Studies have found that up to half of people who call themselves vegetarian also reported having eaten meat in the past two days (Juan et al., 2015). 

Self-Labeling: What Do People Call Themselves?

Inferring Diet Type from Reported Consumption: What Do People Say They Eat?

If you want your definitions of “vegan,” “vegetarian,” etc. to be based on what people actually eat rather than how they describe themselves, you need to look at what they actually eat. If you can measure consumption directly (see below), that’s the most reliable method, but if that’s not an option, you can ask them what they eat without referring to diets. For instance:

Which of the following types of food have you eaten in the past 3 months? Please select all that apply.

• • Chicken (fried chicken, in soup, grilled chicken, etc.)
  • Turkey (turkey dinner, turkey sandwich, in soup, etc.)
  • Pork (ham, pork chops, ribs, etc.)
  • Beef (steak, meatballs, in tacos, etc.)
  • Fish (salmon, tuna salad, fish and chips, etc.)
  • Other seafood (shrimp, crab, mussels, etc.)
  • Other meat (duck, lamb, venison, etc.)
  • Dairy products (cheese, milk, yogurt, etc.)
  • Eggs (omelet, in salad, in baked goods, etc.)
  • None of the above

People can be easily categorized as vegan, vegetarian, pescetarian, or omnivore based on their responses to this question, and if you want to disguise that goal, you can add other foods like fruit, vegetables, grains, coffee/tea, etc.

This question isn’t perfect. The three-month time frame may misidentify newer diet changes, and people may misremember or misreport their consumption (e.g., if they suspect you won’t categorize them as vegan if they admit they ate eggs once). However, it is easy to administer, answer, and analyze, and can be used to good effect if participants are made to feel comfortable answering honestly.

Actual Consumption: Direct Observation of Dietary Choices

Knowing how people self-identify can be useful, but in the end, knowing what they’re actually eating is often what we really want to know. Measuring participants’ behavior directly is always the best way to see what they actually eat.

You might try to partner directly with a restaurant, dining hall, or retail establishment for one-time purchase tracking after your intervention (e.g., Sparkman & Walton, 2017). Or if you’re more ambitious, for a longer data collection arrangement (e.g., Jalil, Tasoff, & Bustamante, 2019). Setting up the partnership may seem daunting, but we have found that restaurants are more amenable than you might think — and it doesn’t hurt to ask!

Using this kind of data as your outcome variable is easy–just remember that you need to link it to any other data you have about individual participants, like survey responses or a record of which condition of an experiment they were in. Depending on the context, it may make sense to use their student number (if it’s already tracked by a dining hall, for instance) or a random number that doesn’t link to any other identifying information about your participants outside the study itself. The latter means your study can be anonymous, which is ethically preferable if it’s feasible.

For more suggestions about using commercial marketing data, see the Humane League Labs (2018) report.

Self-Reported Consumption: Food Frequency Questionnaires (FFQs)

Food Frequency Questionnaires (FFQs) have been commonly used in advocacy research in the past, but they have important limitations. Like other self-reported consumption measures, FFQs are subject to misremembering and misreporting, and those problems get worse with longer and more complicated versions of the scale. If care is taken to maximize honest and thoughtful reporting, the biggest problem with FFQs is their high variability. This means that more participants are needed to find a significant effect. However, these difficulties are somewhat balanced by the fact that it’s usually easier to administer an FFQ than to observe participants’ diets directly, and therefore easier to get larger sample sizes.

A review of validation studies by Humane League Labs (2018) reported results showing that self-reported dietary measurements explained less than a quarter of the variance in actual protein and energy consumption. This was taken to suggest that people may not be accurately reporting the foods they are eating on an FFQ. However, when comparing across groups, the difference between actual and reported protein intake was only around 10% (Freedman et al., 2014).  Also, given that the biomarkers used to validate the FFQ only capture 24 hours and the FFQs covered a whole month, we would normally expect this type of variability regardless–most people’s diet changes from day to day, after all.  Put another way, even if a 30-day biomarker measurement existed, one would still expect test results from a 24-hour version to be somewhat different from the 30-day version.

In short, we don’t think there is evidence of strong systematic bias or a lack of reliability in the FFQ for how it is used in animal welfare research.  (As a final note, current biomarker tests are not specific. For example, pea protein looks the same to the test as beef protein. Given that our research is usually interested in the type of protein consumed rather than just the gross amount, their current utility seems low.)

For these reasons, we think that even though the FFQ has higher variability in measuring diet, its cost effectiveness and its ease of use for both participants and researchers make it a valid, useful tool.  Just make sure you collect a large sample in order to be able to detect an effect given the higher variability in the measure!

Here is an example of an FFQ:

You could use the above to see whether consumption of meat is significantly decreased by humane education. What you should not say is that “only X% of people eat meat after a humane ed session,” or make strong claims about how much consumption decreased (though of course you can report what you found). It’s a tricky distinction, so feel free to attend our office hour if you have questions.

FFQs are easy to use and modify (Cade et al., 2002). A few adjustments you might want to consider to suit your needs include:

• Including only the categories you need and/or combining similar categories (e.g., “Chicken and Turkey”, “Dairy and Eggs”) to make it easier to complete (which reduces variance),
• changing the time scale from 3 months to suit your study,
• changing the frequency options if you have reason to believe they won’t capture the frequencies very well for your population of interest,
• including additional products to disguise the purpose (Hebert et al., 1997), and
• choosing culturally-relevant examples if the FFQ is intended for use in another country or with a specific cultural group in the U.S. (Vergnaud, et al., 2010).

For an example of a successful and acceptable use of an FFQ, please see Faunalytics’ study of how video outreach affects pork consumption.

Best Practice

The gold standard of dietary measurement is tracking actual behavior over a long period of time. However, this is not possible for all researchers or research contexts. We need more validation studies to identify additional ways of reliably tracking diet. In the meantime, use one of the other methods described above if this kind of data is not available to you. A large sample size determined by power analysis can compensate for higher variance in a measure, and strong study design reduces bias. If you need help creating that design, Faunalytics is here to help! Check out our research team’s virtual office hour.

With demographic and personal data, it is particularly important to be cautious about what you collect and how the data will be stored. If you don’t need the information, don’t ask these questions as they are potentially sensitive. If you do ask them, give participants the option not to answer if you can.

Gender

What is your gender?

• Man
• Woman
• Non-binary
• I do not see myself represented in the above options. My gender is ____.

Age

What is your age? ___

State

Where do you live?

• [dropdown list of states]
• U.S. territory (e.g., Puerto Rico)
• I do not reside in the United States

Usage Note: You should not look at your data at the state level unless you have several thousand participants—there will be too few per state for the results to be reliable. Instead, re-categorize the state-level information into regions: Northeast, Midwest, South, and West. We recommend asking participants for their state rather than their region directly because some may be unsure of the regions.

Race/Ethnicity

What is your race/ethnicity?

• Hispanic or Latino/Latina
• White, non-Hispanic
• Black or African-American
• American Indian or Alaska Native
• Asian
• Native Hawaiian or Other Pacific Islander
• Two or more races
• Other

Usage note: This item is adapted from the U.S. census format to combine race and ethnicity into one question for simplicity.

Income

What is your annual household income before taxes?

• Less than $20,000
• $20,000 to $39,999
• $40,000 to $59,999
• $60,000 to $79,999
• $80,000 to $99,999
• $100,000 or more

Researchers are often concerned with the possibility of social desirability bias in their studies: people answering questions to make themselves look good rather than with full honesty. This is particularly concerning in studies run by advocacy organizations, where our goals and motives are generally quite clear to participants. E.g., if they know that the goal of an intervention (like a video) is to get them to eat less meat, that knowledge may conflict with the idea that the goal of research is to accurately measure their meat consumption.

The two best ways to avoid social desirability bias are: 

1. Measure diet directly, rather than using self-report (see “Measuring Diet” above). If your measure isn’t reliant on participants’ honesty, this bias can’t come into play. This method is particularly necessary if option #2 isn’t available to you because your study needs to be presented in a realistic advocacy context.
2. Avoid giving the respondent clues about which answers the surveying organization would prefer. For instance, if you don’t need the study to “look like” advocacy, don’t brand it with an advocacy organization’s logo. You could even partner with a university and use their branding to make it clearer that this is research. And write your questions as neutrally and objectively as possible — have a non-advocate read them to make sure. (See “Writing Your Own Questions” below.)

We used to recommend controlling for social desirability bias if the above options were not available. However, we have since updated away from that recommendation following a review of the literature about flaws in this method (see, e.g., McCrae & Costa, 1983; Piedmont et al., 2000; Uziel, 2010). Although there is still some debate about what socially desirability scales measure, practicing good study design as described in the two methods above is a much safer (and very reliable) way of avoiding this bias.

Acknowledgement: We are grateful to Joachim Stöber, PhD, for his invaluable contributions to this section.

Why use a validated scale instead of your own items?

If you’ve combed through the sections above and don’t see anything that will work for you, you may want to write your own questions. Here are a few suggestions:

1. Keep your question or statement short and simple. Many adults in the U.S. have low literacy. If a student in grade 7 or 8 would have trouble with your question, as many as half of your participants will also have trouble with it. This lowers data quality because they can’t answer properly.
2. Use common response options. This is good practice even if you’re writing your own question or statement, to ensure that you don’t inadvertently use options that are confusing to participants or produce results that are hard to interpret (examples below). Symmetrical scales are easier for participants and researchers, so use them whenever possible. With a symmetrical scale, the difference between “Negative” and “Somewhat Negative” is the same as the difference between “Positive” and “Somewhat Positive.” With an asymmetrical scale, the differences between the options is more subjective and hard to interpret.
   • Five-point, symmetrical scale with a midpoint: Strongly Disagree / Disagree / Neither Agree Nor Disagree / Agree / Strongly Agree
   • Six-point, symmetrical scale with no midpoint: Completely Dissatisfied / Dissatisfied / Somewhat Dissatisfied / Somewhat Satisfied / Satisfied / Completely Satisfied
   • Five-point, assymetrical scale: Not At All Likely / Somewhat Likely / Moderately Likely / Very Likely / Extremely Likely
   • You can replace the words used with your own, but keep the format (e.g., Accurate/Inaccurate, Positive/Negative, Important/Unimportant).
3. Avoid double-barreled questions (questions that ask about two things at once). For instance, imagine asking: “How satisfied were you with how knowledgeable and interesting the tour guide was?” Participants who thought the tour guide was boring but knowledgeable or interesting but inexperienced will have a hard time answering, and you’ll have a hard time interpreting the answers.
4. Use negatives sparingly. Small negation words like ‘not’ or ‘don’t’ are easily missed by participants (e.g., “I often spend time with people who don’t care about animal rights”). Even more importantly, avoid confusing participants with multiple negations in one sentence (e.g., “I never spend time with people who don’t care about animal rights”).
5. Consider the whole range of people who might participate. Think through how you’re going to recruit participants, and who will end up completely your study as a result. For instance, if your survey about your website pops up when someone visits the site, you will get some participants who are visiting for the first time. Do you need a “don’t know” or “not applicable” option on any of your questions? Can they skip some entirely?
6. Don’t use Yes/No for a subjective question. Unless you’re asking something extremely straightforward, you can probably get more information by providing a wider range of response options. For example, if you’re asking “Would you recommend this product to a friend?”, rather than just choosing Yes or No, you could give five choices: Definitely Yes, Probably Yes, Uncertain, Probably No, Definitely No. (Sometimes it’s nice to report a simple percentage of people who said yes, but you can just combine the percentages who said probably or definitely yes.)
7. Test your study. To make sure your questions are short and simple enough, conduct an informal pilot test by having 5 to 10 people complete your study as though they were participants. Ask them to tell you about their thought process and any problems they encountered. Watch for points of confusion, ambiguity, or difficulty finding a response option that fits.