In 1951, Stanley Payne wrote the landmark survey design book, “The Art of Asking Questions,” which advocated for brevity, simplicity, and relevance when designing survey questions. It’s always good to keep these and other principles in mind when planning your research. It’s also important to understand that asking about opinions can overstate the degree to which people think about the subject of the interview. This is particularly important when addressing issues that are not “top of mind,” like animal and environmental protection.

Payne offered the following advice to researchers: “Keep questions short and simply worded, personally relevant to issues close to home, and specific and concrete.” One of his primary concerns was that survey respondents are often willing to provide strong opinions on issues about which they have little knowledge and to which they have given little thought. He wrote that “the penchant of many respondents for answering questions which have no meaning for them poses a major problem for public opinion researchers.” People may have well-formed opinion about present-day issues like the presidential election or the Iraq war, but do they really have solid beliefs when it comes to animal testing, for instance, or maybe aquaculture?

This is important for animal and environmental advocates for a variety of reasons. On the positive side, relatively weak opinions about specific animal/environmental issues may suggest that some people are persuadable when it comes to those issues. If the right arguments and incentives are provided, those with unformed opinions may become more supportive and even take more actions on behalf of animals. On the other hand, it can be difficult to understand where public sentiment really lies, because opinion surveys may not reflect the depth of knowledge or degree of enthusiasm about a specific issue.

Survey question design is further complicated by the use of certain terms, which may or may not trigger latent opinions among the respondents. To help simplify questions, it’s important to keep them short and to avoid combining multiple issues into a single question. It’s also paramount to avoid ambiguity and also important to understand the limitations of questions that may be asked in an ambiguous fashion. As George Bishop wrote in “The Illusion of Public Opinion” (2005):

As Stanley Payne complained, abstract concept words… are loaded with ambiguities. As a result, respondents may give different answers to questions using such terms, not because of genuine differences of opinion on the subject, but rather because they interpret the question differently. The typically sizable demographic differences between whites and African Americans, for example, in their approval rating of President George W. Bush may reflect in large part a fundamental difference in how they interpret the meaning of “handling his job as president.” So it goes too for the “gender gap,” the “marriage gap,” the “generation gap,” and other sociodemographic gaps in public opinion.

There are a number of tools available to survey designers wishing to improve their questions. One of the most important is cognitive interviewing, which in its simplest form involves simply asking “why” someone thinks a certain way. For those who want to deeply understand the meaning of their questions to respondents, using more sophisticated methods may be warranted. In the book “Cognitive Interviewing,” Gordon Willis provides a Question Appraisal System (QAS) to help researchers identify some of the common pitfalls of survey questions. The QAS is presented in the form of a checklist that all researchers should consider before finalizing surveys.

STEP 1: READING:

Determine if it is difficult for the interviewers to read the question uniformly to all respondents.

STEP 2: INSTRUCTIONS:

Look for problems with any introductions, instructions, or explanations from the respondent’s point of view.

STEP 3: CLARITY:

Identify problems related to communicating the intent or meaning of the question to the respondent.

STEP 4: ASSUMPTIONS: Determine if there are problems with assumptions made or the underlying logic.

STEP 5: KNOWLEDGE/MEMORY:

Check whether respondents are likely to not know or have trouble remembering information.

STEP 6: SENSITIVITY/BIAS:

Assess questions for sensitive nature or wording, and for bias.

STEP 7: RESPONSE CATEGORIES:

Assess the adequacy of the range of responses to be recorded.

STEP 8: OTHER:

Look for problems not identified in Steps 1 – 7.

If you’re conducting a survey on animal or environmental protection issues, you would do well to keep these principles in mind. But try not to make your question evaluation overly complicated. In many cases, simply running a draft version of the survey by a few members of your target audience is a sufficient test. In other situations – where the results are critically important to your organization, for instance – it might make sense to bring in some experts to help validate your approach and refine your survey questions.