Evaluating Sample Quality

Introduction

When conducting any research project, sample quality is critical. To get accurate results from your research, you must reach a group of respondents who resemble your target audience as closely as possible. That’s true whether your target is as broad as all adults in the United States or as narrow as vegans who have donated to animal welfare causes. Sample quality is critical to have confidence that the answers you get represent the entire group.

But sample quality is also a complex topic worthy of a college course, so the goal here is simply to give you an overview of questions to ask your vendor about the respondents that they will be providing. For greater understanding of the complexities of ensuring a representative sample you can do additional reading.

For starters, take a look at the ESOMAR guide for online sample quality (PDF), which is an excellent overview of the topic. You may also be interested in the approach to data quality (PDF) for GfK, a high-quality sample provider that Faunalytics uses for our annual Animal Tracker. Lastly, check out the Faunalytics research guidelines.

Questions to Ask about Sampling

Whether you are buying access to a list of potential research respondents (“sample only”) or paying for a full-service market research project, you need to understand several things. These include the source of the respondents, how they are recruited, how they are validated, and how the results will be weighted to adjust for hard-to-reach respondents.

Sources
Where will your respondents come from?

When most households in the United States had a landline phone it was possible to get an acceptably representative sample of the U.S. adult population by doing “random digit dialling” and a phone survey. Now that most of us have cell phones, the best approach is often an online survey using large panels of people who have agreed to answer surveys. For populations difficult to reach online, phone surveys can still be valuable if landline calls are augmented by calls to panelists with cell phones.

Ask your vendor if they have an actively managed research panel. An actively managed panel will have poor responders and non-responders weeded out regularly, and demographic information kept up to date.

If your vendor must supplement your sample to get enough respondents with the characteristics you requested, they may go to another research company. Ask them to tell you if this happens, and how they select a panel partner. How do they ensure the de-duplication of respondents? Some people may be on more than one panel and you don’t want them to answer your survey twice. Don’t accept respondents from panels used for both direct marketing and research because that will introduce a sample bias.

Will they augment the sample with web intercept recruitment (“river samples”)? A river sample may be recruited through online ads or “gated” content, where people are required to answer the survey question(s) before they access a website or article. Will the vendor use a survey router? If so, respondents may be sent to your survey for different reasons, sometimes after not qualifying for another survey. Ask how the router will prioritize which respondents are sent to your survey, because this can introduce a bias.

Recruitment
Ask how often people in the panels being used are contacted to take part in a survey every month. Too many survey invitations may increase your survey’s non-response bias because many people tire of taking surveys. Also find out how often a person can take a survey. Constant survey takers may become conditioned to answering in certain routine ways.

Most online panels will not match your target population. Therefore, it is typical to use demographic recruiting quotas (such as age and gender). Work with the vendor to identify what demographic variables your survey sample needs to be balanced against, and there may be attitudes and behavior that you want to match to the target population also.

What form of incentives will respondents receive for answering the survey? The type and size of the incentive influences why people answer a survey, which could affect how they answer it.

Ask to see the invitation to your survey. Writing a good invitation is an art. You don’t want to bias people by telling too much about the survey, while at the same time you want interest them in taking it. Find out the percentage of survey invitations sent that were accepted, and the percentage of surveys started that were completed. Those impatient people who deleted the invitation or started the survey but dropped out may represent a demographic you need to represent your target audience correctly.

How will you reach respondents who are hard to reach on the internet? Sometimes adding phone and mail contacts is necessary to reach a representative sample.

Validation
How does your vendor screen out false responders and those who try to speed through the survey to get rewarded for minimal effort? How do they confirm the identity of respondents? Reputable sample vendors will have a process for cleaning out false respondents from their panels and validation techniques to verify the approved panel member is the person taking the survey. They will also identify and remove respondents who raced through the survey without thinking about their answers.

Weighting
How will the final sample be matched to your target group? Will they “weight” the results to make up for shortfalls in hard-to-reach respondents? Discuss and agree upon which variables to match your sample against: age, education, or even dietary habits? Weighting gives mathematically greater “weight” to the answers from a smaller group when not enough of them responded to the survey relative to their size within the population of interest.

Be aware that weighting cannot always make up for under-represented groups if the sample from those groups is just too small. Ask for a description of the vendor’s weighting and projection methods and an estimate of how well the sample represents your target population.

Providers’ Sampling Capabilities

Not all vendors have the same sampling capabilities. The resources and sophistication that are applied to getting you the respondents you need will depend in part on the type of vendor you choose.

A full-service market research vendor will typically have the greatest resources and will often have both an online panel of potential survey respondents and the ability to purchase more online survey respondents to merge with them. Sophisticated vendors will be able to go beyond internet surveys and offer augmentation with phone surveys and mailings.

The use of multiple modes of recruitment (online, phone, mail) can be necessary when a nationally representative sample is needed, including more difficult-to-reach demographic groups like those in rural areas and those whose primary language is not English.

Some vendors use address-based sampling, improving their recruiting by using the U.S. Postal Service’s data files of delivery points, the Computerized Delivery Sequence File (CDSF). If you need to understand how the entire U.S. adult population thinks about a topic, you will likely need a full-service research vendor. (For thoughts on how to select the right research vendor for your project, please see “How to Choose a Research Vendor.”)

There are some full-service research providers who conduct research with internet-based respondents only, providing a good representation from large pre-screened internet panels, but with no capacity to represent those not on the internet. This is an important limitation if you need a truly representative survey, but with U.S. internet access approaching 90%, many nonprofits choose to accept the limitations of online research given the cost advantages.

Then there are providers who have a website they recruit from. This is a “convenience sample” that includes only people who visit that website and are willing to take a survey. If you are on a very tight budget and the people who use that website are the right audience for your questions, this can be an alternative. However, you will need to caveat the results with information about the source of the respondents. Ask for information on the demographics of this sample. Do they skew younger, male, better-educated, or higher income than the U.S. average?

Lastly, some companies provide respondents from a pool of people who are seeking to do surveys and other tasks on the internet for pay. Using people who are scouting for work on the internet as a sample source will introduce an unknown and unpredictable sample bias. Even for extremely tight budget projects, it may be better to buy sample lists or use your internal customer lists and field a do-it-yourself online survey through a service like SurveyMonkey. (For ideas about questionnaire design, please see “Questionnaire Do’s and Don’ts.”)

Conclusions

Ensuring sample quality is a complex, but vital process because the answers provided by your research are only accurate if the people you reached truly represent the target group you are trying to understand.

_____________

Dorothy J. Rich, M.B.A., a market research professional recently retired from a career spanning 35 years, human and animal rights activist, and parent to two rescue dogs