Lessons Learned From Our Study With Animal Equality

In our library summary covering our study of Animal Equality’s outreach videos, we covered the basic study design and results. We also alluded to having encountered a number of challenges during its planning and execution. This is probably unavoidable for any undertaking that involves 35 campuses on opposite sides of a continent, thousands of participants, three non-profit organizations, dozens of staff and volunteers… and so on. This is especially true because this kind of large-scale field research hasn’t often been done in the animal advocacy movement before.

We commend Animal Equality for the time, effort, and resources they put into this research. Their commitment to evidence-based outreach, even in the face of challenges, is laudable. We hope others will follow their example! For that reason, we want to provide as much information as possible to other researchers and advocates. We think the Animal Equality study can serve as a blueprint for future studies, particularly those conducted on university campuses.

The Animal Equality study took close to two years to plan, pilot test, run, and analyze. We encourage anyone interested in more detail about the process to have a look at the comprehensive research design document, which was largely written and managed by Faunalytics’ former research director, Kathryn Asher. It covers the period from initial planning up to just prior to data collection for the main study.

In this post, we cover some of the key lessons learned from this study.

Field Research is Messy.

This umbrella “lesson” won’t come as a surprise to anyone who has ever conducted field research, but it is a long way from the controlled environment of a lab or a web survey. This lesson is about setting expectations.

In all types of research, there are trade-offs between internal and external validity: On the one hand, artificially controlling the situation enough that cause and effect can be determined, but on the other hand, keeping it “real” enough that we actually learn something about how things work outside the context of the study. Lab studies err toward the former. Field studies err toward the latter.

Although we ran this study in the field to keep things relatively consistent with Animal Equality’s usual outreach, we also wanted to be confident about cause and effect. That meant keeping the three conditions (360-degree VR video, 2D video, no video) as similar as possible in every way other than the video itself.

To do that, first, we asked the outreach volunteers to standardize their interactions with participants as much as possible. Second, we put the Animal Equality booth under a canopy to hide it from passersby—otherwise the VR technology, which is still a novelty for most people, could bring in a different type of participant to that one condition. While necessary, both of these attempts at standardization presented some problems.

We wrote a research training guide for the outreach volunteers, which is available at the end of the design doc. They were admirably diligent in following it, but practical difficulties inevitably arose. Scripts that sound fine in theory (like asking all participants “How did it go?”) can be awkward in practice when applied to people who may be stunned by what they’ve just watched. This may be especially true for volunteers who are used to trying to engage their audience in meaningful discussion.

Over the pilot testing period, we found a middle ground for the volunteers. We standardized the initial interaction with participants, but allowed for participant-led divergences from there. This reduces our ability to interpret causation (some of the condition effect may be due to conversational differences rather than the video itself) but in a reasonable way. The conversation is part of the experience for anyone who watches an Animal Equality video.

Putting the booth under a canopy also presented problems. First, if the canopy had been plain (unprinted), prospective participants would not have seen any of the signage Animal Equality typically uses—which isn’t externally valid. So instead, it was printed with images of farmed animals, inside and out.

However, the images inside the canopy were “not so happy,” which had the unfortunate effect of putting off some of the participants. Unlike signs on a table, these graphic images came as a surprise to participants who could not see them before approaching. Apart from upsetting some participants, this may have limited our ability to find a difference in pork consumption between the conditions: Even in the control condition, participants got some exposure to negative factory farming imagery.

Having a canopy over the study also seemed to reduce our ability to recruit participants. Seeing others participating likely makes the booth seem more approachable and interesting. That’s not surprising, given what we know of social norms and modelling behavior. In the end, we got fewer participants than we were hoping for, and the canopy may be why.

All of these represented substantial challenges that limited our ability to find a meaningful effect of the videos. I think if we were doing it again, we might make some of these decisions differently. Even still, I’m glad we had as much experimental control as we did.

Field research is messy and imperfect. Expect it to be. You’re not going to walk away with a perfect study with textbook methodology, but that’s okay. (If it’s not okay to you, seriously consider running a lab or online experiment instead.)

Measuring Diet (and Diet Change) is Hard.

This lesson is more specific than the other three, but in this movement, diet is something we care deeply about. Many of our interventions are intended to reduce the amount of animal products individuals consume. Unfortunately, this is something very difficult to measure accurately. We knew this going in to the study, and then we learned it again several times over.

In most animal advocacy research, because we don’t have the funding to individually compensate participants, minimizing their time commitment is of the utmost importance. This means that gold-standard measures like the 24-hour dietary recall are completely out of reach. Participants in the Animal Equality study needed to be able to complete our diet measure in less than a minute for us to have a hope of getting a reasonable follow-up response rate (which we did, in the end!).

To minimize within-person error variance (responding differently at different times when nothing has changed), participants needed to be able to fill out our diet measure consistently. To maximize our ability to detect real changes, they also needed to be able to report their consumption accurately. And to minimize time commitment, it all needed to happen quickly. That’s a tall order.

We did our best to meet these conflicting demands in two ways. First, we pilot tested measures extensively, as described next. Second, we kept our research question as simple as possible, as described in the final “lesson” below.

Pilot Testing is Key.

It’s tempting, when looking at limited time or a limited budget, to think “why run a pilot test?” But pilot tests are an investment. If your outcome measures are well-validated and you have strong reason to believe your hypotheses will be supported, you may not need one. But if your measures or your manipulations might be at all iffy, it’s better to find that out with 100 people than 3,000.

For us, the sticking point was the diet measure. We ran three pilot tests—two of them to look at the diet measure. (The other was for testing the canopy, which we introduced after the first pilot showed that recruitment wasn’t even across conditions.) The first diet pilot showed us that asking participants to count servings of multiple animal products over a two-day period produced too much variance in their responses. Maybe the reason was ambiguity about the definition of a serving, maybe it was memory failure, maybe it was lazy responding to a long measure, or all three.

Regardless, we didn’t have enough confidence in the measure to proceed, and if we hadn’t run this pilot, we would have administered a measure to 3,000 participants that probably couldn’t have picked up significant condition differences. Thus, we ran an additional pilot to test five potential diet measures and chose the one with the least error variance. As noted in the next point, this turned out to be a simple measure of pork consumption, measuring “times consumed” over the past 30 days.

It would have been great if we didn’t have to run three pilots for one study. That consumed a lot of time and resources. However, there were a lot of open questions before we began, and the pilots let us address those before beginning the far more time- and resource-intensive main study. They were unquestionably worth it.

Keep it Simple.

The outreach video Animal Equality was testing in this study focuses on pig farming. Our attitude measures asked about pigs and pork. Our diet measure asked about pork. Would it be nice to know if the videos had any impact on other animal product consumption? Absolutely. But over multiple pilot tests, we learned a lesson about trying to do too much. The more you ask a participant to report, the more they have to remember and the less accurate they’ll probably be for each estimate. Our key effect (condition differences in pork consumption) was marginally significant, and I’m willing to bet it wouldn’t have been if we had tried to measure the kitchen sink.

In our opinion, this is the biggest take-away from this study, and something we’re proud to have done right. It is incredibly tempting to measure more things. And when you’re finished with the study, you’ll find there are things you wish you had measured (baseline diet including more than just pork!). But there will always be more research questions to answer. There will always be something that can be done better next time.

Measure the thing that will answer the question you need to answer: for us, do videos showing factory farmed pigs impact attitudes and behavior toward pigs? If you can answer “nice to know” questions at the same time, great! But if there’s any chance that asking those questions could reduce your ability to answer the key question even a little, be merciless in cutting them. Some answers and more questions are better than no answers and only questions. Field research is messy enough.

As noted above, we encourage anyone interested in learning more about the study to have a look at the library summary, the full dataset, and/or the comprehensive research design document. However, if you have additional questions about the method, please feel free to contact Faunalytics’ research director, Jo Anderson.