The first rule of study design is this: Always know how you’re going to analyze your results before committing to a design.

It’s easy to think “we’ll figure that out when the time comes” and then realize you’ve collected your data in a format that is difficult or impossible for you to analyze. Whether you plan to analyze your results yourself or have someone else do it, we strongly recommend that you outline an analysis plan before collecting data.

Analysis Styles

The analysis phase of a study is the phase after data collection, when the results are used to answer the original questions of the study. There are two main study purposes, and they call for different analysis styles.

Exploratory Studies & Exploratory Analysis

An exploratory study seeks to discover new information when you don’t have much idea of what that information might be.

For instance, if an organization has no idea what aspects of their humane education lectures are most impactful for attendees, they might conduct an exploratory study to get some ideas about how the lectures are perceived. In this case, many of the questions asked might be open-ended (letting participants write or say any response they want), and the data analysis would also be open-ended. An analysis procedure might not be specified ahead of time and typically a large number of possible findings would be considered.

The result is that any relationships which appeared to be statistically significant would still have to be considered somewhat tentative until confirmed with further information. Often, qualitative research methods are useful in this situation. Free response questions and focus group discussions are hard to tabulate, but allow responses which might not be possible to collect through multiple choice questionnaires.

Confirmatory Studies & Hypothesis Testing

For some research, you will have an idea of how you expect your study to turn out. That idea may come from a previous exploratory study, or it may be because you’re testing an intervention that is intended to have a specific effect. Either way, if you are interested in one particular outcome, you can begin with a hypothesis to test. For instance, are students who hear a humane education lecture more likely to go vegetarian than students who don’t?

In this case, it is important to specify ahead of time how you will analyze the data to test your hypothesis. In fact, best practice is to clearly state your hypothesis and analysis plan in a pre-registration document, which is made publicly available on a website like the Open Science Framework (OSF) before you begin collecting data for your study (see Faunalytics’ OSF profile for examples). The biggest advantage of pre-registering is that if your hypothesis is supported, no one will wonder if this is really the hypothesis you intended to test, or how you predicted it would come out.

For instance, in the humane education example above, it seems obvious that you would predict attending a humane education lecture would produce more vegetarians than not attending. But sometimes it isn’t as obvious. Imagine this question instead: Does a humane education lecture that talks about all farmed animals produces more vegetarians than one that focuses in-depth on pigs? If you have a strong hypothesis about which way you expect that to come out (and why), it’s best to pre-register it.

How To Analyze Your Data

This section title is a bit misleading, because training in data analysis is beyond the scope of this guide! Doing it properly takes years of study and hands-on experience. If you don’t have anyone in your organization with statistical training, we strongly suggest that you turn to outside sources for help.

Statistics Without Borders is likely your best bet. They are a volunteer Outreach Group of the American Statistical Association (ASA) that provides pro bono services in statistics and data science to not-for-profit organizations. Whether you just need help with analysis or would like substantive help with designing and running your study as well, they may be able to assist. Either way, we recommend approaching them during the design stage of your project so that they can review the design for feasibility and value of analysis. You can fill out the form on their website or directly email Gary Shapiro, [email protected], to obtain their help.

Alternatively, many data collection companies like YouGov, Ipsos, Toluna, and Research Now offer survey programming and data analysis in addition to finding participants for you. You may want to contact them for a quote. They sometimes offer non-profit discounts as well.

If you do have access to a person with statistical experience but want to talk through the best approach to the data you intend to collect (e.g., regression vs. t-tests), why not chat with the Faunalytics Research Director through our Ask a Researcher office hour?

To Cite This Page:

Faunalytics (). Analyzing Results. Retrieved from https://faunalytics.org/analyizing-results