Challenge 22+ Pilot Impact Study

The aim of this study is to explore new methods for evaluating the impact of Challenge 22+, a global program that encourages people to try a plant-based diet for 22 days. Due to low response rates in the past, a new data collection method was used for the evaluation – WhatsApp messages. The new method lead to increased response rates (50.3%), which in turn provided a more representative data.

Participants were randomly chosen from six different Challenge groups in order to check the effect of three variables: reported diet prior to taking the Challenge, time passed after completing the Challenge (one month or seven months), and number of times the participants signed up for the Challenge (one time or two times).

The study found the following:

• The only significant predictor of diet change we found was diet prior to the Challenge. The time passed after completing the Challenge (one month or seven month) and the number of times a participant signed-up for the Challenge had no influence on diet change.
• Over 77% of frequent meat-eaters, 64% of reducetarians and 34% of almost vegetarians, reported that they reduced their meat consumption after completing the Challenge.
• A very small amount of participants who were veg*n prior to taking the Challenge returned to eating meat (about 3%).

A calculation of the number of portions of meat saved resulted in the conclusion the average Challenge participant reduces 86 portions of meat per year.

Challenge 22+: Project Background

Challenge 22+ is a global program that encourages people to try a plant-based diet for 22 days, while focusing on the social aspects that accompany the transition. The program was developed by Israeli organization Animals Now, and it was launched in Israel in March 2014.

The program runs all year round, and new cycles of the Challenge constantly open. The participants of each cycle are added to a secret Facebook group. This means that each participant joins the same group together with all those who start the Challenge on the same day as them. This nurtures a sense of belonging to a community of people, going through a journey together. In addition to receiving practical advice in the form of pre-written content, such as daily challenges, bi-daily emails, etc., participants are also provided with guidance from personal mentors. The mentors are required to go through extensive training before they start volunteering. Moreover, every group is supervised by registered dietitians.

The program’s strength lies in its unique ability to address the individual needs of each person, while supporting thousands of participants at a time. The number of participants in each cycle of the Challenge ranges between 300 to 1,000 and above.

In the last two years, the project has grown rapidly and now includes specialized tracks suited for audiences from different backgrounds and cultures, and in several languages. The four main ones include the International Challenge (in English), the Israeli Challenge, the U.S. Challenge and the U.K. Challenge. In 2018, 77,349 people participated in the Challenge. A Spanish Challenge will be launched in August.

The project’s staff includes seven part-time employees, all based in Israel, as well as over 1,200 volunteers from around the world. Over the years, the staff of Challenge 22+ aided other animal protection groups and organizations in developing their own local programs. Countries in which partners developed similar programs under our guidance include Denmark, Norway, Taiwan, Romania and more.   

Problems With Email Surveys

Each year, Challenge 22+ conducts an internal analysis of the project’s impact. While the results are generally satisfactory, we are faced with a low response rate. For example, in a survey held in 2018 and distributed via email, most participants reported that they remained vegan (67%), and many others reported that they plan to progress towards a plant-based diet. Although these results were very encouraging, they were based on a low response rate of 14%.

There’s a risk that the low response rate affects the results of the survey. For example, there’s a chance that participants who had a better experience in the Challenge and succeeded in making the dietary change they aimed for, will tend to answer the survey more than other. An effect in the opposite direction is also possible.

For these reasons, we decided to look for better ways collect data, with the aim of increasing the response rates, thus eliminating some potential problems like a sampling bias.

Participants

The survey was sent to 1,431 Challenge participants from Israel, who signed up for the program during 2018 and were assigned to one of six separate Challenge groups. 720 study participants responded (50.3%). The study attempted to examine the following variables as potential predictors of dietary change:

1. When was the participant surveyed: one month or seven months after completing the Challenge.
2. Number of times participant signed-up for the Challenge: one or two times.
3. Challenge group (one of six)

Procedure

The data was collected between November 2018 and February 2019. We held a survey using a new method for collecting the data – WhatsApp messages. The survey was sent via a WhatsApp message to the chosen participants. In order to maximize response rates, the text message was sent directly from Challenge 22+’s Director, Neta Rosenthal. The content of the message was personalized: it included the participant’s first name, and we made sure to use gender appropriate pronouns and verb forms (it is very difficult to compose a gender neutral message in Hebrew).

This is a sample message sent to Ariel, a male participant:

Hi Ariel,
This is Neta from Challenge 22+.
I noticed you have completed the Challenge recently. Congrats!
I want you to know that the team and I are here for you in the future as well. If you need any help or advice, let us know and I’ll gladly take care of it. This is exactly what we’re here for.

I would like to ask you something a bit personal, I hope it’s ok. If you could please share with me (I promise to keep it between us):

How often did you eat meat, chicken or fish in the last month:
Daily.
Up to five times a week.
Between 2 and 4 times a week.
Once a week or less.
Not at all.

We decided to ask participants about their actual meat consumption instead of their self-definition because studies have shown that there are gaps between these two notions.

After a participant responded, we thanked them, and proceeded to send a follow-up question about their diet prior to the Challenge:

Thank you for your answer Ariel. I hope it’s ok if I ask one more question. How often did you use to eat meat, chicken or fish before the Challenge?
Daily.
Up to five times a week.
Between 2 and 4 times a week.
Once a week or less.
Not at all.

Results

Response Rate
Response rates were exceptionally high: 50.3% of people who were sent the survey completed it (meaning they answered both questions). This is a big improvement from previous analyses we conducted, in which response rates were around 15%.

Initial Analysis

We examined the following items as potential predictors of dietary change:

1. Participant’s reported diet prior to the Challenge: we divided participants into four categories:

• Frequent meat-eaters: participants who chose survey option 1 or 2, i.e. they consumed meat at least five times a week.
• Reducetarians: participants who chose survey option 3, i.e. they consumed meat two to four times a week.
• Almost vegetarians: participants who chose survey option 4, i.e. they consumed meat once a week or less.
• Veg*ns (vegans and vegetarians): participants who chose survey option 5, i.e. they consumed no meat.

2. When was the participant surveyed: one month or seven months after completing the Challenge.

3. Number of times participant signed-up for the Challenge: one or two times.

4. Challenge group.

Note that we use the term meat throughout this report to indicate any kind of animal flesh, including chicken and fish, even when it’s not specifically mentioned.

A regression analysis simultaneously considering these variables found that only previous diet was a significant predictor of change. Namely, eating more meat pre-Challenge was associated with a greater likelihood of reduction following the Challenge (p < .001).  Controlling for pre-Challenge diet, the other factors had no influence on diet change. For this reason, we present the results of our study by pre-Challenge diet but do not separate by these other factors.

Factors Affecting Retroactive Self-Report

Requesting participants to retroactively report their own diet prior to the Challenge has some shortcomings, and we wanted to see whether the factors introduced in the previous section had any effect on the answers we received. A regression analysis simultaneously considering factors b-d found that the time passed since the participant completed the Challenge (one month or seven months) was a significant predictor of pre-Challenge diet (p = .03), and challenge number was marginally significant (p = .09), but natural group was not significant.

This means that all else being equal, being surveyed later (at seven months) predicted giving lower estimates of pre-Challenge meat consumption. This may indicate memory bias in these respondents. However, bias in this direction lowers our ability to detect a significant reduction in meat consumption for these participants, and therefore this does not pose a problem for interpretation of the overall reduction in meat consumption (presented in section 4). In addition, all else being equal, participating in the Challenge for the second time was associated with marginally lower pre-Challenge meat consumption. This makes sense because their pre-Challenge consumption was likely already lowered by their previous Challenge.

Analysis According To Diet Prior To Taking The Challenge  

The following sections each present the data obtained for each of these categories, accompanied by a discussion of the results.

Although they were not asked about consumption of animal products other than meat, some participants responded saying they were vegan. We decided not to analyze them as a separate group, because our focus was on meat consumption, being that the meat industry is responsible for the vast majority of animal suffering.

Figure 1: Participants’ diet prior to taking the Challenge.

Frequent Meat-Eaters

Fig. 2 summarizes the reported post-Challenge diet of participants who reported consuming meat at least five times a week before joining the Challenge (frequent meat-eaters).

Figure 2: Reported diet after the Challenge of frequent meat-eaters across all groups.

The results are encouraging: only 21.7% of frequent meat-eaters participants didn’t make any change, or made very small changes, to their diet, while more than 77% of them reduced their meat consumption. 29.2% made moderate changes, and 48.5% made a substantial change. We define moderate change as reducing meat consumption by two portions a week, and substantial change a reducing more than that.

Reducetarians

Figure 3 summarizes the reported post-Challenge diet of participants who reported consuming meat two to four times a week before joining the Challenge (reducetarians). No participant reported an increase in meat consumption, and about 64% reported reducing their meat consumption.

Figure 3: Reported diet after the Challenge of reducetarians.

Veg*ns And Almost Vegetarians

Challenge participants who were already veg*n or almost vegetarian when signing up can be divided into two, according to their main motivation for signing up. First, there are those who already made a dietary change and eliminated significant amounts of meat and/or animal products from their diet, but have difficulties maintaining their new diet. The others are those who still haven’t completed the transition to veg*nism, and join the Challenge in order to make it happen.

For some participants of the first type, signing up for the Challenge is a final attempt to try and be veg*n before returning to consuming meat. These people have a difficulty maintaining a plant-based diet for health, culinary and social reasons, among others. Indeed, a small percentage (less than 3%) from each group goes back to eating meat after completing the Challenge.

Almost Vegetarians

About a third of the participants from this group became veg*n, while the rest did not change their diet significantly, and they continue consuming small amounts of meat.

Figure 4: Reported diet after the Challenge of almost vegetarians.

Veg*ns

The overwhelming majority (97.8%) of Veg*ns surveyed reported they did not increase their meat consumption after participating in the Challenge.

Figure 5: Reported diet after taking the Challenge of participants who were veg*n before signing up.

Vegans

As mentioned above, even though participants were only asked about meat consumption, some participants voluntarily reported that they were vegan. It is not unlikely that there were more vegans, who responded with survey option 5 and did not elaborate.

Table 6: Veg*n and vegan participants.

In table 6 we see that 19.3% of participants who were frequent meat eaters prior to the Challenge reported being veg*n after completion. Out of those, 32.3% reported being vegan.

16.1% of participants who were reducetarian prior to the Challenge reported being veg*n after completion. Out of those, 8% reported being vegan.

32.6% of participants who were almost vegetarian prior to the Challenge reported being veg*n after completion. Out of those, 14.3% reported being vegan.

97.9% of participants who were veg*n prior to the Challenge reported being veg*n after completion. Out of those, 15.8% reported being vegan.

97.7% of participants who were vegan prior to the Challenge reported being vegan after completion.

Calculating The Number Of Meat Portions Reduced By Challenge Participants

From this data, we can calculate the number of meat portions reduced by the average Challenge participant. We estimate that an average participant cuts 86 portions of meat per year from their diet (this amounts to 12.3 animals per year, based on an estimate that an average person consumes 50 animals per year).

The calculation was made as follows: frequent meat-eaters were counted as being able to reduce up to 6 portions a week, reducetarians as being able to reduce up to 3 portions a week and almost vegetarians as being able to reduce up to one portion a week. Participants who were veg*n prior to taking the Challenge and remained that way were counted as if they were saving one portion per week. This was done because many veg*ns have difficulties maintaining a veg*n diet, and go back to consuming meat. For similar reasons, participants who reported being almost vegetarian prior to the Challenge and remained so were counted as if they were saving 0.5 portions per week.

We added an additional saving of 0.5 portions for those who reported being vegan. This is because ultimately, we are interested in the number of animals saved, and counting meat portions is just a convenient metric for doing that. Since the egg and dairy industries are also responsible for the death of many animals, we decided to include it in the calculation. The table below summarizes the number of meat portions saved by each type of participant.

Table 7: Number of meat portions saved per week per participant, according to diet before and after the Challenge.

Summary And Discussion

This study was a pilot study, aimed at improving our understanding of the impact of Challenge 22+ by using a new method for data collection in order to improve response rates, which in turn improve the accuracy of the data obtained.

The new response collection method proved successful in increasing response rates (50.3%), thus eliminating some methodological issues related to low response rates. We see three main reasons for the increase in response rates. First, the platform itself – Whatsapp is an instant messaging platform, and therefore users perceive the response process as faster. Second, we asked participants only two question in total, and at first it appeared as if there was only one (the second question was sent only to those who had answered the first). Finally, the message was personalized. It included the participant’s first name, used the appropriate gender, and was sent in the name of the program director. All these factors contributed to the (accurate) feeling that we have a genuine interest in the participants’ personal experience.

One unexpected advantage of this method was that after responding to the survey, some participants took the opportunity to continue their dialog with the program, ask for advice and even sign up for another round of the Challenge or for the alumni group.

We found that over 77% of frequent meat-eaters, over 64% of reducetarians and 34% of almost vegetarians, reduced their meat consumption after completing the Challenge. The Challenge is also successful in providing support to veg*ns: very few veg*n respondents reported increasing their meat consumption after taking the Challenge (about 3%).

It is important to note that the study sampled only Israeli participants. Therefore, the application of its conclusions for the other Challenge programs is limited. Other than the obvious differences in culture and in the staff between the programs, Israeli Challenge participants also receive a personal phone call from their mentor when they begin the Challenge. Currently, this is not the case for the other programs, and it can have an effect on the impact of the program, as well as on response rates

The study also had some weaknesses, other than the above-mentioned issue of number of respondents. First, there is a risk that social desirability had an effect on the results, as the study was conducted in the name of the project. Not only that, it was sent from the project manager herself. Our request to answer the survey honestly and the promise to keep the answers confidential may have balanced this effect.

Second, the short length of the survey also has a disadvantage – we did not collect demographic data such as age, gender, location, etc. This information could have provided us with additional insights.

Future Research

In the future we intend to examine Challenge participants from other, non-Israeli programs. Drawing conclusions from this study, we will implement several changes.

We also intend to find a way to match the information we receive from Challenge participants regarding their diet when they sign up for the program with survey respondents. That way, we will have a more accurate measure of their diet prior to the Challenge.

We will also try to measure the long term effect of the Challenge on a single Challenge group. We will randomly split the group in half, and survey each half in a different time after they complete the program.

Some participants reported themselves as pescatarians. We did not include them as a separate group in the analysis for methodological reasons, but we might want to look into that group more systematically in the future.

***

We thank Jo Anderson from Faunalytics for her major help with the statistical analysis, as well as for her insightful comments, and Ido Ziv-Li for his assistance.