Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms

October 15, 2014

group of people standing in front of a shack

By Leah Quin, Abt Associates

Just-launched evaluations of potentially groundbreaking agricultural “pull” mechanisms in Africa – pioneered by the multi-donor AgResults initiative – took center stage at the first major international conference on impact evaluations and systematic reviews in Asia. Hosted by the International Initiative for Impact Evaluation (3ie), the September 1-5, 2014, “Making Impact Evaluation Matter” event in the Philippines brought together 600 researchers, policymakers, program managers and development practitioners.

Among these was Stephen Bell of Abt Associates, who is advising Abt Associate’s quantitative evaluation team for AgResults led by Tulika Narayan. Abt has initiated quantitative impact evaluation of two agricultural market innovations in Kenya and Nigeria under AgResults. These innovations – on-farm storage in Kenya and a biocontrol of Aflatoxin in Nigeria called AflasafeTM – represent a new approach to agricultural development, one that offers potential profits to the private sector while advancing social improvements like more farm family income and healthier foods for consumers.

Traditional “push” mechanisms for achieving development objectives – program grants to NGOs, government agency actions, informational campaigns – have not generally produced widespread, sustainable results. So donor organizations are exploring the use of “pull” mechanisms to achieve development objectives by awarding cash prizes to the private sector for achieving agreed-upon targets.

Reliable evaluation is crucial to determine if this approach works and what adjustments could make it better. In the September presentation, “Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms,” Bell laid out how Abt is designing these evaluations to measure the effect of the pull mechanism on private sector and smallholder behavior.

Quantitatively assessing the impact of these initiatives poses special challenges, particularly concerning the counterfactual – i.e., what farmers and relevant private sector players would have done in the absence of “pull” prizes. Abt’s evaluations use a range of causal inference strategies adapted to in-country circumstances: random assignment of villages to early versus late intervention implementation, comparative interrupted time series analyses, and specification tests based on “untreated outcomes.”

Design of these approaches involved the participation of AgResults private sector Implementers and designs were often selected based on their input. For example, when the seven Implementers in the Nigeria AflasafeTM Pilot told Narayan at a meeting in Nigeria that they would be unable to visit all 66 targeted villages in the first year, she suggested a way to randomly select villages for each year, thus allowing evaluators to conduct a baseline survey of the yet-untouched villages for later comparison. Narayan had Implementers write names of the villages and put them in a jar, which she then pulled out to select villages for each stage—a simple, transparent approach to randomized controlled trials, often called the “gold standard” for evaluation. The Nigeria evaluation team includes Abt’s Judy Geyer and Mikal Davis.

In Kenya, randomized controlled trials proved impractical, as Implementers will be rolling out innovations in all target villages at once. Abt assessed communities outside the project zone but found them too different to make a good comparison. So the evaluation will assess data from target villages prior to intervention and after, a method called comparative interrupted time series analyses, which responds flexibly to on-the-ground reality. The Kenya evaluation team includes Abt’s Fatih Unlu and Elizabeth Ness-Edelstein.

View the full presentation presented by Abt Associates at the 3ie Conference on September 2, 2014.

Photo: Abt evaluators Tulika Narayan and Betsy Ness-Edelstein in Embu, Kenya with farmers who are using PICS bags, or Purdue Improved Crop Storage, one of the on-farm storage tools being assessed as a “pull” mechanism through the AgResults project. These farmers use the bags for sorghum, though most farmers would probably use them for maize.

Recent News & Blog Posts

men and women gathered in a circle talking outside

Engaging Women’s Producer Organizations in Senegal to #InspireInclusion

March 7, 2024

The AgResults PM Team recently met with the Ndjoudien Women Producer Organization in the Kaffrine region of Senegal.…

People sitting at Africa Food Forum stage talking

AgResults and Land O’Lakes Venture37 Reflect on Gender Inclusion in Tanzania at #AGRF2023

September 12, 2023

When food systems are inclusive, understandable they are stronger, more resilient, and more sustainable. But the question of…

Poster with headshots of who is speaking at Africa Food Systems Forum

Exploring Gender Inclusion in Tanzania’s Dairy Sector at #AGRF2023

August 28, 2023

Building better food systems demands that youth and women play a central role—that’s what Africa’s Food Systems Forum…

Photo of people on a stage

AgResults Indonesia Aquaculture Challenge Project Announces Year 2 Prize Winners

June 13, 2023

To mark the end of a successful second year, the AgResults Indonesia Aquaculture Challenge Project held a series…