Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms

Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms
Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms

Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms

October 15, 2014

By: Leah Quin, Abt Associates

Just-launched evaluations of potentially groundbreaking agricultural “pull” mechanisms in Africa – pioneered by the multi-donor AgResults initiative – took center stage at the first major international conference on impact evaluations and systematic reviews in Asia. Hosted by the International Initiative for Impact Evaluation (3ie), the September 1-5, 2014, “Making Impact Evaluation Matter” event in the Philippines brought together 600 researchers, policymakers, program managers and development practitioners.

Among these was Stephen Bell of Abt Associates, who is advising Abt Associate’s quantitative evaluation team for AgResults led by Tulika Narayan. Abt has initiated quantitative impact evaluation of two agricultural market innovations in Kenya and Nigeria under AgResults. These innovations – on-farm storage in Kenya and a biocontrol of Aflatoxin in Nigeria called AflasafeTM – represent a new approach to agricultural development, one that offers potential profits to the private sector while advancing social improvements like more farm family income and healthier foods for consumers.

Traditional “push” mechanisms for achieving development objectives – program grants to NGOs, government agency actions, informational campaigns – have not generally produced widespread, sustainable results. So donor organizations are exploring the use of “pull” mechanisms to achieve development objectives by awarding cash prizes to the private sector for achieving agreed-upon targets.

Reliable evaluation is crucial to determine if this approach works and what adjustments could make it better. In the September presentation, “Designing Rigorous Impact Evaluations of Agricultural ‘Pull’ Mechanisms,” Bell laid out how Abt is designing these evaluations to measure the effect of the pull mechanism on private sector and smallholder behavior.

Quantitatively assessing the impact of these initiatives poses special challenges, particularly concerning the counterfactual – i.e., what farmers and relevant private sector players would have done in the absence of “pull” prizes. Abt’s evaluations use a range of causal inference strategies adapted to in-country circumstances: random assignment of villages to early versus late intervention implementation, comparative interrupted time series analyses, and specification tests based on “untreated outcomes.”

Design of these approaches involved the participation of AgResults private sector Implementers and designs were often selected based on their input. For example, when the seven Implementers in the Nigeria AflasafeTM Pilot told Narayan at a meeting in Nigeria that they would be unable to visit all 66 targeted villages in the first year, she suggested a way to randomly select villages for each year, thus allowing evaluators to conduct a baseline survey of the yet-untouched villages for later comparison. Narayan had Implementers write names of the villages and put them in a jar, which she then pulled out to select villages for each stage—a simple, transparent approach to randomized controlled trials, often called the “gold standard” for evaluation. The Nigeria evaluation team includes Abt’s Judy Geyer and Mikal Davis.

In Kenya, randomized controlled trials proved impractical, as Implementers will be rolling out innovations in all target villages at once. Abt assessed communities outside the project zone but found them too different to make a good comparison. So the evaluation will assess data from target villages prior to intervention and after, a method called comparative interrupted time series analyses, which responds flexibly to on-the-ground reality. The Kenya evaluation team includes Abt’s Fatih Unlu and Elizabeth Ness-Edelstein.

View the full presentation presented by Abt Associates at the 3ie Conference on September 2, 2014.

Photo: Abt evaluators Tulika Narayan and Betsy Ness-Edelstein in Embu, Kenya with farmers who are using PICS bags, or Purdue Improved Crop Storage, one of the on-farm storage tools being assessed as a “pull” mechanism through the AgResults project. These farmers use the bags for sorghum, though most farmers would probably use them for maize.

The AgResults initiative is a partnership between: