Updates from our External Evaluator, Abt Associates on AgResults Evaluations

As the external evaluator for the AgResults pilots, Abt uses an evaluation framework grounded in economic theory regarding the behavior of economic agents in agricultural markets, coupled with rigorous quantitative and qualitative evaluation methods, to evaluate pilots’ impact on:

• Private sector engagement and market development
• Smallholder income and adoption of the technology
• Scale, cost-effectiveness, and sustainability

Our evaluation of each pilot generally progresses along the following stages:

• An Initial Qualitative Assessment to gather data about the pilot context, theory of change, and external factors that might affect pilot performance, which informs our plans to conduct the evaluation as explained in the design protocol;
• Baseline visits, surveys and interviews from which to codify and analyze data for a baseline report;
• Endline visits, surveys and interviews to gather data that are analyzed in an endline report;
• A cost-effectiveness analysis to determine the cost of each pilot’s impact in comparison to traditional development mechanisms; and
• A sustainability analysis for some pilots to evaluate the continued impact of AgResults two years after the pilot ends.

A pilot-specific summary of our progress is presented below:

• Nigeria Alfasafe Pilot: We are using a quasi-experimental design to evaluate the impact of the AgResults pull mechanism on farmer adoption of a biocontrol called Aflasafe to reduce aflatoxins in maize. In Spring 2017, we completed the endline farmer survey, and have nearly completed analyzing the resulting quantitative data. We will soon travel to Nigeria to conduct the qualitative endline survey for incorporation into our final evaluation report.
• Kenya On-Farm Storage Pilot: We are using an interrupted time-series approach to evaluate the extent to which AgResults has increased the adoption of storage technologies that will improve food security and income for smallholder farmers. We submitted an addendum to our baseline in October 2016, which presents an updated time-series analysis and baseline values for key outcomes. Abt will review monitoring data as the pilot is implemented in preparation for the endline data collection and analysis at the end of the pilot.
• Zambia Biofortified Maize Pilot: We are using a mixed-methods approach including in-depth qualitative research with a quantitative survey of urban consumers to assess the impact of the pull mechanism on the development of the market for Vitamin A fortified maize in Zambia. In December 2016, we collected additional qualitative data about the perception of implementers regarding some recent changes to the design of the incentive structure for the pilot in order to further inform the baseline. We will continue to conduct qualitative interviews as appropriate until the endline data collection and analysis take place at the end of the pilot.
• Uganda Legume Seeds Pilot: We are using a mixed-methods evaluation with pre-post quantitative analysis to determine the extent to which AgResults has increased the sales of improved seeds in Uganda. We completed the collection of qualitative data for the baseline report in April and May 2017. The team is synthesizing these survey results to develop the baseline report.
• Vietnam Emissions Reduction Pilot: We are assessing the feasibility of using a randomized controlled trial to evaluate the impact of the AgResults pull mechanism on reducing greenhouse gas emissions from rice farming while increasing farmer returns. We conducted an initial qualitative assessment and design trip to Vietnam in August 2017. We will design our evaluation of this pilot based on information from the solvers’ presentations of the technologies that will be tested during Phase 1 of the pilot.
• Brucellosis Vaccine Pilot: We are applying a mixed-methods approach combining a qualitative assessment, a counterfactual disease study, and cost-effectiveness simulations to determine the effects of the prize mechanism on private sector involvement and investment in developing an improved vaccine for Brucellosis. We are reviewing the judging panel notes to gather data about how the judges are approving solver applications. We will conduct interviews of the solvers after all the applications have been received and will synthesize the results into our baseline report.