An independent lens to gauge impact

External Evaluation

For all AgResults projects, the External Evaluator determines on a scientific basis if each Pay-for-Results (PfR) prize competition achieves its objectives. Abt Associates and IDinsight, which jointly serve in this role, apply the following approaches in their work:

Abt Associates and IDinsight split evaluation responsibilities for the different AgResults projects: While Abt Associates leads evaluation for the Brucellosis and Tanzania Dairy Challenge Projects, IDinsight leads evaluation for the FMD Vaccine and Senegal Crop Storage Finance Challenge Projects. Abt Associates also served as the External Evaluator for the now-closed projects in KenyaNigeriaUgandaVietnam, and Zambia.

Evaluator Questions

  1. What has been the impact of the AgResults Challenge project on private sector involvement in the development and uptake of agricultural innovations?
  2. What has been the impact of the AgResults Challenge project on smallholder farmers’ uptake of innovative technologies?
  3. What has been the impact of the AgResults Challenge project on smallholder farmers’ incomes?
  4. What evidence exists that the AgResults prize competitions are scalable and that their effects will be sustainable in the medium to long term?
  5. What has been the impact of the AgResults Challenge project on poor customers’ demand for derivative food products?
  6. What is the evidence on the scale of any effect on private sector investment and uptake, and on the cost effectiveness of AgResults as an approach?
  7. What lessons can be learned about the best practices in the design and implementation of agricultural prize competitions?

AgResults Theory of Change

AgResults uses incentives to overcome market failures for specific agricultural technologies. The prize competition model, which is a sub-class of Pay-for-Results approaches, incentivizes the private sector to work towards pre-defined results through rewards that are paid out only if the results are achieved. AgResults projects use prize competition incentives to encourage key private sector actors’ involvement in the market for a technology, either by developing the technology or encouraging its adoption by poor households.

The failure of a market to develop can be caused by low supply, low demand, a weak policy environment, or a combination of these. This diagram below shows some examples.

Constraints Limiting Market for Agriculture Technology

Diagram showing agriculture technology market constraints

Evaluation Framework

The Evaluator is charged with assessing whether the prize competition (what was previously called pull mechanism) incentives address the key underlying market failures and engage the private sector in the market while leading to smallholder farmer impact as implied in the evaluation questions.

The Evaluator employs a common evaluation across all AgResults prize competitions, applying economic theory to examine how each incentivizes the private sector to overcome the market failure and improve smallholder farmers’ welfare. It also identifies behaviors of market actors that might impede or aid development impact on smallholders farmers.

The common evaluation framework allows for comparison across AgResults’ prize competitions and generalization of results and lessons learned. In all AgResults project evaluations, the Evaluator applies rigorous qualitative and quantitative data gathering and analytic methods to test economic theory-driven hypotheses. The diagram below displays how the Evaluator answers all evaluation questions in its charge.

Diagram of evaluation framework

Evaluation Stages

1. Initial Qualitative Assessment

To inform each Challenge project-specific evaluation design, conduct an initial qualitative assessment to assess the project context, form a hypothesis of expected impact, examine the expected heterogeneity of impacts, identify external factors that might inhibit or boost impact, and meet with key stakeholders to secure their cooperation and buy-in.

2. Baseline

During the baseline stage, gather information on market for the technology, its adoption, initial engagement of potential competitors, as well as smallholder farmers’ livelihoods and agricultural practices.

3. Endline

During the endline stage, gather information on competitor activities, development of the technology market, and smallholder farmer adoption and income gains.

4. Cost-Effectiveness Analysis/Synthesis

At the end of AgResults, synthesize the findings across Challenge projects to draw lessons on the design and implementation of prize competitions. Assess AgResults’ cost effectiveness in achieving key outcomes across projects and also as compared to traditional approaches.

5. Sustainability Analysis

For key projects, evaluate the sustainability of AgResults’ impact by assessing private sector engagement in technology provision and the continuation of favorable development outcomes two years after the project ends.

Find out more about the Learning Library.

Related Documents

document icon

Evaluation Baseline Report: Senegal Crop Storage Finance Challenge Project

February 28, 2024

This evaluation baseline report for the AgResults Senegal Crop Storage Finance Challenge Project provides initial insights from IDinsight’s Impact Evaluation baseline, which involved data collection from 2,248 smallholder farmer respondents in Central and Southern Senegal in October-November 2022. It examines smallholder farmer storage and sales practices, net income, access to finance, and awareness of the WRS and AgResults project. These insights are crucial for ongoing impact assessments, shedding light on the project’s influence on net income and storage practices among smallholder farmers in Senegal.

EvaluatorLearningLearning LibrarySenegal Crop Storage Finance

document icon

Evaluator’s Sustainability Assessment: Vietnam GHG Emissions Reduction Challenge Project

May 23, 2023

Two years’ after the project’s conclusion, Evaluators found that several companies and many farmers were continuing to use (with some adaptation) rice production systems that had been introduced by companies during the competition. Farmers were more likely to use those systems if they received continuing support from the company and if water management systems were reliable.

EvaluatorLearning LibraryVietnam Emissions

document icon

Evaluator’s Gender Assessment: Tanzania Dairy Productivity Project

This assessment, conducted during project implementation, found significant diversity in the roles men and women play in households’ dairy activities, with female farmers often less well positioned than males to achieve and benefit from dairy intensification. While the project’s design offsets some risk of negative outcomes to women, payment-for-results projects have limited leeway to promote gender equity.

EvaluatorLearning LibraryTanzania Dairy

Senegal Crop Storage Finance Process Tracing Evaluation – Baseline

October 6, 2022

This report from External Evaluator IDinsight documents findings from the baseline Process Tracing evaluation of the AgResults Senegal Crop Storage Finance Challenge Project. The purpose of the baseline Process Tracing evaluation was to assess the state of the storage finance market during early stages of the project and assess if the preconditions are in place for the project to have its intended impact.

EvaluatorSenegal Crop Storage Finance

Evaluator Lessons Learned 8: Impact Evaluation Methods Popular

October 4, 2022

This brief by External Evaluator Abt Associates discusses quantitative methods for conducting impact assessments of smallholder farmer benefits from donor-devised interventions that stimulate private sector engagement in markets for technologies benefiting the farmers. The brief presents three AgResults Pay-for-Results competitions as examples and offers seven lessons for evaluators.

EvaluatorKenya StorageNigeria AflasafeVietnam Emissions

Evaluation Final Report: Vietnam GHG Emissions Reduction Challenge Project

January 28, 2022

This evaluation final report by Abt Associates shows that Pay-for-Results spurred private sector development and dissemination of emissions-reducing, yield-enhancing technology packages among smallholder farmers in Vietnam. A 2100-farmer survey showed that AgResults farmers achieved 14 percent higher yields than comparison farmers and used practices associated with lower emissions. However, project measurements of emissions were uncertain, so evaluators could not determine emissions impact. Future projects should pay attention to reliability of field-based emissions measurements.

EvaluatorVietnam Emissions

Evaluator Lessons Learned 7: Vietnam GHG Emissions Evaluation Findings

January 27, 2022

This brief summarizes the final evaluation report of the Vietnam GHG Emissions Reduction Challenge Project, showing that PfR can spur the private sector to develop and disseminate emissions-reducing technology packages for smallholder farmers. AgResults farmers achieved 14 percent higher yields than comparison farmers and were more likely to use practices associated with lower GHG emissions. The project’s main challenge was obtaining reliable measurements of emissions.

EvaluatorVietnam Emissions

Evaluation Design: Tanzania Dairy Productivity Challenge Project

May 17, 2021

The evaluation design for the Tanzania Dairy Productivity Challenge Project by Abt Associates lays out a mixed-methods approach. To assess impact on dairy producers, the evaluator will use a quasi-experimental design to gauge technology adoption and subsequent impacts on income and consumption. The approach will use a structure, conduct, performance framework to assess market impact.

EvaluatorTanzania Dairy

Summary of Evaluator Findings and Lessons from AgResults Prize Competitions: 2013-2020

May 11, 2021

This report by External Evaluator Abt Associates summarizes independent evaluation findings from competitions in Nigeria, Kenya, Uganda, and Zambia and offers guidance for future prize sponsors. It provides evidence that Pay-for-Results (PfR) competitions can spur development of markets for technologies benefitting smallholder farmers. It also offers guidance on how to determine whether PfR is an appropriate design option and how to maximize farmer benefits.

EvaluatorKenya StorageNigeria AflasafeUganda LegumesZambia PVA Maize

Sustainability Assessment: Kenya On-Farm Storage Challenge Project

May 7, 2021

This sustainability assessment by External Evaluator Abt Associates looking at the Kenya On-Farm Storage Challenge Project affirms that Pay-for-Results projects can catalyze development of a smallholder- and gender-inclusive, sustainable market for beneficial agricultural technologies. Since the project ended, there have been new entrants to the market, ongoing product innovation, and a 250% increase in sales of on-farm storage products attributable to AgResults.

EvaluatorKenya Storage

Evaluation Design: FMD Vaccine Challenge Project

January 19, 2021

The evaluation design for the Foot and Mouth Disease (FMD) Vaccine Challenge Project, by IDinsight, lays out a mixed-methods approach relying primarily on qualitative process-tracing techniques. The document highlights the role of data collected from potential vaccine manufacturers, buyers, policymakers, and others to explore how the project impacts the development and uptake of FMD vaccines in Eastern Africa.

EvaluatorFMD Vaccine

Evaluator Lessons Learned #6: Zambia Biofortified Maize Evaluation Findings

January 4, 2021

This brief summarizes the close-out assessment of the Zambia Biofortified Maize Challenge Project. External Evaluator Abt Associates found that the competition helped develop a niche market for Pro-Vitamin A maize, although Vitamin A-deficient populations were a small portion of those who consumed the maize. The brief underscores the importance of directly targeting intended beneficiaries in project design.

EvaluatorZambia PVA Maize

Evaluation End-of-Project Update: Nigeria Aflasafe Challenge Project

November 21, 2020

Evaluator Abt Associates revisited the Nigeria Aflasafe Challenge Project at its conclusion, two years after the comprehensive mixed-methods assessment. They found that private sector involvement in the Aflasafe-treated maize market had increased despite reduction in the AgResults incentive, but that the Aflasafe-treated maize market’s sustainability was uncertain given general maize market instability and delayed implementation of aflatoxin-related policies.

EvaluatorNigeria Aflasafe

Evaluation Final Report: Nigeria Aflasafe Challenge Project

November 18, 2020

The Nigeria Aflasafe Challenge Project evaluation (published in February 2019, end-of-project update added November 2020) found that the project succeeded in creating a niche market for Aflasafe-treated maize. Aflasafe adoption increased by 56 percentage points, and average net income from maize increased by 16 percent. The sustainability of the market is uncertain.

EvaluatorNigeria Aflasafe

Evaluation Design: Vietnam GHG Emissions Reduction Challenge Project

November 11, 2020

The evaluation design for the Vietnam GHG Emissions Reduction Challenge Project, by Abt Associates, lays out a mixed methods approach, including a randomized control design. The document highlights the role of interviews, surveys, and farmer diaries to explore the effect of the Vietnam project on the market for emissions-reducing technologies and to estimate impacts on farmers’ knowledge, attitude, practices, and incomes.

EvaluatorVietnam Emissions

Evaluator Lessons Learned #3: Nigeria Aflasafe Project Evaluation Findings

November 1, 2020

This brief, updated in November 2020, summarizes evaluation findings for the Nigeria project. External Evaluator Abt Associates found that AgResults helped create a niche market for Aflasafe-treated maize and an increase in farmers’ uptake of Aflasafe and net maize income. Market sustainability is uncertain due to unstable market conditions and delays in implementation of aflatoxin-related policies.

EvaluatorNigeria Aflasafe

Evaluation: Vietnam Challenge Project RCT Adherence Report

October 7, 2020

Abt Associates’ evaluation design of the Vietnam GHG Emissions Reduction Challenge Project envisions a randomized control design. This report assesses the success of the initial randomization and confirms that competitors adhered to rules respecting communes set aside for control; it also confirms similarities between control and treatment groups at baseline.

EvaluatorVietnam Emissions

Evaluator Lessons Learned #5: Incorporating a Gender Lens into Challenge Projects

This brief by External Evaluator Abt Associates explores how Pay-for-Results (PfR) projects could potentially perpetuate disadvantages that women face in agricultural market systems and suggests project design approaches to improve gender outcomes. It also presents results from AgResults’ Kenya PfR project that led to unique benefits for women.

EvaluatorKenya Storage

Final Report: Nigeria Aflasafe Challenge Project

August 26, 2020

More than 4.5 billion people in developing countries are exposed to aflatoxins, one of the world’s most carcinogenic substances, through their diets. Yet a biocontrol solution exists: AflasafeTM. The AgResults Nigeria AflasafeTM Challenge Project (2013-2019) used a Pay-for-Results (PfR) prize competition to motivate the private sector to overcome critical barriers preventing widespread adoption of AflasafeTM among Nigerian smallholder farmers working in the maize sector. Over six years, the competition engaged 41 competitors to reach 75,788 farmers, producing more than 315,000MT of AflasafeTM-treated maize.

EvaluatorNigeria Aflasafe

Evaluation Baseline Report: Brucellosis Vaccine Challenge Project

August 14, 2020

This baseline report by External Evaluator Abt Associates describes the Brucellosis vaccine landscape at baseline. Evaluators found that the vaccine market is weak and no new vaccines to control Brucellosis have been developed in recent years. Evaluators also found that the Milestone 1 prize seemed to incentivize some new entities to initiate Brucellosis vaccine R&D.

Brucellosis (Global)Evaluator

Evaluator Lessons Learned #4: Kenya On-Farm Storage Challenge Project Evaluation Findings

July 10, 2020

This brief by External Evaluator Abt Associates summarizes the evaluation of the AgResults Kenya On-Farm Storage Challenge Project. Abt Associates found that the competition created an emerging market for improved on-farm storage devices and increased adoption of on-farm storage technologies. However, impacts on farmer income were small. The brief underscores the challenge of constructing an accurate theory of change.

EvaluatorKenya Storage

Can Pull Mechanisms Achieve Agriculture Development Objectives? Evidence from AgResults External Evaluation

June 5, 2020

This DFID Chief Economist Seminar Series presentation from February 2020 summarizes External Evaluator Abt Associates’ findings about the use of “pull mechanisms” or “prize competitions” to develop markets that provide innovative agricultural technologies to smallholder farmers. Findings suggest that pull mechanisms can create markets as long as the binding constraints inhibiting market development are within the manageable interest of the companies competing for the prize.

EvaluatorKenya StorageNigeria AflasafeUganda LegumesZambia PVA Maize

Brucellosis Vaccine Challenge Project: 2019 Interim Assessment

May 15, 2020

This assessment by External Evaluator Abt Associates reflects the experience of competitors in the Brucellosis Vaccine Challenge Project from mid-2018 to mid-2019. Most competitors reported that the competition had helped advance their B. melitensis vaccine R&D. However, many of them anticipated challenges completing the safety and efficacy trials due to resource constraints.

Brucellosis (Global)Evaluator

Evaluation Final Report: Kenya On-Farm Storage Challenge Project

April 30, 2020

The external evaluation of the Kenya On-Farm Storage Challenge Project found that the project created an emerging market for improved on-farm storage products in Kenya. Impacts on farmer income and food security were small, and the main reported benefit of adoption was reduced use of pesticide dust in maize stored for home consumption.

EvaluatorKenya Storage

Evaluation Design: Brucellosis Vaccine Challenge Project

August 1, 2019

The evaluationdesign for the Brucellosis Vaccine Challenge Project lays out the approach to assessing the project’s influence in stimulating the development of an improved Brucella melitensis vaccine. Evaluators used a qualitative assessment and counterfactual disease study to examine technology development and used cost-effectiveness simulations as well.

Brucellosis (Global)Evaluator

USAID BFS Evaluating Pull Mechanisms, Initial Lessons

November 22, 2018

This presentation includes an overview of AgResults pull mechanisms, the evaluation framework for the external evaluation and initial lessons learned. The authors are Tulika Narayan and Denise Mainville of Abt Associates and Denise Mainville Consulting.

Evaluator

Evaluation Baseline Report: Uganda Legume Seed Challenge Project

August 1, 2018

This report presents baseline evaluation results for the Uganda Legume Seed Challenge Project, which was terminated early. It includes findings from a survey of 1,800 smallholders, in-depth interviews with a subset of farmers, and seed quality assessment results using seed from 300 farmers who reported using quality seed.

EvaluatorUganda Legumes

Evaluator Lessons Learned #2: Pull Mechanisms for Overcoming Market Failures in the Agriculture Sector

November 1, 2017

Pull mechanisms require a solid understanding of the market and the motivations of the private sector. This November 2017 brief, with examples from AgResults’ work in Kenya, provides guidance for development practitioners interested in incorporating pull mechanisms in their own work.

EvaluatorKenya Storage

A Framework for Evaluating Innovation Challenges

March 1, 2017

Governments and philanthropists are increasingly considering prize contests to spark creative solutions to thorny social problems. But only a few contests have been systematically evaluated. This March 2017 evaluation framework — based on AgResults’ work to evaluate Brucellosis vaccine development — can be applied to any innovation challenge.

Evaluator

Evaluation Design: Uganda Legume Seed Challenge Project

February 7, 2017

The Uganda Legume Seed Challenge Project’s evaluation design envisions a performance evaluation using a pre-post design to assess project success in creating a smallholder-inclusive market for quality legume seed. It uses a structure-conduct-performance framework to assess market impact and a baseline-endline comparison of smallholder farmer survey data to understand smallholder benefits.

EvaluatorUganda Legumes

Evaluation Baseline Report: Zambia Biofortified Maize Challenge Project

April 4, 2016

The baseline report for the Zambia Biofortified Maize Challenge Project evaluation uses data from 212 interviews of maize market actors (including smallholders, rural non-farming residents, retailers, and millers), as well as 1,013 interviews with urban shoppers. It presents findings about market actors’ and consumers’ awareness, production, purchase, and consumption of pro-Vitamin A maize.

EvaluatorZambia PVA Maize

Evaluation Design: Zambia Biofortified Maize Challenge Project

July 13, 2015

The evaluation design for the Zambia Biofortified Maize Challenge Project presents how evaluators will assess whether the project motivates industrial maize millers to invest in and drive expansion of the pro-Vitamin A maize market. The report presents the structure-conduct-performance framework, pre-post comparison design, and data collection methods (qualitative interviews; quantitative survey of low-income urban consumers).

EvaluatorZambia PVA Maize

Evaluation Baseline Report: Kenya On-Farm Storage Challenge Project

April 20, 2015

This report presents baseline findings for the Kenya On Farm Storage Challenge Project based on survey data from 4,765 farming households and extensive qualitative data. It describes low uptake of improved on-farm storage at baseline but identifies latent demand in the market. The report also describes farmers’ maize revenue and food security.

EvaluatorKenya Storage

Evaluation Design: Nigeria Aflasafe Challenge Project

December 30, 2014

The evaluation design for the Nigeria Aflasafe Challenge Project lays out the approach to assess the project’s influence in stimulating a smallholder-inclusive market for Aflasafe-treated maize. Evaluators assess market impact using a structure-conduct-performance framework. To evaluate smallholder benefits, the design proposes a randomized control trial (later adapted to a quasi-experimental design).

EvaluatorNigeria Aflasafe

Evaluation Design: Kenya On-Farm Storage Challenge Project

June 27, 2014

The Kenya On-Farm Storage Challenge Project’s evaluation design report lays out the approach to assess whether the project stimulates a smallholder-inclusive market for improved on-farm storage devices. Evaluators assess impact on the market using a structure-conduct-performance framework. To evaluate resulting smallholder benefits, the report describes a short interrupted time series design.

EvaluatorKenya Storage