External Evaluation

The AgResults Steering Committee appointed Abt Associates to serve as the External Impact Evaluator for the AgResults initiatives. The Evaluator’s role is to determine on a scientific basis if the AgResults Pay-for-Results (PfR) prize competitions - previously referred to as pull mechanisms - achieve their objectives. In that role, the Evaluator does the following:

  • Uses a mix of quantitative and qualitative methods
  • Assesses whether Challenge projects produce private sector behaviors or social outcomes that are better or different than without PfR incentives
  • When possible, compares outcomes of prize competitions against outcomes of traditional "push" approaches
  • Analyzes cost effectiveness of Challenge projects and sustainability of results

Evaluator Questions

1
What has been the impact of the AgResults Challenge project on private sector involvement in the development and uptake of agricultural innovations?
2
What has been the impact of the AgResults Challenge project on smallholder farmers’ uptake of innovative technologies?
3
What has been the impact of the AgResults Challenge project on smallholder farmers' incomes?
4
What evidence exists that the AgResults prize competitions are scalable and that their effects will be sustainable in the medium to long term?
5
What has been the impact of the AgResults Challenge project on poor customers’ demand for derivative food products?
6
What is the evidence on the scale of any effect on private sector investment and uptake, and on the cost effectiveness of AgResults as an approach?
7
What lessons can be learned about the best practices in the design and implementation of agricultural prize competitions?

AgResults Theory of Change

AgResults uses incentives to overcome market failures for specific agricultural technologies. The prize competition model, which is a sub-class of Pay-for-Results approaches, incentivizes the private sector to work towards pre-defined results through rewards that are paid out only if the results are achieved. AgResults projects use prize competition incentives to encourage key private sector actors’ involvement in the market for a technology, either by developing the technology or encouraging its adoption by poor households.

The failure of a market to develop can be caused by low supply, low demand, a weak policy environment, or a combination of these. This diagram shows some examples:

Constraints Limiting Market for Agriculture Technology

Evaluation Framework

The Evaluator is charged with assessing whether the prize competition (what was previously called pull mechanism) incentives address the key underlying market failures and engage the private sector in the market while leading to smallholder farmer impact as implied in the evaluation questions.

The Evaluator employs a common evaluation across all AgResults prize competitions, applying economic theory to examine how each incentivizes the private sector to overcome the market failure and improve smallholder farmers' welfare. It also identifies behaviors of market actors that might impede or aid development impact on smallholders farmers.

The common evaluation framework allows for comparison across AgResults' prize competitions and generalization of results and lessons learned. In all AgResults project evaluations, the Evaluator applies rigorous qualitative and quantitative data gathering and analytic methods to test economic theory-driven hypotheses. The diagram below displays how the Evaluator answers all evaluation questions in its charge.

Evaluation Stages

1. Initial Qualitative Assessment

To inform each Challenge project-specific evaluation design, conduct an initial qualitative assessment to assess the project context, form a hypothesis of expected impact, examine the expected heterogeneity of impacts, identify external factors that might inhibit or boost impact, and meet with key stakeholders to secure their cooperation and buy-in.

2. Baseline

During the baseline stage, gather information on market for the technology, its adoption, initial engagement of potential competitors, as well as smallholder farmers' livelihoods and agricultural practices.

3. Endline

During the endline stage, gather information on competitor activities, development of the technology market, and smallholder farmer adoption and income gains.

4. Cost-Effectiveness Analysis/Synthesis

At the end of AgResults, synthesize the findings across Challenge projects to draw lessons on the design and implementation of prize competitions. Assess AgResults’ cost effectiveness in achieving key outcomes across projects and also as compared to traditional approaches.

5. Sustainability Analysis

For key projects, evaluate the sustainability of AgResults’ impact by assessing private sector engagement in technology provision and the continuation of favorable development outcomes two years after the project ends.

Find out more about the Learning Library.