Learning is at the heart of AgResults

This brief by External Evaluator Abt Associates discusses quantitative methods for conducting impact assessments of smallholder farmer benefits from donor-devised interventions that stimulate private sector engagement in markets for technologies benefiting the farmers. The brief presents three AgResults Pay-for-Results competitions as examples and offers seven lessons for evaluators.

This report from External Evaluator IDinsight documents findings from the baseline Process Tracing evaluation of the AgResults Senegal Crop Storage Finance Challenge Project. The purpose of the baseline Process Tracing evaluation was to assess the state of the storage finance market during early stages of the project and assess if the preconditions are in place for the project to have its intended impact.

This evaluation final report by Abt Associates shows that Pay-for-Results spurred private sector development and dissemination of emissions-reducing, yield-enhancing technology packages among smallholder farmers in Vietnam. A 2100-farmer survey showed that AgResults farmers achieved 14 percent higher yields than comparison farmers and used practices associated with lower emissions. However, project measurements of emissions were uncertain, so evaluators could not determine emissions impact. Future projects should pay attention to reliability of field-based emissions measurements.

This brief summarizes the final evaluation report of the Vietnam GHG Emissions Reduction Challenge Project, showing that PfR can spur the private sector to develop and disseminate emissions-reducing technology packages for smallholder farmers. AgResults farmers achieved 14 percent higher yields than comparison farmers and were more likely to use practices associated with lower GHG emissions. The project’s main challenge was obtaining reliable measurements of emissions.

This sustainability assessment by External Evaluator Abt Associates looking at the Kenya On-Farm Storage Challenge Project affirms that Pay-for-Results projects can catalyze development of a smallholder- and gender-inclusive, sustainable market for beneficial agricultural technologies. Since the project ended, there have been new entrants to the market, ongoing product innovation, and a 250% increase in sales of on-farm storage products attributable to AgResults.

This report by External Evaluator Abt Associates summarizes independent evaluation findings from competitions in Nigeria, Kenya, Uganda, and Zambia and offers guidance for future prize sponsors. It provides evidence that Pay-for-Results (PfR) competitions can spur development of markets for technologies benefitting smallholder farmers. It also offers guidance on how to determine whether PfR is an appropriate design option and how to maximize farmer benefits.

The evaluation design for the Tanzania Dairy Productivity Challenge Project by Abt Associates lays out a mixed-methods approach. To assess impact on dairy producers, the evaluator will use a quasi-experimental design to gauge technology adoption and subsequent impacts on income and consumption. The approach will use a structure, conduct, performance framework to assess market impact.

This assessment by External Evaluator Abt Associates, updated in December 2020, found that the AgResults project played a role in creating a niche market for PVA maize, although with limited development impact because the market involved mostly better-off smallholder farmers and urban consumers. Findings suggested that the market would likely be sustained in the short to medium term.

This brief summarizes the close-out assessment of the Zambia Biofortified Maize Challenge Project. External Evaluator Abt Associates found that the competition helped develop a niche market for Pro-Vitamin A maize, although Vitamin A-deficient populations were a small portion of those who consumed the maize. The brief underscores the importance of directly targeting intended beneficiaries in project design.

This brief, updated in November 2020, summarizes evaluation findings for the Nigeria project. External Evaluator Abt Associates found that AgResults helped create a niche market for Aflasafe-treated maize and an increase in farmers’ uptake of Aflasafe and net maize income. Market sustainability is uncertain due to unstable market conditions and delays in implementation of aflatoxin-related policies.

The Nigeria Aflasafe Challenge Project evaluation (published in February 2019, end-of-project update added November 2020) found that the project succeeded in creating a niche market for Aflasafe-treated maize. Aflasafe adoption increased by 56 percentage points, and average net income from maize increased by 16 percent. The sustainability of the market is uncertain.

Evaluator Abt Associates revisited the Nigeria Aflasafe Challenge Project at its conclusion, two years after the comprehensive mixed-methods assessment. They found that private sector involvement in the Aflasafe-treated maize market had increased despite reduction in the AgResults incentive, but that the Aflasafe-treated maize market’s sustainability was uncertain given general maize market instability and delayed implementation of aflatoxin-related policies.

This brief by External Evaluator Abt Associates explores how Pay-for-Results (PfR) projects could potentially perpetuate disadvantages that women face in agricultural market systems and suggests project design approaches to improve gender outcomes. It also presents results from AgResults’ Kenya PfR project that led to unique benefits for women.

More than 4.5 billion people in developing countries are exposed to aflatoxins, one of the world’s most carcinogenic substances, through their diets. Yet a biocontrol solution exists: Aflasafe^TM. The AgResults Nigeria Aflasafe^TM Challenge Project (2013-2019) used a Pay-for-Results (PfR) prize competition to motivate the private sector to overcome critical barriers preventing widespread adoption of Aflasafe^TM among Nigerian smallholder farmers working in the maize sector. Over six years, the competition engaged 41 competitors to reach 75,788 farmers, producing more than 315,000MT of Aflasafe^TM-treated maize.

The evaluation design for the Foot and Mouth Disease (FMD) Vaccine Challenge Project, by IDinsight, lays out a mixed-methods approach relying primarily on qualitative process-tracing techniques. The document highlights the role of data collected from potential vaccine manufacturers, buyers, policymakers, and others to explore how the project impacts the development and uptake of FMD vaccines in Eastern Africa.

This brief by External Evaluator Abt Associates summarizes the evaluation of the AgResults Kenya On-Farm Storage Challenge Project. Abt Associates found that the competition created an emerging market for improved on-farm storage devices and increased adoption of on-farm storage technologies. However, impacts on farmer income were small. The brief underscores the challenge of constructing an accurate theory of change.

Abt Associates’ evaluation design of the Vietnam GHG Emissions Reduction Challenge Project envisions a randomized control design. This report assesses the success of the initial randomization and confirms that competitors adhered to rules respecting communes set aside for control; it also confirms similarities between control and treatment groups at baseline.

The evaluation design for the Vietnam GHG Emissions Reduction Challenge Project, by Abt Associates, lays out a mixed methods approach, including a randomized control design. The document highlights the role of interviews, surveys, and farmer diaries to explore the effect of the Vietnam project on the market for emissions-reducing technologies and to estimate impacts on farmers’ knowledge, attitude, practices, and incomes.

This assessment by External Evaluator Abt Associates reflects the experience of competitors in the Brucellosis Vaccine Challenge Project from mid-2018 to mid-2019. Most competitors reported that the competition had helped advance their B. melitensis vaccine R&D. However, many of them anticipated challenges completing the safety and efficacy trials due to resource constraints.

This DFID Chief Economist Seminar Series presentation from February 2020 summarizes External Evaluator Abt Associates' findings about the use of “pull mechanisms” or “prize competitions” to develop markets that provide innovative agricultural technologies to smallholder farmers. Findings suggest that pull mechanisms can create markets as long as the binding constraints inhibiting market development are within the manageable interest of the companies competing for the prize.