January 30, 2020
By Tulika Narayan, Abt Associates
As the External Evaluator for AgResults, Abt Associates uses a mix of quantitative and qualitative methods to determine if the AgResults Pay-for-Results prize competitions achieve their objectives. Tulika Narayan serves as the Research Director.
Pay-for-results (PfR) approaches are gaining traction with donors both because they pay only if results are achieved and because they offer the possibility of channeling significant private sector resources into development. One type of PfR approach is the “Challenge Project,” which pits multiple private sector actors in a race to win prizes awarded based on their performance.
AgResults, a $145 million multi-donor initiative, is running a series of such competitions, which award prizes to private sector actors for achieving agriculture development objectives: In Nigeria, competitors won prizes for every metric ton of AflasafeTM-treated maize they procured from smallholder farmers. (AflasafeTM is a biocontrol product that addresses cancer-causing aflatoxins that contaminate maize and other crops in humid climates.) In Kenya, competitors shared prizes in proportion to the volume of improved on-farm storage capacity they sold to smallholder farmers. AgResults has an ongoing competition in Vietnam with grand prizes and proportionate prizes for competitors who increase farmer adoption of low-emission, rice-production technologies.
In designing these Challenge Projects, an important decision is determining the size of the prize. Prize sizing usually focuses on the potential competitors: What prize amount would incentivize competitors to engage in the new market? As the External Evaluator, we argue that an additional consideration should be included if the donors want a positive net return on their investment: Prizes should be based on expected benefits. The prize needs to be higher than what would incentivize private sector to engage in the market, and lower than the expected benefits minus other program costs. In other words, the prize should be such that the return on investment is positive. Ideally, the return on investment should be greater than other alternative approaches to achieve the same goal. If competitors appear to be motivated only by a prize much higher than the expected net benefits, a PfR project may not be the most cost-effective way to achieve the development objective. A comparative cost-effectiveness analysis can assess if traditional approaches, such as grants, are a better strategy.
Let us consider the recent AgResults Nigeria project to assess whether the prize amount allowed the donors to generate a positive return on investment.
Did the benefits outweigh the costs in Nigeria?
Yes, they did. In Nigeria, considering the total cost of the program and the scale of adoption by the end of the third year (when the impact evaluation was completed), the AgResults competition cost $85 for every $100 increase in annual net maize revenue per farmer. This does not include the project’s primary expected benefit: better health from avoiding the consumption of aflatoxin-contaminated maize by farmers and other consumers. This implies that the project had a positive return on the donors’ investment and the prize amount was reasonable. If adoption of AflasafeTM continues and farmers continue to benefit, the return on donor investment will be even higher. In fact, creating a sustained market for the technology is one of the key expected benefits of challenge projects. This highlights the importance of considering the timing of the benefits and analyzing what works best for the appropriate time period. In some markets it may be reasonable to expect that the benefits begin to accrue much farther out in the future. These future returns need to be accounted for.
How can program designers define prize value before the Challenge Project begins?
Before defining the prize value, program designers must estimate the total benefits after accounting the number of expected beneficiaries. This estimate helps set the prize amount given the associated project management and verification costs. Understandably, conducting the analysis in advance means tackling uncertainty in the expected benefits. It is important to recognize if there is significant uncertainty up front. Such awareness can also provide a basis for defining minimum thresholds before the prize is paid out to ensure a positive return on investment or value for money.
Let us consider the AgResults Vietnam example: The Vietnam project anticipates that encouraging 16,000 farmers to apply rice production technologies that reduce GHG emissions and increase rice yields is an achievable goal. It expects to pay out $3.3 million in prizes over four years. Adding in management and verification costs of $4.6 million, the total project cost is about $7.9 million. The expected benefit of adopting low emissions technology ranges from $29 to $302 per farmer annually depending on assumptions about area under cultivation per farmer (averaging 0.25 hectares1), expected income increase from rice cultivation per farmer (5% to 30% increase over annual rice income of $380 per farmer2), expected reduction in GHG emissions (1 CO2e MT/hectare to 18 CO2e MT/hectare per year3) and the social cost of carbon ($42 per MT CO2e).
After discounting for returns that accrue later at a 12% discount rate, the total benefit of the project will range from $942,000 to $17.3 million by the end of the two-year implementation phase of the project. Since AgResults expects project management and verification costs at $4.6 million, the project will not yield a positive return at the lower end of the range of benefits. However, if the benefits are at the high end of the range, then the project will yield a positive return. As a benchmark, in the first rice season of Spring 2019, the average reduction in GHG emissions was 0.6 CO2e MT, which is close to the lower bound for GHG emissions.
As mentioned above, an expectation for AgResults projects is the sustained use of the technologies. To account for these benefits, one needs to make forward-looking assumptions on the scale of adoption: will the scale of adoption increase, decrease, or remain the same? Ideally all scenarios should be evaluated to help benchmark the prize. As an example, in Vietnam if the 16,000 farmers continue to adopt the technology for a total of 20 years, the present value of benefits would range from $7.1 to $73 million. At its current $7.9 million cost, the project would not yield positive returns in the lower range but could afford a very large prize if the benefits are in the higher range.
Alternately, a prize of $2.5 million (instead of $3.3 million) would have ensured that even in the low scenario, the project yields a positive return. If a higher prize value is needed to attract competitors, defining prize value and parameters that set a minimum threshold before prizes are paid may address the possible negative return on investment. This could be set by defining a minimum threshold of total emissions reduction achieved, or the number of smallholder farmers reached by competitors or both at which benefits are greater than cost. Most importantly, we argue that assessing the expected benefits, and the uncertainty therein, is critical before program designers define the prize value.
1. USAID Analysis of Investments for achieve Low Emissions Growth rice survey in Vietnam, 2013.
2. USAID Analysis of Investments for achieve Low Emissions Growth rice survey in Vietnam, 2013.
3. At the lower end the emissions can in fact increase. Assuming that all technologies result in a reduction, we have used 1 CO2e MT/hectare as the lower bound.