Lessons Learned Article 5: The Verification Trade-Off in Pay-for-Results Financing

Home
Lessons Learned
Lessons Learned Articles
Lessons Learned Article 5: The Verification Trade-Off in Pay-for-Results Financing

By Parasto Hamed and Justin Kosoris

Background

Post-harvest losses present a significant challenge for Sub-Saharan African smallholder farmers (SHFs). Each year, an estimated 10 to 20% of grain produced in East and Southern Africa, equivalent to $1.6 billion in value, is lost due to poor harvest and post-harvest practices.1 In countries like Kenya, SHFs lack access to and awareness of affordable on-farm storage devices, a situation that leads SHFs to sell their production immediately after harvest when market prices are the lowest due high supply. With the ability to store grain safely and with reduced losses, SHFs would be empowered to make better decisions about when to sell their grain, increasing food security and income potential and providing motivation to increase production. The private sector has not shown much appetite to change business practices to increase sales to SHFs due to low awareness and perceived low demand in that market.

     Lessons Learned

  • The Kenya Pilot’s theory of change hinges on SHF’s adoption of on-farm storage devices, and required detailed reporting that proved impossible for implementers to provide. 

  • Subsequent increases in verification requirements involved a tradeoff between complexity and cost, and in retrospect could have been minimized through more realistic assumptions regarding end user profiles of sold devices.

  • Requiring post-launch product testing in the Eastern Region delayed activities and reduced the number of implementers participating in the pilot.                           

     

Designing a Pilot to Address the Post-Harvest Storage Challenge.

The AgResults Kenya On-Farm Storage Pilot attempts to address the challenge of reducing post-harvest loss by incentivizing private sector actors working in Kenya to develop, market, and sell new or redesigned on-farm storage devices to SHFs, or those cultivating five hectares or less. Under the AgResults pay-for-results, or “pull mechanism”, model, monetary awards are given to participants that achieve predetermined results. In the Kenya Pilot, AgResults based the distribution of awards on the cumulative storage capacity sold by the private sector companies to SHFs.


The pilot has three objectives aligned to AgResults’ overall goals:

  • Improve the economic welfare and food security of SHFs by reducing post-harvest loss by using improved on-farm storage devices.

  • Develop a sustainable long-term market solution.

  • Test a new model of incentivizing the private sector to meet SHF storage needs with goods and services during the pilot and in the future.2

      

The pilot operates in two regions in Kenya: 1) The Rift Valley, which produces 60 percent of the country’s maize; and 2) the Eastern region, the third largest maize-producing region in the country with significant post-harvest losses due to the high presence of Larger Grain Borer (LGB) and other grain-damaging pests.3 These two regions were chosen because of the immediate impact the devices could have on post-harvest loss and because of the presence of agro-dealers. In both regions, most smallholder farmers are on average 7km away from the closest agro-dealer, making these regions attractive to begin operations due to proximity.4 By monitoring only two regions, the external evaluator could use the adjacent areas as counterfactuals to track devices sold and compare the level of grain loss using enhanced storage devices to grain loss using traditional storage devices.5

    

In each region, to qualify for the pilot’s prizes, the participating private sector companies (“implementers”) must sell the equivalent of 21,000 metric tons (MT) of adjusted storage capacity6 to SHFs. Eligible storage devices cannot exceed a capacity of 540kg, to better address SHF needs; the capacity requirement assumes that each family member consumes 90kg of grain per season and 540kg would satisfy a family’s needs.
The pilot offers two prize structures, one for each target region:

  • In the Rift Valley there are two prizes: the mid-point prize and the end-of-pilot prize. The mid-point prizes of $750,000 are awarded to the first five implementers that reach the threshold of 21,000 MT of adjusted storage capacity sold to SHFs. The end-of-pilot prize is a proportional distribution of $1,000,000 split amongst all implementers that reach the 21,000 MT threshold. 

  • In the Eastern Region, there is one end-of-pilot $3,000,000 prize awarded proportionally to all implementers that surpass the 21,000 MT minimum threshold. The Eastern Region has only a single prize due to the time required at the start of the pilot to test devices for effectiveness against LGB. 

The adjusted storage capacity seemed like a clear and verifiable indicator on which to base payments and avoid potential fraud. However, as learned over time, basing payments on sales to SHF complicated the verification process and required several iterations of the verification design, as well as experiential learning, to arrive at the right level of verification. The AgResults Kenya experience has provided valuable insight on how seemingly simple and reasonable-sounding decisions during the design stage can greatly influence implementation of a prize contest, as well as the right level of verification to be reasonably sure of final outcomes.

    
Kenya Pilot Verification Requirements

All AgResults pilots employ some form of third-party verification that assesses the production or sales of a technology, product, or device by each implementer against the requirements laid out in the pilot’s rules. By verifying results using clear rules and requirements, donors can ascertain the results of the contests while at the same time dissuade fraudulent behavior. However, the more rigorous a verification scheme is, the greater the cost. The Kenya Pilot required two aspects of verification: proof of efficacy of the on-farm storage technology (in the Eastern Region), and proof of sales of unsubsidized on-farm storage devices to SHFs.

    
The first aspect of verification, the efficacy test, was a central component for Eastern Region eligibility in the original business plan. However, a key assumption, that the Kenyan Bureau of Standards or another similar body had preexisting standards from which AgResults could develop an efficacy test, proved incorrect. During pilot start-up, AgResults determined that there were no Kenyan or other standards for LGB-proof on-farm storage devices. This meant AgResults would need to design and create its own set of testable standards that all implementers would accept. The LGB test provided two outputs: first, to determine whether storage devices prevent internal insect damage to grain during a six-month storage period in Eastern Region conditions; and second, to determine whether LGB could penetrate into storage devices from outside. The LGB test was the only way an implementer could become eligible.

    
The second aspect of verification involved sales to SHFs. The Kenya Pilot’s theory of change hinges on smallholder adoption; therefore, the pilot’s indicators are not only based on unsubsidized sales within the two pilot regions, but also that those sales are traceable specifically SHFs. Participating companies must sell devices to farmers at a price that recovers the cost of production to avoid dumping and that incentivizes companies to establish a sustainable business model that would continue after the pilot ends. A maximum size limit of 540kg per device was also set with the assumption that SHFs would find these devices cost-effective, but that large-scale farms would find them inappropriate for their volume needs.

    
To verify sales and determine if they meet the pilot’s requirements, AgResults contracted a sales verifier to audit sales reports submitted by each implementer. For sales to qualify under the Pilot, companies must provide certified data including the region in which the sale took place and that it was purchased by a SHF. This information is provided to the verifier to corroborate the information and ensure no fraudulent activities have taken place. Sales audits are conducted every four months with scheduled on-site visits to review sales ledgers, inventory and bank statements as well mystery shopping to confirm the price of the devices and spot checks of sales to confirm the location of sales.

     
Verification Challenges

LGB Penetration Test - Storage devices inside their Perspex boxes

In theory, the sales verification process should provide the pilot with the verification needed to ensure the legitimacy of the sales and confirm smallholder adoption, but in practice, it has proven more difficult. In the original business plan, the pilot design consultants7 made the assumption that companies would be able to track device sales down to each SHF who purchased a device, and that companies would provide a name and mobile number for verification purposes. During initial implementation, it became clear that while companies sell devices through diverse and multi-tiered distribution networks, including wholesalers, retail outlets, and agro-dealer networks, companies had no ability to trace sales beyond their first point of sale. To ask otherwise in such a complex network would require adoption of expensive and sophisticated sales tracking systems that would be too burdensome for companies and their networks to adopt.

    

AgResults quickly learned that while companies were receptive to improvements to their tracking systems, it became clear that the original verification scheme pushed companies outside of their standard operating model and made implementers hesitant to join the Pilot. As a result, AgResults worked to develop proxy methods of estimating the eventual number of sales to SHFs based on population demographics in target counties. However, due to the lack of required information including farm size in the national census, the pilot did not have a way to determine the percentage of SHFs in the target counties. Subsequently AgResults decided to implement a series of surveys to determine the percentage of SHFs and the market penetration of sold devices to be confident that all sales counted towards the prize thresholds actually went to SHFs.

    
With regards to LGB, AgResults determined the need to develop new standards for the LGB test only after the pilot launched and implementers had joined. The test began six months after the first implementers joined the pilot and lasted for six months. During that period, although implementers could sell devices during the testing phase, most implementers waited for the results before engaging fully in the competition to reduce their risk in case of test failure. The test thus reduced the amount of time implementers had to sell their devices and reach the minimum sales threshold. Another drawback of the LGB test was that its expense and complexity meant AgResults could only run it once, eliminating future implementers from joining the Eastern Region completion later in the pilot. The pilot admitted three new implementers after the LGB test, but only to the Rift Valley contest.

    
Response to Challenges: Verification Surveys

AgResults adjusted the verification scheme to include three new surveys: 1) a randomized rural household survey (“rural survey”), 2) a mid-point market share survey (“mid-point survey”), and 3) an end-of-pilot market share survey (“end-point survey”).

  • The Randomized Rural Household Survey would establish the proportion of SHFs residing in each target county. The survey would also estimate the number of pilot-eligible products purchased by SHFs and would inform sampling for the mid-point and end-of-pilot surveys.

  • The Mid-Point and End-of-Pilot Market Share Surveys were large surveys designed to obtain specific data on the number of SHFs who purchased the AgResults storage devices. The survey sample size would allow AgResults to confirm the previously verified sales actually went to SHFs. These surveys would move the burden of end-user tracking and reporting from the companies to the Pilot.

In July 2017, AgResults hired a consulting firm to conduct the rural and mid-point surveys. The rural survey, completed in December 2017, found most notably that AgResult’s working, albeit conservative, assumption of 70% of SHFs as a total population was low. According to the survey results, 96% of the devices sold in both Rift Valley and Eastern regions were to SHFs. Of those devices, only 5% were giveaways or subsidized, meaning that 95% of the devices qualified towards the prize thresholds. Therefore, the updated adjusted purchase rate for SHFs in both Rift Valley and Eastern Regions is 91.2% (i.e. 96% X 95%). This percentage (91.2%) applies to all verified sales to calculate the adjusted storage capacity sold to SHFs, and to track each Implementer’s progress against the 21,000 MT threshold targets.

    
Enumerator interviewing respondents in Sikhendu sub location, Trans Nzoia CountyAdditionally, the results reinforced the original assumption that the subset of on-farm storage devices that qualify for the Pilotdonot suit the needs of medium and large-scale farmers. AgResults learned that the counties that had the highest percentage of medium- to large-scale farmers saw the fewest devices sold, and the counties with the highest percentage of smallholders saw the most devices sold. The report showed that fewer than 4% of devices were sold to non-SHFs, including people with no access to land, providing more evidence that the devices were not of interest to non-SHFs. 

Based on the results of the rural survey and the five sales audits that had taken place to date, AgResults determined that the pilot had enough evidence to assert that SHFs were the buyers of the target devices. AgResults therefore decided to cancel the mid-point and end-point surveys.

   

   
Conclusion

The goal of the Kenya Pilot is to promote effective on-farm storage devices that meet the needs of SMFs in Kenya. To measure that SHF impact, the Pilot required additional verification that proved to be expensive and inefficient. In retrospect, if the design were simplified from the start and simply counted all sales of SHF-friendly on-farm storage devices, the verification process may not have hindered implementation and AgResults may have increased the scale and reach of the Pilot. Despite the verification challenges, the Pilot has achieved notable success in incentivizing new and improved on-farm storage distribution networks geared towards the SHF market in Kenya. Future learning articles will explore changes in networks and business models in response to the pilot incentive.

   

Lessons Learned and Recommendations

The criterion to award prizes based on sales of on-farm storage devices to SHFs added a layer of verification that proved impossible to require of implementers and in hindsight was not necessary. The business plan assumed the implementers collected end-user data and would be easy to report if a SHF purchased their device. AgResults quickly learned that implementer only tracked sales to the first point of sale and do not have the capacity or systems in place to track beyond that point. To overcome this barrier, AgResults added a new layer of costly verification that included three surveys to compensate for the lack of implementer tracking capacity. However, in the end the Pilot found that the initial rural survey was enough to support the existing verification scheme, and was able to avoid moving forward with the two larger, more costly market share surveys.

    
The Kenya Pilot’s theory of change focuses on SHF adoption of on-farm storage devices, which increased verification requirements. The Pilot defined eligible sales as purchases made explicitly by SHFs under the assumption that the private sector actors would be able to provide sales information on the end-user. The assumption proved to be false, adding a more complex layer to verification and increasing the overall management cost of the pilot.
Recommendation: When designing a pilot, prize verification schemes should be carefully structured to only require information requirements that the private sector can provide. Simple and straightforward reporting reduces the burden placed on both the pilot and as well as the implementers and can make the pilot more attractive to join.


Verification of the Kenya Pilot involves a tradeoff between complexity and cost. To counter the lack of ability for implementers to report sales at the SHF level, the Pilot added additional verification measures in the form of large surveys. However, given strong baseline data, the fact that allowable devices were tailored for SHFs, the lack of fraudulent results in the sales audits, and the percentage of SHFs in each region, we now realize that the verification surveys were not needed to be confident of results.
Recommendation: Pull mechanisms should focus on increasing private sector investments in providing devices tailored to the target market and assume that target market will be the principal buyer. Simplifying the verification requirements allows implementers to focus on sales, receive prizes earlier, and potentially reinvest incentives during the contest and allow implementers to reinvest their incentives.

    
Requiring additional criteria for pilot participation after pilot launch may delay implementation and reduce the number of implementers. The Kenya pilot found that it would be necessary to require all implementers participating in the Eastern Region competition to pass an LGB penetration test. This test became more complicated and time-consuming than planned, which delayed investing in marketing and distribution of devices. Due to the lengthy testing process and the high cost, the pilot could only conduct one test. Therefore, only the implementers that signed up at the beginning of the pilot were able to participate in the Eastern Region competition, reducing the number of potential pilot implementers.
Recommendations: Pay-for-results prizes should attempt to set clear entry rules and conduct any pre-requisite testing before the contest begins. The testing should be available to later entrants if possible to increase the scope of the competition.

   


1. Missing Food: The Case of Postharvest Grain Losses in Sub-Saharan Africa. The World Bank, 2011, pp 17-18.

2. AgResults Kenya On-Farm Storage Pilot Business Plan, p. 6.

3. In 2013, Kenya replaced its 8 provinces with 47 counties. For the Pilot’s purposes, the counties respectively located within the former Rift Valley and Eastern provinces are considered the “Rift Valley” and “Eastern” regions.

4. AgResults Kenya On-Farm Storage Pilot Business Plan, p. 58.

5. AgResults Kenya On-Farm Storage Pilot Business Plan, p. 24.

6. The adjusted storage capacity per device is determined based on the total capacity of the device multiplied by the number of years the device will provide viable protection for grain.

7. The Kenya On-Farm Storage Pilot Business Plan was developed in 2012 by Dalberg under what was then known as the Agriculture Pull Mechanism Initiative, which was renamed AgResults in 2013.

   

About AgResults

AgResults is a $122 million collaborative initiative between the governments of Australia, Canada, the United Kingdom, the United States, and the Bill & Melinda Gates Foundation to incentivize the private sector to overcome market barriers and develop solutions to food security and agricultural challenges that disproportionately affect people living in poverty. The initiative designs and implements agricultural development prize competitions, also referred to as pay-for-results or pull mechanisms, which are innovative development finance programs that incentivize the private sector to work towards a defined goal to receive a monetary award.

  

About AgResults Lessons Learned Series

One of the primary objectives of AgResults is to better understand how well pull mechanisms work to overcome market failures in agricultural development. The lessons learned series explores AgResults’ experience designing and implementing agricultural-focused pull mechanisms, with the goal of providing key lessons and recommendations that development practitioners should take into account when designing similar programs.

The AgResults initiative is a partnership between:
Connect: