Lessons Learned Brief 1: The Importance of Setting Up a Verification Baseline

Home
Lessons Learned
Lessons Learned Briefs
Lessons Learned Brief 1: The Importance of Setting Up a Verification Baseline

Background

Southeast Asia is a leading global rice production region. However, typical intensive rice crop management practices, especially those designed to increase yields on smaller plots of land with associated high fertilization levels, often result in high emissions of methane and nitrous oxide greenhouse gases (GHGs). The AgResults Vietnam GHG Emissions Reduction Pilot addresses this situation by incenting the testing and widespread use by smallholder farmers (SHFs) of novel tools, products, and agronomic practices that reduce GHG emissions while increasing yields. The Pilot is being implemented in Vietnam’s northern province of Thai Binh and contains two phases:

  • Phase I: Low GHG Technology and Agronomic Practice Adoption: During this period, selected Implementers will test their tool, product, or agronomic technique on controlled plots during two consecutive rice-growing seasons. Prizes based on the ability to increase yields while reducing GHG emissions will be provided at the end of each growing season and at the end of the phase. AgResults will use direct, field-based measurement to determine yield increases and GHG reductions, as well as verify the use of proposed new technologies.
  • Phase II: Scaling of Behaviors, Tools, and Products to Large Numbers of Smallholder Farmers (SHFs): Implementers will work over four consecutive rice growing seasons to increase the number of SHFs adopting successful solutions that lower emissions and increase yields. Prizes will be based on a formula consisting of reduced GHG emissions, increased yields, number of SHFs reached, and repeated use of the technology. Since the scale will be greatly increased, Phase II will rely on remote sensing, mobile data collection, and process modeling to verify implementation of improved practices, quantify GHG reductions, and determine yield gains across rice fields.

The Importance of Verification

A key challenge in the Vietnam Pilot is setting up fair and cost-effective verification. In Phase I, this means verifying results against a defined Performance Baseline (PB) - or set of common rice crop management practices from field preparation to harvest that result in current GHG emissions outputs and rice yields. While SNV, as the Pilot Manager (PM), is tasked with overseeing field implementation, AgResults has also engaged Applied GeoSolutions (AGS) as the Verifier for the Vietnam Pilot.

A prize award based on improved yields and reduced emissions must set a clear and realistic PB for “normal” rice crop management practices. This is important for several reasons:

  • A baseline with low standards would make it easy for any implementer, including those using “new” technologies that are actually relatively standard, to achieve results relative to that baseline, thus incentivizing continuation of current practices.
  • A baseline with standards based on uncommon or site-specific practices might benefit select implementers who are already practicing those “improved” practices, thus dis-incentivizing competition and trial of new technologies. 
  • Although Phase II will rely on remote sensing as the cornerstone of verification, Phase II implementers must have proven their technologies in Phase I. Therefore it is crucial to the overall success of the Pilot that the baseline is set correctly before Phase I.

Setting the Performance Baseline through Household Surveys

To set the PB, provided as an attachment to the Pilot Request for Applications (RFA) to enter the contest, the AgResults Secretariat and AGS worked to finalize a Verification Design document that outlines the specifics of the Verification, including setting the PB using a household survey to capture the key rice crop management practices that influence GHG emissions and yield potential. These include:

  • Fertilizer use: type, application rates, timing, and methods
  • Rice husk and straw residue management: fraction removed from the field, fate and timing of incorporation
  • Water management: continuous flooding, mid-season drain for applications of agrochemicals and pesticides, “Alternate Wetting and Drying” 
  • Tillage practices: frequency, timing, and depth 
  • Organic amendments: type, amount applied, timing of application
  • Rice varieties: growth duration and planting density 

AGS, working with Vietnam’s Institute for Agricultural Environment (IAE), developed and implemented the 720 household survey across Thai Binh, randomized on production area and soil type. In addition to the management and cost information, which we collected for both spring and summer rice production systems, the survey also accounted for a range of biophysical conditions, including major soil types for each of the eight Districts in Thai Binh, as well as relative elevation gradients (sorted into high, medium and low).

     

Baseline Analysis and Determination

The baseline survey results provided, for the most part, clear indications of those management practices that could be considered “standard”. In certain cases, there were circumstances that merited additional analysis. If the baseline were set using averages, then the job would be simple. However, one cannot average out most agriculture practices. Just as thinking of the average family size as including 2.3 children in terms of real-world application is challenging, similarly a farmer cannot drain their field or apply fertilizer 2.3 times. We present some examples below:

  • “Competing” standards: In some cases, the survey produced “competing standards”, or practices that were closer to an even split of respondents that merited additional analysis to determine the overall baseline. For example, the split between respondents direct-seeding versus transplanting rice shoots was 40-60. As a result, the baseline stipulates the use of both as acceptable, but importantly will not allow implementers to propose a switch from one to the other as a basis of their “improved” technology.
  • Setting tougher yield standards: For repetitive-type practices like fertilizer application, there was greater variability than for practices with two potential outcomes. In those cases, the prevailing outcome was to set a standard that incentivizes a tougher competition with respect to increasing yields. For example, for fertilizer application, although most farmers applied two applications, 40% of all farmers were applying nitrogen fertilizer at least three times (see chart above). Therefore, since it is considered somewhat common and feasible, the baseline has been set at three applications. A similar histogram analysis produced the water drainage baseline practice.
  • Incentivizing best practices: Incorporation of rice straw into fields is a major contributor to GHG emissions. However, the full removal of straw may hurt field health in the end due to reduced organic matter. While most farmers reported not incorporating rice straw back into the field, there was a significant proportion that did report this practice. One could make the argument that incorporation of straw is a relatively common practice. If we set the baseline to include straw incorporation, it would be an easy “technology” to simply remove the straw and thus most likely achieve GHG reductions, but setting a baseline that does not include straw incorporation might deter implementers from proposing that practice due to the GHG emission potential. Therefore, we set stipulations for the baseline for straw incorporation that depend on the set of practices that each implementer proposes – if implementers propose incorporating straw as part of their technology, then they will be assessed against a baseline that also incorporates straw, and vice versa. Thus, the PB will not create a disincentive to incorporate straw, while still allowing flexibility for fair judgement of non-incorporative solutions that focus on other aspects of field management.

Stakeholder Buy-In

One of the risks in conducting surveys is response bias – or receiving inaccurate or untruthful responses based on what the respondent thinks is common or the interviewer wants to hear. To mitigate this risk, AgResults sought input and buy-in on the PB from multiple stakeholders outside of the Secretariat, Verifier, and Pilot Manager. Stakeholders included Ministry of Agriculture and Thai Binh provincial government representatives. During the stakeholder consultations, important considerations emerged, such as defining multiple baseline situations to allow for up to three categories of typical rice varieties as well as direct sowing versus transplanting. These refinements will help increase accuracy in measuring the change in GHG emissions and help reduce the risk that certain implementers come into the contest with an inherent advantage. They have also taken into consideration the potential for response by the private sector organizations who would respond to the RFA that includes the PB.

   

RFA Response

The novel nature of the prize contest was responsible for much of the initial questioning that SNV received from prospective applicants. Based on the initial outreach to prospective applicants, AgResults decided to hold RFA workshops to go over the contest rules and the PB. The workshops proved critical in increasing applicants’ awareness of the process and the PB against which they would be measured if accepted into Phase I. Applicants raised concerns about the potential transparency of control plots, but through the workshops and other direct outreach SNV was able to assuage these concerns. As a result, 24 prospective implementers applied, of which 11 were accepted into Phase I.

     

Implementing the Phase I Baseline

Initially, the Phase I contest rules required implementers to run their improved technology plots alongside an implementer-controlled PB plot. However, after continued discussions between AgResults and local stakeholders during the setting of the PB, AgResults made a change to the rules published in the RFA. The Verifier is now directly managing the PB control plots against which to judge all implementers for reduced GHG emissions and increased yields. Each control plot is located as close as possible to the corresponding Implementer’s testing plot and is on land with similar agro-ecological conditions. AgResults made the change to reduce fraud and the chance of neglect of control plots by Implementers.

The relatively novel GHG verification protocol has required other adjustments during the lead-up to Phase I. One adjustment concerned the GHG measurement chambers, which sit in the fields as an open-top box and are closed 24 hours in advance of each sample taken from the box’s collected gas. Before the RFA, AGS envisaged the boxes as measuring 45cm by 40cm at the base, encompassing four rice hills. However, variable spacing proposed by Implementers has led to differing opinions on the proper dimensions to capture accurate and fair GHG measurements across all fields. AGS and IAE consulted with three international experts, proposing either a larger single chamber size or a variable size that would capture four hills per Implementer field. In the end, the Verification team settled on a variable chamber size that will increase consistency in measurements.


A second decision relates to the frequency of sampling versus the number of chambers. The original plan called for three chambers per field, with samples kept apart from each other. A new proposal calls for each field’s samples pooled during each distinct measurement event. Pooling allows us to increase the number of sampling events for the same cost, therefore increasing the accuracy of the overall GHG measurements within each field. The consulted experts validated this concept while noting that the verification design still has a weakness, as there are only one test field and one control plot per implementer. In case of a major weather event or other unplanned external factors that ruin one or more field tests, AGS can use its modeling system to estimate GHG reductions and yields based on inputs. However, this is a last resort as Phase I tests are designed to provide critical inputs that will calibrate those models for accuracy in Phase II.


The timing of the Pilot approval and RFA has given all parties limited time to set up test and control plots. For instance, IAE has faced challenges in quickly finding suitable plots and local farmers to run the plots, although the participation by the provincial government’s extension service has aided this recruitment significantly. The teams are also adapting their communications, with SNV playing an important role in pushing the Verifier to meet deadlines and report more regularly. We expect that Phase I’s second cropping season will be smoother and will incorporate all of what we have learned so far in the run-up to Phase I.

     

Lessons Learned and Recommendations

The Pilot is still in early stages, so broad lessons must wait. However, based on the above-described process to set the baseline, we can recommend practices based on lessons gleaned from the development of the baseline for the first cropping season. These initial baseline-focused lessons and recommendations for future cropping seasons are in the box below. As implementation continues, the Pilot will continue to test its assumptions made in early implementation to determine what changes are needed for the next Phase I crop in Spring 2018.

Recommendations 

  • Use objective data, but cast a wide net: AgResults carried out a baseline survey to avoid potential bias that would arise from using site-specific surveys. While this baseline provided mostly clear recommendations for baseline practices, we did encounter some areas for which multiple standard practices exist. By allowing multiple baseline practices depending on the Implementer, we still rely on objective data but allow for some site specificity.
    • Recommendation: Performance baselines for pull mechanisms dealing with field-based technologies should use objective data sources, but should also allow for variability based on relevant biophysical characteristics or several competing “standard” practices.
  • Groundtruth to validate data: Once defined, the baseline benefited from additional adjustment due to stakeholder consultations. This stakeholder buy-in helped AgResults finalize the baseline through confirming the validity of the survey data as well as recommending adjustments that we had not considered, such as increasing baselines to include multiple typical rice varieties.
    • Recommendation: Those developing a similar baseline should consult appropriate local expert stakeholders to reduce the risk of inaccurate data based on bias or untruthful responses, as well as provide important perspectives not previously seen.
  • Be adaptable and flexible to adjustments that will be required on the ground: Agriculture is and art and science, which is evident in how quickly we have had to adjust to actual implementation realities. Adjustments to hold RFA workshops to encourage applications as well as verification adjustments to set chamber size and measurement protocols, while unplanned, have proven vital in the initial success of the first cropping season. 
    • Recommendation: A similarly designed pilot is doomed to fail without allowing for adaptability. Given realities on the ground and reactions from the private sector, flexibility in the design of verification plan is crucial to allow for success. Those running similar programs should have clear protocols for making and instituting these quick-response actions, including a small but always available decision-making group with the authority to make the changes.

More Resources on the Vietnam GHG Emissions Reduction Pilot 

Download The Importance of Settling Up a Verification Baseline Here

Download the Vietnam Pilot one-pager here

AgResults Celebrates Kickoff of Phase I of the Vietnam GHG Emissions Reduction Pilot with Workshop and Field Visits

Phase I of the Vietnam GHG Emissions Reduction Pilot Surfaces Innovative Tech for Reducing GHG Emissions in Rice Farming

Multi-donor initiative launches a multi-million dollar prize competition to reduce GHG emission in Vietnam

From Our Blog: Confidence and Excitement Regarding Progress in the GHG Emissions Reduction in Vietnam Pilot - Reactions from Vietnam Pilot Field Coordinator, Justin Kosoris

    

About AgResults Lessons Learned Series

AgResults is a $122 million collaborative initiative between the governments of Australia, Canada, the United Kingdom, the United States, and the Bill & Melinda Gates Foundation to incentivize the private sector to overcome market barriers and develop solutions to food security and agricultural challenges that disproportionately affect people living in poverty. The initiative designs and implements prize competitions, also referred to as pay-for-success or pull mechanisms, which are innovative development finance mechanisms that incentivize the private sector to work towards a defined goal. AgResults has two primary objectives: 1) to overcome market failures in agriculture through provision of results-based economic incentives; and 2) to test the effectiveness and efficiency of pull financing.

      

One of the primary objectives of AgResults is to better understand how well pull mechanisms work to solve market failures in agricultural development. The lessons learned series explores AgResults’ experiences in designing and implementing pull mechanisms, with the goal of providing key lessons and recommendations that development practitioners should consider before designing agricultural-focused pull mechanisms.

The AgResults initiative is a partnership between:
Connect: