June 2, 2017
Randomized Control Trials (RCTs) are the most rigorous way of identifying the impacts of a program. RCTs of the AgResults pilots allow us, the External Evaluator, to compare “affected” and “unaffected” farmers – randomly selected groups that are similar except for whether they were affected by the AgResults intervention.
During our initial design trip to Nigeria, we found a unique opportunity to apply a RCT to evaluate the pilot. The implementers—maize aggregators—had plans to scale up their work in villages in phases, adding a few villages each year. When we asked them how they planned to decide which villages to work in each year, they did not have a specific plan. We asked them if they would like to use a lottery to determine the phasing, and they liked that because it helped them address concerns of favoring one village over the other. Therefore, we proposed an RCT evaluation design and randomized which villages would join in the first, second, third, and fourth years, with plans to use the fourth year villages as our control villages (see Figure 1, below). (More detail about the original design is posted on RIDIE.)
Figure 1. Village Randomization Plan
Recognizing the complexity of ensuring the implementers’ adherence to the RCT design, we did not want the success of our work to depend solely on the successful protection of the control villages. Therefore we planned for a back-up quasi-experimental design – the next most rigorous method – and identified some comparison villages in Katsina state -- a geographic area where AgResults did not plan to go, but which is similar to the treatment area. This design was also applicable to one implementer who had different phasing plans and could not participate in the RCT.
In late 2013, the evaluation team worked with the RCT maize aggregators to finalize the village lists. In 2014, the maize aggregators began working with the first randomly selected cohort. In 2017, the last cohort of randomized villages was phased into the maize aggregators’ AgResults effort. What happened between 2014 and 2017 that allowed us to judge the fidelity of the RCT?
Monitoring data collected each year since initial random assignment suggested the maize aggregators’ plans evolved dynamically over four years. Two of the original maize aggregators stopped participating in AgResults. The remaining four maize aggregators found unanticipated constraints and opportunities, and deviated from their randomization plans. Even the implementer who participated in the quasi-experimental design from the start achieved only half of his implementation plan. What proved difficult?
First, village recruitment was not as easy as expected for the implementers. For example, the costs for maize aggregators to work with villages depend on proximity and security, so implementers kept changing their plans as security problems arose in different areas. The business opportunity for maize aggregators in different villages also varies, for reasons including droughts, prior business relationships, and availability of government infrastructure. As external evaluators, we anticipated that all of these factors would shape the maize aggregators’ initial list of villages they planned to go to over four years; however, maize aggregators often adjusted their plans due to new or altered circumstances. Moreover, maize aggregators found revenue-driven advantages from deviating from the RCT plan. By working in villages that offered better business prospects, maize aggregators had a better chance of increasing the output of Aflasafe-treated maize. As a result, our RCT did not work – in the treatment group, far fewer villages were treated than expected and some of the control villages became contaminated (i.e., were treated in the early years despite plans for them to be left untreated until the fourth year).
Given these findings, we are now completing the Nigeria evaluation using a quasi-experimental design using information from our comparison area in Katsina. We are comparing results in the comparison area to results in a similar set of treated villages. These results will be available later in 2017.