External Evaluation: Can we evaluate tech development for smallholders?

External Evaluation: Can we evaluate tech development for smallholders?
External Evaluation: Can we evaluate tech development for smallholders?

External Evaluation: Can we evaluate tech development for smallholders?

October 11, 2016

AgResults’ Brucellosis Vaccine Pilot is the only AgResults pilot that focuses solely on technology development; the other five offer prizes for spurring technology adoption. Evaluating an initiative’s impact on technology adoption is easier because we can survey individuals or households to estimate adoption rates and adoption’s spillover effect on social and economic well-being. But what can we measure to evaluate a technology development prize? Is there anything to learn, other than whether or not the technology was developed? It turns out there is!

Currently, there is limited information about the impact of the hundreds of innovation prizes that have been offered in recent years, as only a fraction of them have been systematically evaluated.[i] That is, we do not know if observed solutions to problems targeted by innovation prizes resulted because of the prizes or if they would have been developed without prizes over a similar time period. Neither do we know if innovation prizes are a cost-effective way to produce these solutions, relative to standard government contracting, which pays entities to undertake specified activities without tying compensation to results.

Abt Associates recently developed an evaluation framework for the Brucellosis Vaccine Pilot that seeks to contribute to this learning. The evaluation framework presents methods and outcomes that we will use to answer the following research questions about the Brucellosis prize’s performance, impact, cost, and lessons learned:

The Brucellosis Vaccine Pilot design assumes that the prospect of winning the prize will spur potential solvers to invest in R&D activities, which will lead to successful vaccine development. To understand if this is the case, we will conduct a qualitative assessment to explore why solvers choose to participate and remain engaged in the pilot in comparison to capable entities that choose not to participate. We will also use qualitative methods to explore whether the vaccine, if created, is likely to be well-suited for and adopted by small-scale livestock producers in developing countries.

The Pilot’s chief aim is development of a particular vaccine that satisfies pre-specified criteria. To help assess if the prize in fact spurred the B. melitensis vaccine’s development, we will compare its development progress to the development progress of vaccines for a set of diseases somewhat similar to Brucellosis in terms of (1) characteristics of the disease (e.g., affecting small mammals, communicable to humans), (2) being “neglected” with limited vaccine R&D activity, and (3) not being the subject of a pull mechanism prize in the same time period. Solvers’ attempts may also yield notable innovations short-of, beyond, or tangential to the specified B. melitensis vaccine. We will assess these innovations by reviewing public documents and conducting interviews with participating solvers, potential solvers who chose not to participate, and other stakeholders such as academics and non-governmental organizations facilitating vaccine development. 

Another premise for this pilot is that an innovation prize is a cost-effective way to spur development of a B. melitensis vaccine that can be successful in rural areas of developing countries. We will examine the cost-effectiveness of the pilot by comparing the costs of competition governance and the prize awards to the projected number of small ruminant infections expected to be averted—and the expected cost savings to society of that result.

Similar to the evaluation of all AgResults pilots, the chief aim of the Brucellosis Vaccine Pilot evaluation is to learn about the effectiveness of pull mechanisms. By learning and sharing what works and what does not work, evaluations of technology development prizes intend to help sponsors get better at designing and implementing prizes to achieve development outcomes.  

The AgResults initiative is a partnership between: