The Evidence to Policy Pipeline: How Open Policy Analysis Can Transform Deworming Policy

This post was written by CEGA Director of Operations Lauren Russell, CEGA Project Scientist Fernando Hoces de la Guardia, Evidence Action Chief Engagement Officer Grace Hollister and Evidence Action Manager of Cost-Effectiveness Analysis Grace Morgan. This post can also be found cross-posted on CEGA’s blog.

At CEGA and Evidence Action, we believe that investing in rigorous research — and applying the resulting evidence to the work of government, NGO, donor, and private sector partners — is essential to solving the world’s greatest challenges. In fact, our organizations play critical roles in evidence generation and at-scale impact, and we share a commitment to ensuring that robust evidence translates into informed policy decisions, and moreover, to a more inclusive approach to research that helps bridge the gap between knowledge generated in the field (mainly by international research teams) and local policy discussions.

The outcome of this research to policy pipeline is perhaps best illustrated by the case of school-based deworming, where rigorous research that found positive impacts of deworming on children’s education and health helped fuel nationwide scale-up in many endemic countries. Now, we have gone one step further — combining evidence and real-world implementation data through a novel approach called Open Policy Analysis (OPA) — to develop a tool that can better equip policymakers in their decision-making. We’re excited about the use of this tool to expand deworming, and more broadly for the deworming OPA, described below, to serve as a model for the translation of evidence to policy.

What kinds of evidence do busy policymakers actually need to make informed decisions, often with implications for millions of people? And what is the best way to deliver it to them?

According to a 2017 report by AidData — which examines how leaders in LMICs source and use data to accelerate development — policymakers placed the highest premium on evidence and analyses that generate context-specific insights, demonstrate a clear understanding of local realities, and leverage national survey data. Policymakers further noted that after in-person interactions (formal and informal), written analytical products such as memorandums, policy briefs, and short technical papers are the next best mechanism for discovering new data and analysis to help decision-making. These findings suggest that policy analysis — the process through which evidence is contextualized for new settings to generate policy estimates that inform policy debates — plays a critical role in connecting evidence with policy.

Much of the research in development economics — think randomized evaluations and other types of rigorous impact assessments — generates knowledge about policies that were implemented in the past, often with a view toward informing future decisions. Even when studying pilot programs, the time lag between the conception of a research project and the generation of policy-relevant findings can span several years. In contrast, policymakers must operate with significantly shorter time horizons for decision-making (typically from weeks to a few months).

What’s more, there has been an explosion of rigorous research examining policies in LMICs in the last few decades. Add to this a greater focus on open science and mechanisms to promote it, which has further increased the credibility of evidence generated by researchers. Despite these advances, the practice of policy analysis remains rather opaque — and so the evidence-policy pipeline remains weak.

In order for this evidence-policy pipeline to be effective, the data, methods, and assumptions that guide decision-making must be open, transparent, and account for users’ limited time. A widespread lack of transparency in how policy analyses are conducted, applied, and timed leaves ample room for misinformation and “alternative facts” to influence policy debates, allowing different parties to cherry-pick their own facts and resulting in suboptimal decision-making.

In response to this conundrum, researchers at CEGA have pioneered a tool to improve transparency — Open Policy Analysis. OPA — described in detail on CEGA’s website and in this earlier blog post — translates the lessons from open science into policy analysis by opening up underlying data, code, and materials, and providing clear accounts of methodological decisions and resulting outputs. The framework is premised on principles of free and public reporting and reproducibility of data, assumptions, and analysis.

CEGA is making deep investments in how policy analyses are conducted, contextualized and shared — with the goal of quickly and accurately informing specific policy decisions. In a partnership with Evidence Action, the parent organization of the Deworm the World Initiative, we have developed an OPA project for mass school-based deworming in LMICs, an intervention designed to improve health, education, and long-term wellbeing of hundreds of millions of children worldwide.

This project combines rigorous evidence and real-world implementation data to translate robust research into policy recommendations. Drawing on results from a randomized evaluation of a deworming intervention in Kenya, led by CEGA Faculty co-Director Edward Miguel and Nobel Laureate Michael Kremer (with coauthors Sarah Baird, Joan Hamory, Michael Walker), and deworming program implementation data from Evidence Action — we created a comprehensive analysis of the costs and income benefits of mass deworming across several settings.

This OPA accounts for the complex programmatic aspects of deworming interventions that Evidence Action deeply understands through its support of government-run mass school-based deworming programs, such as the variations in costs and longevity. Importantly, users of the OPA can adjust assumptions and customize modeling criteria, enabling context-specific estimates of deworming intervention impacts. In line with what the literature suggests decision-makers need, this new deworming OPA generates transparent, contextually-appropriate, and streamlined analyses to inform effective (and ongoing) policy making.

The key result of the OPA is net present value (NPV), which expresses the per capita increase in lifetime earnings for a child dewormed net of the costs of the intervention. For example, users can find the replicated results from the original study in Kenya, showing that the per capita net increase in lifetime earnings for those dewormed was between $130-$740. Using data averaged from geographies where Evidence Action supports government deworming programs, the per capita net increase in earnings is estimated to be $290 — a significant return to an individual for such a simple and low-cost treatment (between $0.06-$0.86), especially in a LMIC setting. The positive NPV of deworming in geographies where worms are endemic (more than 1 billion children globally are at risk of worm infection) illustrates that the magnitude of the benefits of deworming far exceed the cost of programs.

At a practical level, Evidence Action will use the deworming OPA to evaluate prospective — as well as retrospective — policy impacts of deworming interventions and develop context-specific recommendations for government partners. The OPA enables Evidence Action to make country-specific cases for deworming beyond the disease data and extending to estimated economic impacts. This in turn can help align resource and programming priorities that are often in tension, for example to evaluate the best use of funds for program expansion by weighing the relative cost-effectiveness of deworming in various geographies. For example, the analysis shows how net income benefits would be much higher in India (around $310 per child treated) than in Vietnam (around $50). These differences are driven by variations in unit costs of treating each child and worm prevalence rates across geographies.

We are optimistic that policy analysts in governments, donors, and NGOs will use the deworming OPA to inform financing and programming decisions, by enabling them to access both robust evidence and existing program data. This OPA translates evidence into terms that ministries of finance — which evaluate cost investments and make decisions about how to allocate scarce resources — and ministries of health and education — which design and implement deworming programs — can jointly understand.

Researchers can use this OPA to identify gaps in knowledge and better direct intellectual resources to specific needs that may ultimately affect policy decisions. Those studying deworming can dive deep into each input to the policy analysis (e.g., prevalence of infections or effect on earnings) and prioritize future research around unanswered questions. Researchers may also use this OPA to inform future sample size calculations (that justify the choice of minimum detectable effect sizes based on how it would affect the final policy estimate, as opposed to ad-hoc), by linking a minimum effect size (associated with a given sample size) to an actual policy estimate.

Given the promise of OPAs, we seek to institutionalize their use by developing and testing tools, practices, and community standards, and supporting individuals and organizations in operationalizing them. With policymakers needing clear, timely, context-specific, and rigorous evidence to make better policy decisions, we believe that OPA is one critical link in the pipeline. OPA strengthens the link between evidence and policy, maximizes efficiency by automating regular analyses, and introduces clarity in how evidence is used by diverse stakeholders. We are excited for the potential of the deworming OPA in assisting policymakers’ decision-making on critical health and education investments. As a result, the deworming OPA can help improve the integrity of policies that impact hundreds of millions of lives.