By Caleb Watney, Matthew Esche, and Tim Hwang
Emily Oehlsen:
Another observation that comes to mind is that sometimes thinking about a single paper or a single experiment is not the right unit. One example here is there were a number of RCTs that were run around water quality, but they were all individually underpowered to look at mortality because mortality is rare. Michael Kremer – who recently won the Nobel Prize – did a metaanalysis and found a big, statistically significant effect from these water quality interventions on mortality. That meta analysis played a significant role in GiveWell’s decision to scale up Evidence Action’s Dispensers for Safe Water program. Using this as one example to say that sometimes an individual experiment isn’t enough in and of itself to be decisive, but it can be coupled with other types of evidence that can then lead to a bigger decision.
I observed this from afar, but it’s an example that I’ve always found, pretty inspiring. Evidence Action was founded in 2013 to scale evidence-based, cost-effective programs. They had their core programs around deworming and scaling free, reliable access to safe water. But then they also had this program called No Lean Season where I think the original experimental evidence was from Bangladesh. It involved giving people both information and then small loans, so that they could migrate to other parts of the country, when seasonal work was scarce where they lived. The original RCT evidence showed that this was a pretty promising intervention for poverty alleviation, and so, Evidence Action started to scale it up. Then they ran two more RCTs as it scaled, that showed that it was less effective than they had expected, and they ended up shutting down the program.