The case for mass school-based deworming in endemic countries stands on two legs: First, the body of rigorous evidence that supports mass deworming as a cost-effective intervention. Second, mass treatment without first testing for infection is cheap, safe, and an efficient strategy for reaching lots of kids quickly. Claims in a recent paper by Calum Davey and colleagues and the Cochrane Review by members of the same research group that have been picked up in the press are based on flawed analysis, and could threaten an emerging public health success story.
Controversy Sells?
Alexander Aiken, Calum Davey, James Hargreaves and Richard Hayes were tasked by the International Initiative for Impact Evaluation (3ie) to replicate a randomized controlled trial of the provision of deworming pills in schools in Kenya by Edward Miguel and Michael Kremer. Replication and transparency in science are great, and we need more of it. However, we also need to be careful what incentives we create for the replicators—controversy sells after all, right?
And indeed, Aiken et al. look to have done their darndest to generate controversy. When they follow the same statistical strategy in the original paper, which passed peer review of perhaps the top economics journal, Econometrica, they find a few errors but arrive at the same fundamental result that deworming increases school attendance. This replication effort is tremendously useful.
Poor Science
Then, Aiken et al set out to parse the data differently than Kremer and Miguel. And sure enough, if you make enough changes to how you do the analysis, you can make the key result go away. Is this really surprising? Unfortunately, what Aiken et al do is statistically not best practice. They chop up the data into smaller subsamples for analysis; they include data on kids before deworming treatment starts in the deworming treatment sample. No surprise: These choices, which your Stats 101 professor would counsel against, make the key result weaker.
The reparsing of the data to get a different result is not a “failure to replicate”. And at the time of the 3ie publication this point was made by several commentators. Kremer and Miguel and their colleagues noted this. Berk Ozler discussed this on his blog. Michael Clemens wrote a paper urging us to be careful about the use of the word “replication” when authors are going far beyond that in their analyses. Other researchers that had been part of the 3ie replication effort have independently questioned the value of “replications” that go far beyond that remit and then suggest that a weakness in the original research had been uncovered.
It’s unfortunate that the peer review process at the International Journal of Epidemiology did not encourage Davey and colleagues to substantively address the feedback that their data parsing choices are statistically flawed compared to the choices made in the original research by Kremer and Miguel. This has resulted in a manufactured controversy that could threaten real programs that help real kids.
Mass Treatment of Kids is Most Effective
That brings us to the second leg of school based deworming: Mass treatment of all kids in places where worms are endemic is the cheapest and most efficient way to reach infected kids. Treating uninfected kids has no side effects. The medication is very cheap. The tests are very expensive. The Cochrane Review gets this one wrong when they suggest that we need to test before treatment to get health benefits. Presumptive treatment is an operational, pragmatic approach to treating as many sick kids as possible with a limited amount of money.
We have made incredible progress over the past few years in getting more kids to have the chance to live worm-free lives. We can not let weak scholarship and a flawed peer review process — let alone expensive treatment strategies — get in the way of this and hurt kids in the poorest countries around the world.
