Dispensers for Safe Water: An Updated Review of the Evidence

Dispensers for Safe Water provides safe water through a household water chlorination service to four million people in Kenya, Malawi and Uganda. Dispensers represent an important innovation in the rural water sector because they solve several challenges that have traditionally hindered sustainable, quality water services. 

Why are we taking a close look at the evidence of impact of Dispensers for Safe Water?

We only implement programs that have been rigorously evaluated, that have proven and measurable impact, and that are cost-effective. We periodically review and assess the evidence base for our programs to ensure that we are aware of potential changes which may impact our assessment of the underlying strength of evidence for a program, and to adapt our programs.  The following is our current assessment of the evidence base for Dispensers for Safe Water.

In 2014, we summarized the previous evidence in a post: Chlorine Dispensers Work for Safer Water. By 2017 new studies prompted us to re-evaluate whether the evidence base for Dispensers for Safe Water remained sufficiently robust to meet our criteria for impact and cost-effectiveness.  Led by the Chair of our Board of Directors, Amrita Ahuja, we conducted a rigorous review process with input from external experts and thought leaders in the field of water and health.

What’s The Upshot?

After a thorough review that we detail below, we continue to have confidence that Dispensers for Safe Water is backed by strong evidence and is highly cost-effective. 

What Did We Examine?

We conducted a literature review that included: systematic reviews, meta-analyses and very recent literature (i.e. that became available after the last meta-analysis was published) linking water quality and health, and studies relevant for understanding chlorine’s impact on water quality. 

We sought to answer the following questions:

1. What is the causal link between chlorine treatment and water quality?
2. What is the causal link between water quality and health outcomes?
3. What is the role of user adoption in realizing health benefits?
4. What are criticisms about survey biases in the literature and where do we stand regarding those?

Below is a synopsis of evidence in each of these areas, followed by a detailed summary of key studies.

The Synopsis

Evidence from lab and field tests show that chlorine does indeed improve water quality.  Chlorine is very effective at killing most bacterial and viral pathogens. It is less effective against parasites like cryptosporidium and when there is a large amount of organic matter in water. Safety concerns with dilute chlorine are minimal, and it is widely used as a disinfectant in water treatment plants around the world.  Chlorine can provide residual protection for up to three days (depending on storage conditions), which means that it not only disinfects but can also prevent recontamination. This is important in places where people store collected water in their homes (as is common in many of the areas where Dispensers for Safe Water works), as safe handling may be hard to guarantee.

We also have confidence that chlorine actually kills many of the pathogens that cause diarrhea in real-world settings.  The key evidence comes from the Global Enteric Multicenter Study, published in 2013, that identified the main pathogens causing diarrheal disease among children under five from seven different sites across Asia and Africa. The most attributable cases of diarrhea were due to four pathogens: rotavirus, Cryptosporidium, enterotoxigenic Escherichia coli (ST-ETEC), and Shigella.  Chlorine is effective against three of these four pathogens.  A re-examination of this data identified six pathogens as important in causing the majority of moderate to severe childhood diarrhea, and chlorine is effective against five of these. 

The basic mechanism by which water treatment improves health outcomes continues to be strongly supported by the evidence base. There are two independent strands of literature:

(a) A vast epidemiological literature that suggests significant reductions in diarrhea – roughly 25% on average – in communities provided with a drinking water quality intervention. Four systematic review articles survey and assess the evidence for water quality interventions in developing countries on child health, particularly diarrheal diseases. Three of them conclude that the health benefits of interventions like chlorination are clear, but dependent on consistent use, which can be a challenge with some technologies.  The fourth concludes that water treatment can be helpful, but does not find improvements from chlorination to be statistically significant when adjusting for potential bias from lack of blinding. Two recent studies, conducted after the last review was published, also show that water treatment significantly reduces diarrhea, including one from rural Kenya, in areas close to those we work in. 

(b) An economics literature that demonstrated that the expansion of safe water projects reduces child mortality.

Three of the four systematic reviews mentioned above consider the impact of sustained use of water treatment and health.  All three reach the conclusion that high adherence is important for maintaining impact. This is, of course, not surprising: A product cannot provide health benefits if people don’t use it. We note that Dispensers for Safe Water has a verified high sustained use by users. 

The key outcome measured by most of these studies is reported diarrhea, rather than a more objective measure, and in most studies, treatment is not blinded from study participants or researchers.  Hence, some have raised concerns that survey response bias accounts for most or all of the reductions in diarrhea attributed to water treatment. While we agree there should be more studies with objectively verifiable outcomes, the concern seems exaggerated.  Self-reported outcomes are correlated with more objectively verifiable measures such as the sustained use of water treatment and microbial concentrations in water; we would not expect this to be the case if outcomes were driven primarily by survey response biases.  Additionally, to account for all or most of the observed effects of water treatment, bias from lack of blinding would need to be very high.  Also, recent studies which try to adjust for the lack of blinding through innovative survey strategies continue to show that water treatment reduces diarrhea.  

And Finally…

The biggest “threats” to the effectiveness of our program are actually good news:  Economic growth and development mean that people are getting less poor, water infrastructure is improving, and child mortality is going down. In terms of our program design, this means that we need to make sure that we are working in areas of high need (high diarrhea rates, poor health outcomes, poor access to health care, high poverty) and are achieving high levels of user adoption within those areas.

Our current program monitoring data suggests that we are operating in areas of high need, where diarrhea and child mortality rates are high.  Additionally, adoption remains high, and the vast majority of those who treat their water have water that meets WHO standards for safety.

In Summary

We reviewed the evidence base linking water chlorination to health outcomes. We found that chlorine does indeed improve water quality and is very effective at killing most diarrhea-causing pathogens. Having reviewed two independent strands of literature – a vast epidemiological literature and economic literature – we found that the basic mechanism by which water treatment improves health outcomes continues to be strongly supported by the evidence base. Perhaps unsurprisingly, we found that high adherence to household water treatment interventions is important for maintaining impact. And finally, we discussed concerns with survey biases and assert that these seem overblown. We suggest a more practical approach followed by Cochrane review authors who apply some reasonable discounting methods, and who still find health impacts from chlorine are significant.

Key Takeaway

We have therefore emerged from this review process with confidence in the evidence behind our program.

If you like to read in more detail what evidence we examined, please read on for a more detailed review.

Below we present more detail from the findings of our review of the current evidence base.

The epidemiological literature on the health benefits of safe water is complemented by evidence from lab and field that chlorine does indeed improve water quality.  Chlorine is very effective at killing most bacterial and viral pathogens. It is less effective against parasites like cryptosporidium and when there is a large amount of organic matter in water. Safety concerns with dilute chlorine are minimal, and it is widely used as a disinfectant in water treatment plants around the world.  Chlorine can provide residual protection against pathogens for up to three days (depending on storage conditions), which means that it not only disinfects but can also prevent recontamination. This is important in places where people store collected water in their homes, as safe handling may be hard to guarantee.

Regarding the effect of chlorine on diarrheal microbes, the original Global Enteric Multicenter Study (GEMS), published by Kotloff et al. in 2013, was the largest conducted study on diarrheal disease in developing countries. It was monumental for our understanding of the underlying causes of diarrhea. The study identified the main pathogens causing diarrheal disease among children under 5 from seven different sites across Asia and Africa. The most attributable cases of diarrhea were due to 4 pathogens: rotavirus, Cryptosporidium, enterotoxigenic Escherichia coli (ST-ETEC), and Shigella.  Chlorine is effective against three of these four pathogens.  Liu et al. (2016) later reassessed the causes of diarrhea found in the GEMS study.  The six most attributable pathogens were: Shigella, rotavirus, adenovirus, ST-ETEC, Cryptosporidium, and campylobacter. These six pathogens accounted for 77.8% of all attributable diarrhea in the GEMS sample. Chlorine is effective against five of the six most attributable pathogens.

Evidence from the Epidemiology Literature

Four systematic review articles survey and assess the evidence for drinking water quality interventions in developing countries having an impact on child health, particularly diarrheal diseases. Three of them conclude that the health benefits of water quality interventions like chlorination are clear but dependent on consistent use, which can be a challenge with some technologies.  The fourth concludes that water treatment can be helpful, but does not find improvements from chlorination to be statistically significant when adjusting for potential bias from lack of blinding.  These reviews focus on “point-of-use” treatment. Dispensers may be thought of as “point-of-collection” treatment, rather than point-of-use; but epidemiologically this should not matter.  The reviews include:

Clasen et al. (2015), the most recent Cochrane Review on the topic, includes 55 studies, most of which were conducted in low- or middle-income countries (LMICs) covering over 84,000 participants. Results indicate that improving the microbiological quality of water at the point of use significantly reduces diarrhea, by about a quarter for chlorination and 50 percent for filtration, in the studies reviewed. The review indicates that these impacts hold even in areas with poor sanitation and at improved and unimproved sources of water. The authors caution not to interpret this as meaning that filters are much more effective than chlorination (or vice versa) since these were not compared head-to-head but instead evaluated in different contexts with different study populations and designs.

Hence, the authors conclude that: “Interventions that address the microbial contamination of water at the point-of-use may be important interim measures to improve drinking water quality until homes can be reached with safe, reliable, piped-in water connections. The average estimates of effect for each individual point-of-use intervention generally show important effects.”  Sub-group analysis also suggests that effectiveness improves with adherence, so optimizing coverage and long-term usage remain important.

Waddington et al. (2009), a 3ie Review, covers 65 rigorous impact evaluations of water, sanitation and/or hygiene interventions on diarrhea morbidity, covering 71 distinct interventions assessed across 130,000 children in 35 developing countries during the past three decades. It concludes that point-of-use water quality interventions appear to be highly effective – and indeed, more effective than water supply or source treatment in reducing diarrhea – but that this is very sensitive to the ability of the program to sustain high rates of adoption. A product cannot provide health benefits if people don’t use it.

Arnold and Colford (2007) conducted a systematic review of all studies that measured diarrheal health impacts in children and the impact on water quality of point-of-use chlorine drinking water treatment. Twenty-one relevant studies were identified and a meta-analysis provided summary estimates of the intervention effect. The intervention reduced the risk of child diarrhea by 29%, and up to 40% if you look at effects only among people actually using the product, as opposed to only being offered it.

Wolf et al. (2014) included 61 studies on improved sources or water treatment at home, and 11 studies on sanitation. Overall, improvements in drinking water and sanitation were associated with decreased risks of diarrhea. The authors found that use of water filters and provision of high-quality piped water and sewer connections, were associated with greater reductions in diarrhea compared with other interventions.  Chemical disinfection was found to be effective, but not when taking into account bias from lack of blinding.  The authors conclude that: “The results show that inadequate water and sanitation are associated with considerable risks of diarrheal disease and that there are notable differences in illness reduction according to the type of improved water and sanitation implemented.”

It is worth highlighting individual studies (in preparation for publication or published) that became available after the above reviews were published:

Null et al. (in process) present results from a three-year cluster randomized controlled trial of point-of-collection (POC) chlorination via chlorine dispensers in Western Kenya. The study sample included forty-nine water sources (824 households), randomly assigned to receive a dispenser versus the control group with access to socially marketed bottles from shops. Adoption was more than five times higher in treatment versus control even at 3-years post-dispenser installation, and households in the treatment group had significantly less contaminated drinking water and lower rates of diarrhea than those in the control group. 

Ercumen et al. (2015) in a study in Bangladesh also showed that both safe storage alone and safe storage plus chlorination significantly reduce diarrhea (by 31 and 36 percent respectively) in children under two, underlining the benefits of improved water.  Diarrhea was self-reported, but other health measures not affected by water quality (e.g. ear infection, skin disease) were also collected, with no significant differences between treatment and control.

Evidence from the Economics Literature

The following papers in the economics literature complement the epidemiologic evidence on water quality and add to our understanding of the health impacts (child mortality) of improved water and sanitation.  On their own, they show the importance of improved water quality rather than of point-of-use chlorination.  However, combined with the strong evidence that point of use chlorination improves water quality, they make a strong contribution to the case for chlorination’s impact on health.  This is further strengthened by recent evidence on the links between water quality and diarrhea, and evidence showing that many (although not all) of the microbes that cause moderate to severe diarrhea are ones that chlorine is effective against.

Cutler and Miller (2005) use historical variation in the timing and location of water filtration and chlorination technology adoption across U.S. cities to identify the contribution of improved water quality to improved public health in U.S. cities. They find that safe water was responsible for about half the observed decline in mortality and nearly two-thirds of the reduction in child mortality in cities.

In a rural setting Watson (2006) analyzes the introduction of various water and sanitation interventions on Native American reservations in the United States from 1960–98. She leverages differences in the size, timing and location of these interventions to isolate the impacts of the program, distinct from other factors affecting infant health. The study suggests that a 10% increase in the fraction of homes with improved water and sanitation services reduced infant mortality by 4%. Infant mortality also fell among local residents not living on the reservation, and thus without access to the new services—which suggests that there may have been positive externalities associated with the program.

Galiani, Gertler, and Schargrodsky (2005) study a privatization reform in 30% of municipal water companies in Argentina in the 1990s to identify the impact of the expanded coverage with safe water. They estimate that child mortality overall fell 5–7% in areas that privatized their water services because the reform did succeed in expanding coverage with safe water.  The effect was largest in the poorest areas, at around 24%, and was not correlated with causes of death unrelated to safe water.

There are typically two concerns voiced with these studies.  First, they are not randomized trials, and second, they are far in time, space and specific type of project from modern day household chlorination.  On the first point, while they are not randomized controlled trials, they use careful statistical techniques to identify causal links between water infrastructure improvements and health benefits.  Also, especially given reducing childhood mortality, randomized trials on mortality require increasingly large sample sizes and are becoming increasingly expensive.  Hence, we will likely have to rely on high-quality statistical studies of these kinds to contribute to our best available information on childhood mortality.  On the second point, the environments in which these studies were conducted were similar to those in developing countries today in that water was not the only source of disease microbes (e.g. there was often open defecation/incomplete unsafe sanitation, animals in the environment, etc.).  Also, the basic biological pathway – reduced contamination leads to reduced disease – remains similar between the two types of programs. 

For an evaluation in a present-day rural setting that is most closely comparable to the setting where dispensers are used, Kremer et al. (2011) study a “spring protection” program that improves water quality at the source in rural areas in Kenya. Spring infrastructure investments reduce fecal contamination partially, but not completely, just as one might think of chlorine doing (though of course without the benefits of residual protection). Nonetheless, child diarrhea, as reported by mothers, falls by one-quarter.  The estimates of health benefits are fairly close to those in the epidemiology literature.

Three of the four systematic reviews mentioned above consider the impact of the sustained use of water treatment and health.  All three reach the conclusion that high adherence is important for maintaining impact. This is, of course, not surprising: A product cannot provide health benefits if people don’t use it. We note that Dispensers for Safe Water has a verified high sustained use by users.  For example, Classen et. al. (2015) conclude that “Evidence indicates the more people use the various interventions for improving water quality, the larger the effects, so research into practical approaches to increase coverage and help assure long-term use of them in poor groups will help improve impact.” Waddington et. al. (2009) and Arnold and Colford (2007) both note that maintaining adoption over time remains important in maintaining impact.   Brown and Clasen (2012), using a mathematical modeling approach, also suggest that adherence is important for high adoption.  

The reviews all document the short duration of the studies that are available for their analysis, and the need for more trials that are blinded or rely on objective measures of health outcomes instead of self-reported diarrhea.  However, the key concern with the quality of evidence seems to center around the lack of blinding in most studies, and the failure of the few blinded studies that exist to show impacts (Schmidt and Cairncross, 2009).  While we think this points to a real issue, and support the call for more blinded studies and studies with objective outcomes, we also think it is somewhat overblown for the following reasons:

• Blinded studies constitute only a small fraction of the studies on water treatment, and as discussed below, all these studies have fundamental problems. This is not to deny that more blinded studies would not be useful; however, those we currently have are not very informative, and should be given limited weight in assessing the evidence. A fuller discussion of existing blinded studiesis provided below
• The impact of chlorination is large, and survey biases would have to be very large to explain away the full impact seen in studies of chlorination. The most recent Cochrane review tries to adjust for the lack of blinding, and finds that chlorine is beneficial even when accounting for this. The impact is somewhat attenuated (20 percent versus 25 percent), but still meaningful. (Note: The authors caution that their methodology for adjusting for lack of blinding is adapted from medical trials and not validated in field trials. However, at the very least this analysis would suggest that if the full impact of chlorination is from survey bias, these biases would be very large as compared to what is seen in medical trials.)
• Two recent studies which at least partially try to address issues of survey bias have shown strong positive results of water treatment: (1) Null et. al.’s study (in preparation for publication) includes infrequent surveys, which minimize survey bias and (2) Ercumen et. al. 2015 use placebo outcomes (ear infection, skin conditions) which do not improve with water treatment, even though diarrhea does.
• Many studies of chlorination, in addition to measuring diarrhea, a subjective measure, also measure indicator variables for microbial contamination of water (e.g. E. Coli concentrations), an objective measure, and chlorine adoption, which is also objectively measurable. There is a well-established link (albeit the relationship is not necessarily linear or straightforward) between reduced contamination and reduced diarrhea. For example:
  • Luby et al. (2015) assess the impact of the microbiological quality of water on childhood diarrhea in Bangladesh and found that E. coli contamination was associated with increased childhood diarrhea in the period immediately following testing. This study was conducted only within the control group of a larger study, so there was no risk of bias because participants were recipients of an intervention. The timing of effects also suggests a link from contamination to diarrhea and not the other way around. This is consistent with other studies which suggest that reduced contamination leads to less or less severe intestinal diseases – e.g. Gruber et al. show that E. Coli concentrations are positively correlated with diarrhea incidence, and Mintz et al. show that salmonella incubation period and severity is strongly correlated with dose of contaminated food consumed in a disease outbreak.
  • As mentioned above, all three systematic reviews which consider the issue draw a link between adoption of water treatment (which is objectively verifiable, in the case of chlorine treatment) and health impact

Blinded Studies

In most epidemiological studies on water treatment, the key outcome is one that is reported rather than objective – i.e. a parent (usually a mother’s) reports diarrhea instances in children. The concern is that in the absence of blinding, all or most reductions in diarrhea are subject to survey response bias, especially because of the few studies that do blind treatment, most do not show that water treatment reduces diarrhea. 

Clasen et al. (2015), the most recent Cochrane Review that covers 55 studies, explicitly considers the potential bias from the fact that the majority of the studies are not blinded.  It adopts techniques from the medical trial literature to adjust meta-analysis results for the lack of blinding.  The results are attenuated (approximately 20 percent reduction in diarrhea from chlorine treatment instead of an approximately 25 percent reduction if lack of blinding is not accounted for) but still positive and significant even when this potential is accounted for.  The review also specifically considers nine trials that were blinded and notes that all had significant problems with either study design or “power” (i.e. the sample size being large enough to detect outcome differences between treatment and control groups, given actual diarrhea rates and adoption).  Five were in areas with high quality water, including four in developed countries (three in the USA (Colford et al, 2002, Colford et al, 2006, Colford et al, 2009), one in Australia [Reller et al. 2011] and one in Ghana [Jain et al. 2010], but with relatively low levels of contamination). The study in Ghana by Jain et al. 2010 also included a placebo arm which itself may have reduced up to 90% of the contamination, and one other study also had a placebo that likely reduced contamination.  In addition to having an effective placebo, this study also had high rates of reported water consumption from untreated sources.  Two other studies [Austin, 1993] and [Boisson et al. 2013] had low adoption rates and were hence likely underpowered to detect effects.  The Boisson (2013) study also had much lower than anticipated diarrhea rates and the other [Austin, 1993] is not peer reviewed.  The remaining study [Krichhoff et al. 1985] included only 112 people from 16 households.  

While many studies have problems, the above-mentioned are fundamental issues which undermine the extent to which studies can be informative. For example, what can one say about the effectiveness of water treatment if water is not contaminated in the first place?  What can one say about the comparative effectiveness of a treatment if the placebo control intervention also removed the vast majority of microbes? The issue of adoption is also a very important one. Most epidemiological studies are designed to be able to detect effects at the 5 percent significance level with 80 percent power (i.e. detect effects 80 percent of the time that they are actually present).  They are also typically designed to achieve take-up rates of 60 percent or higher.  With adoption rates that are lower than projected, the risk of missing a real effect is increased significantly beyond 20 percent.  If take-up rate is more like 30 percent (e.g. Boisson, 2013 [15]), then your treatment group consists of 70 percent people who are not using the intervention and diluting any effects on the 30 percent of users.  Hence, any effects experienced by those 30 percent of users become very hard to detect without large sample sizes. 

Our Take on Blinding

While blinding remains an important issue, and more studies with blinding and/or with objective outcomes would be useful, the concern seems exaggerated for the following reasons:

(1) Recent research suggests that diarrhea reduction and diarrhea rates do seem to be correlated with real outcomes such as the sustained use of water treatment and microbial contamination; we would not expect this to be the case if reductions were driven primarily by survey response bias. 

(2) For all or most of the diarrhea reduction from chlorine to come from survey effects, the effects of blinding would be much larger than typically seen in most medical studies

(3) The existing blinded studies have fundamental flaws, which make them unable to provide much information on whether water treatment reduces diarrhea;  hence, their not showing impacts of water treatment is neither surprising, nor a reason to rethink the entire (much larger and more robust) body of work on water treatment

In summary, while we think that the need for blinded trials and more trials with objective measures is real, we think that it is overstated. Assuming that all or most of the estimated effects of chlorination come from survey effects in the absence of blinding (a) ignores some nuances in recent literature which suggest otherwise; (b) ignores an established link between lower microbial contamination and disease; (c) suggests that survey effects are very large and (d) gives undue weight to a few blinded studies that cannot provide much information on diarrheal disease and health at the expense of a larger and more robust body of work.  A more reasonable approach seems to be that followed by the Cochrane review authors, with possibly some reasonable discounting of impacts due to biases from lack of blinding.

If you made it this far…

Congratulations! We are motivated and inspired to emerge from this review process with confidence in the evidence behind our program, and we invite you to follow our progress.