Michael Peyton Jones

Charity Update I: Development Media International

This is the first of a series of blog posts on charities we’re re-evaluating this year.

 is an organization we’ve had our eye on for some time. Back in 2013 we 

 their Head of Public Engagement, Will Snell, and in May last year we 

To recap what we already know, DMI is an organization focussed on a health intervention that is potentially very effective, but difficult to assess: mass-media campaigns. Their main technique is short, targeted radio spots of 60 seconds or less. These provide small pieces of health information in a memorable form, and the hope is that if people actually respond with behavioural change, then we may see a substantial drop in child mortality.

The big question is how much of an effect DMI’s efforts actually have. Their starting point is a statistical model which they developed in conjunction with the 

London School of Hygiene and Tropical Medicine

. This estimates a reduction in child mortality of 

. Such a high effectiveness is consistent with previous analyses, such as the 

 that ranked media campaigns very highly as a means for combatting HIV/AIDS. That’s a good starting point, but what really excites us is that they’ve been following this up with a large-scale randomized-controlled trial across Burkina Faso (see the 

The RCT is a long-running procedure: it started in 2012, and the full, end-line results are not expected until the end of this year. But DMI have recently released the midline survey results, which give us 

The midline results come in the form of survey results measuring various self-reported metrics across the treatment and control groups. Overall, they look promising, showing a consistent improvement in the treatment group. Even after discounting their estimates significantly, GiveWell give their current estimate as $5,236 to save a child life. That’s within a factor of two of some of our other recommended charities (GiveWell 

 that AMF, one of our top charities, saves a child’s life for $3,340), and DMI’s own estimate is much lower.

However, we do have some concerns about the interpretation of these results. GiveWell provides a comprehensive list, but we’d like to highlight just three points here:

The results come from self-reported survey data. This means they are always going to be less reliable than assessment of objective measures, such as mortality data.

Lack of blinding. Both the participants and the surveyors may have been aware of which group the participant was in - the survey itself contained questions on whether they had heard the radio spots.

Asymmetries between the control and treatment groups. There were some relevant health differences before the treatment began, and there were other health programmes running simultaneously with the RCT.

The self-reporting and blinding problems will not persist to the endline data - for that, DMI are going to measure mortality statistics. However, they do present problems for the midline data. The lack of blinding is the most worrying: lack of blinding can lead to 

, and its effects are hard to account for in retrospect. However, DMI have confirmed that the questions which specifically relate to their programmes appear in the last section of the questionnaire. That should mean that the interviewees will only be unblinded for the last section at worst, although the interviewers are still unblinded. While not ideal, this is much better than having a full lack of blinding.

The problem with asymmetries between groups will remain even for the endline results. GiveWell highlights three prominent health programmes that might have had some effect on child mortality during the running of the trial. By and large, it seems likely that the effects were present in both groups, and the independent evaluators at the LSHTM plan to adjust for such baseline imbalances between the control and intervention arms in their analyses.

Finally, we have some worries about the portability of DMI’s programmes. Unlike many other health interventions, which have a biological pathway of action that we can expect to work consistently across humans, DMI’s radio spots rely on culturally changing behaviour. As such, they may run into communication barriers that are hard to discover - in one country an older female voice might be regarded as authoritative, whereas in another the reverse might be true. However, DMI have extensive experience in developing these programmes, and perform a great deal of formative research and testing for efficacy. If the RCT provides an endorsement of the overall methodology, then we are fairly confident that cultural hurdles can be overcome.

Overall, we broadly agree with GiveWell that DMI is an excellent organization with plenty of room for funding, and that the midline RCT results are promising. DMI’s approach of spreading simple, but effective, health information is very scalable, and there are a wealth of possible extensions to the programmes (for example, further simple improvements in 

). With that in mind, we consider DMI to be a promising charity, and if the endline results from the RCT prove good, we are likely to consider them one of our top charities.

Thanks to DMI for clarifying several points with us directly, and to GiveWell for an excellent baseline report.

Our Charities Update Is Coming!

It’s that time of year - we’re re-evaluating our 

Over the next weeks we’re going to release a series of blog posts for a few charities we’re currently considering recommending. This will include several charities we haven’t recommended before, as well as some that we already do. We won’t be discussing all our current top charities, however, as in most cases the relevant facts have not changed much.

Once we’ve released all the blog posts we’ll update the 

 page. This is of particular interest to anyone who gives via the 

 - a change in our top charities will result in a change in how we distribute funds from the Trust.

Michael Peyton Jones

Articles

Charity Update I: Development Media International

Our Charities Update Is Coming!