Sally Murray

Thought the question of the most effective charity was settled? There are always more questions.

This Tuesday, charity evaluator GiveWell removed the Against Malaria Foundation (AMF) from their top-recommended charities list. What was the reason?

GiveWell’s central concern is that AMF may fail to use new donations to fund bed nets, or only purchase these bed nets with great delay. In the past two years, AMF have spent less than one tenth of the funds they have received (~$1m, out of over $12m) and although AMF say they are currently in talks about distributions worth $40m,[1] GiveWell want more evidence that AMF really can ‘scale-up’ to this sort of level; if and when AMF distribute the $12m they currently hold, GiveWell will review their case and say they are likely to recommend them once again.

It might seem odd that AMF are failing to distribute their funds. There seem to be plenty of at-risk people who don’t have insecticide-treated bed nets… So what’s the problem?

Well, it seems that although many health charities and Ministries of Health want nets, many won’t accept them under AMF’s conditions- particularly, their conditions for strong monitoring of net distributions. To receive nets from AMF, for example, an organisation must take photos and videos of net distributions, and this imposes some cost and hassle for the distributor. Although AMF’s intention with such requirements is to deter the stealing of nets, net distributors may prefer to avoid the hassle, and hold out for the chance of obtaining nets from less demanding donors.

We have all sorts of questions to pose to AMF, and we hope to soon hold a Skype conversation with AMF’s founder Rob Mather. What would you like us to ask? What do you think of GiveWell’s demotion of AMF? Which charity will you be donating to following this update?

NB GiveWell suggest other possible reasons AMF has struggled to use its funding, and we recommend you to read their 

Is it good to campaign for increased government aid to cost-effective interventions? Part 2

, I explained that if most government health aid is highly cost-effective, lobbying for governments to spend more on any specific cost-effective health aid project may be a bad investment. This, roughly, is because such lobbying would merely move aid from one highly cost-effective intervention to another- doing no overall good.

In this post, I will explain why we do indeed have some reason to believe that most government health aid is highly cost-effective, and so, advocacy of the kind described above may be a bad investment for an effective altruist.

As the world's biggest national donor to health aid, and the target of much advocacy, we began by investigating the cost-effectiveness of the US’s overseas development assistance (ODA) for health. (We later roughly check our results against other countries).

We attempted to estimate the cost-effectiveness of US health ODA by reading descriptions of its projects, and estimating their cost-effectiveness using secondary literature, mainly the DCP2. The individual projects that comprise the US's health aid are listed in the OECD's online QWIDS database, which provides a brief description of each aid project, as well as the amount of aid committed and distributed to it. There are thousands of projects in the database for each year, so for efficiency we took a randomised sub-sample of about 200 projects for the year 2010 (with greater representation of projects according to the amount of funds they received). Because this sample was relatively small, we later conducted an informal eye-scan of a far larger number of projects (~1000) to check if the results we'd obtained seemed representative- it seemed that they were.

Using the project descriptions, we matched each US health ODA project in our sample with its closest equivalent project in the DCP2- one of the most respected documents offering cost-effectiveness estimates for a wide range of public health interventions. In rare cases where we had better data regarding an intervention's cost-effectiveness, from, e.g. GiveWell or WHO-CHOICE, we used this improved estimate.

By thus matching project descriptions with cost-effectiveness estimates for most-similar programmes, we obtained a ball-park estimate of the cost-effectiveness of each intervention-type in our sample of US health aid. We then used this data to explore means, medians, spread, and so on.

So what did this process uncover? The basic story seemed to be that most US health aid is allocated to highly cost-effective final interventions: neonatal and early childhood care, polio eradication through vaccinations, HIV prevention (e.g. family planning, education campaigns), water hygiene and sanitation (WASH), and so on. The estimated mean cost-effectiveness of our sample was 34 DALYs averted per $1000- around three times as cost-effective as the highly cost-effective interventions that we have found charities lobbying for (mass drug administration (MDA) for neglected tropical diseases (NTDs), or the distribution of insecticide-treated bednets (ITNs) for malaria- which we tend to model as averting around 10 DALYs per $1000). This high cost-effectiveness was not due to a few very highly cost-effective interventions skewing our mean upwards: most of the projects (i.e. the sorts of projects listed above) came out as cost-effective at more than 10 DALYs averted per $1000 (i.e., better than deworming etc).

It is important to note that USAID's health programmes almost certainly do not have a mean cost-effective of 34 DALYs averted per $1000. Almost certainly, US health ODA is considerably less cost-effective than this (for reasons outlined in the footnotes). But importantly, many of the reasons (2-4 in the footnote) to downgrade this estimate could well constitute reasons to also downgrade expectations of the cost-effectiveness of mass drug administration for NTDs, the distribution of ITNs and so on, at least when these are delivered by a government aid agency. If both are less cost-effective, then again, moving money from an average health aid project to ITN distribution (etc) could still do no good.

We should also note, however, that an estimate that US health ODA is on average of similar cost-effectiveness to deworming is less bizarre than you might think. Giving What We Can recommends donating to deworming and ITN charities because of their high marginal cost-effectiveness: given the existing health programmes that are already being carried out, these are the remaining, underfunded, programmes with the greatest expected positive impact per dollar. It is quite possible that before GWWC came along, the 'lowest hanging fruit'- the most cost-effective interventions- were already being swept up and funded fairly well, and that funding these programmes continues to take up most of countries’ health aid budgets. The average cost-effectiveness of any organisations that plucked these low-hanging fruit should be better than their marginal cost-effectiveness, and could well be better than the cost-effectiveness of the interventions GWWC have identified as still underfunded (especially because GWWC focuses only on interventions that private donors can support).

We have also not checked how far our results for USAID generalise to other aid agencies with great thoroughness. Very briefly, we can say that the UK's Department for International Development has a very good reputation (generally better than USAID's), and within health also seems to focus on highly cost-effective interventions (evidenced by its annual reports, and QWIDS). A brief scan of other major aid donors’ health aid portfolios (Germany, Sweden and The Netherlands) using QWIDS suggests a similar composition to that of the US, except with considerably larger proportions spent on "health sector development"; we struggle to assess the cost-effectiveness of ‘health sector development’, or even to gleam exactly what it involves, so remain unsure as to how this should affect our cost-effectiveness estimate (Can you help?).

In summary, we don't have enough evidence to conclude that US health aid- or health aid in general- has a mean cost-effectiveness similar to that of highly cost-effective interventions like mass drug administration for NTDs or free ITN distribution. However, we have enough evidence to suggest this may well be the case.

And if this is the case, and if (discussed in my previous blog), successful advocacy for any particular cost-effective health aid intervention tends to move money away from a competing project of mean cost-effectiveness, this form of advocacy would do little good; it would certainly be a bad investment for an effective altruist. Due to these doubts and uncertainties, we cannot yet recommend donations to organisations whose main claim to impact is through the performance of advocacy that encourages governments to increase their spending on particular cost-effective aid interventions. If we are to recommend such organisations in future, we will need to receive much stronger evidence that they generate large, positive improvements in the overall cost-effectiveness of government aid.

*Reasons to think USAID’s health aid is not cost-effective at 34 DALYs per $1000.

1. Some of the funding for each project is spent on the administration and bureaucracy that support it. We expect that if it accounts for admin costs at all, the DCP2 treats admin costs as much lower than those actually borne by USAID, which has many levels of disbursement, hires expensive consultants, and so on.

2. Aid projects are not described to a good level of detail in the QWIDS database. This made it hard to identify the DCP2 interventions with which each project best corresponded, and thus to attach the 'correct' DCP2 cost-effectiveness estimate to it.

3. The cost-effectiveness estimates in the DCP2 are themselves tentative and sometimes inaccurate to an important degree (see, e.g. 

http://blog.givewell.org/2011/09/29/errors-in-dcp2-cost-effectiveness-estimate-for-deworming/

). We do not have good evidence to inform how we should adjust each estimate to take account of, for example, regression to the mean.

4. Our model assumes that if USAID stopped funding a health project, this project would stop completely. This is probably not true in many cases. It's likely that without USAID's help, developing country governments would fund and manage at least some of the programmes currently funded by USAID.

5. USAID’s health aid is $8 billion per annum. If each $1000 of this really were averting 34 DALYs, USAID’s health aid would be averting 264 million DALYs per year. If they were managing ‘just’ 10 DALYs per $1000 - a plausible, though very uncertain, estimate for the good done by deworming treatments - they would be averting 80 million DALYs a year. How plausible are those figures? To us, they both sound like a lot. The number of child deaths per year reduced by 5 million between 1990 and 2011, which suggests around 250 million more DALYs averted per year due to all reductions in child mortality over the past two decades. Could US health ODA be responsible for the equivalent of all of this gain (at 34 DALYs per $1000) or a third of it (at 10 DALYs per $1000)? Or consider that the burden of disease in Africa is 375 million DALYs (SOURCE: p. 60 of 

). Could the US be averting the equivalent of 70% of this burden (34 DALYs per $1000) or even 20-25% (10 DALYs per $1000)? To put the plausibility of such achievements into context, US health ODA is roughly 10% of total health spending in low and low to middle income countries. Furthermore, some improvements in health probably come from improvements in incomes, agriculture, urban planning, education and so on, rather than health spending directly. US health ODA could have a disproportionately great impact by being better-targeted (as evidenced by the QWIDS database research cited in this article), and less likely to be lost through corruption, than other sorts of health financing. It it also possible that US health aid would achieve less than average due to expensive layers of bureaucracy and salary payments, political priorities, and a lack of democratic accountability to recipients. Overall we judge it unlikely that US health ODA is achieving as much for global health as is implied by a cost-effectiveness of 34 DALYs per $1000, but are less sure of the plausibility of 10 DALYs per $1000. (What do you think?)

Sally Murray

Articles

Thought the question of the most effective charity was settled? There are always more questions.

Is it good to campaign for increased government aid to cost-effective interventions? Part 2