Why and how Giving What We Can evaluates evaluators

A major focus of Giving What We Can’s research team is evaluating charity evaluators and grantmakers — this page explains what this is, why this is our focus, and how we do it.

We want to make the best possible recommendations to donors looking to maximise their impact. To be able to serve donors with a variety of values and starting assumptions, and to cover as many causes and charities as possible, we rely on third-party expert evaluators for our recommendations and grantmaking, rather than evaluating individual charities ourselves. We evaluate evaluators and their methodologies so that we can always rely on the highest-quality and most up-to-date recommendations available across a range of causes.

What does it mean to evaluate evaluators?

We evaluate evaluators to decide which evaluators:

  • We ask to advise the grantmaking for our cause area funds — our default recommendations for most donors.
  • We rely on for our charity and fund recommendations.

At a high level, we do this by:

  • Forming an understanding of the strengths and weaknesses of a respective evaluator and its approach to charity evaluation and grantmaking.
  • Deciding whether and how we defer to the evaluator (ideally in direct comparison with at least one other impact-focused evaluator working in the same cause area).
  • Transparently explaining and justifying that decision to donors.

In late 2023, we conducted the following six evaluations:

Why evaluate evaluators?

We think it’s the best way we can help donors maximise their impact.

We don’t have the capacity to evaluate individual charities ourselves — there are far too many for just one research team to cover, more than a million in the US alone! — so we need to turn to other expert evaluators and grantmakers focused on impact. By our count, there are now over a dozen impact-focused grantmaking and charity evaluation organisations, some of which provide different charity recommendations in the same cause area. This leaves us, other effective giving organisations, and donors with an important choice on whose recommendations to follow.

Before 2023, we made this choice based on factors like the public reputation of an evaluator in the effective giving ecosystem, and whether its stated approach seemed to broadly align with our donors’ goals. But we wanted to do better, and thought it would be valuable to provide donors with more information about evaluators.

Beyond making our recommendations to donors, we think there are several extra benefits to evaluating evaluators:

  • As above, we’re not the only effective giving organisation that relies on evaluators for their recommendations. We think our work can also help others make more informed decisions about which evaluators to defer to.
  • We think we can provide value to the evaluators themselves, by providing them with independent feedback on their work.
  • These kinds of evaluations can improve the incentives for existing and new evaluators to deliver the best work they can.

There are substantial limitations to our first iteration of this project, which we did in 2023, but we nevertheless think that this is a significant improvement on the status quo, in which there were no independent evaluations of evaluators’ work. We discussed some of our concerns with this status quo when we first announced our research direction at the end of 2022.

How we evaluate evaluators

In this section, we highlight:

  • Some of the principles we use to guide our evaluations.
  • How we chose which evaluators to evaluate in 2023.
  • Our process for our 2023 evaluations.
  • The limitations of our approach.

Our principles

As we do with our impact evaluation, we aim for usefulness, transparency, and justifiability, rather than comprehensiveness and procedural uniformity. Put another way, we aim to transparently communicate how we use our judgement to find the areas we think are most useful to investigate, to come to a justifiable decision on whether and how to defer to an evaluator. Some implications of this approach include that we are flexible in what we choose to investigate (making each evaluation different) and open to stopping an evaluation once we feel able to make a justifiable decision.

We also aim to avoid surprises for evaluators by alerting them of our thinking throughout the process. This is in part because we want to work with evaluators to understand and improve their approach, rather than just judging them, and also because we value their expertise.

How we chose which evaluators to investigate in 2023

In our first iteration of this project in 2023, we looked into six different evaluators across three high-impact cause areas:

How we chose the cause areas we investigated

We think of a “high-impact cause area” as a collection of causes that, for donors with a variety of values and starting assumptions (“worldviews”), provide the most promising philanthropic funding opportunities. Donors with different worldviews might choose to support the same cause area for different reasons. For example, some may donate to global catastrophic risk reduction because they believe this is the best way to reduce the risk of human extinction and thereby safeguard future generations, while others may do so because they believe the risk of catastrophes in the next few decades is sufficiently large and tractable that it is the best way to help people alive today.

Because of our worldview-diverse approach, we chose to evaluate evaluators in the three cause areas we think contain some of the most cost-effective funding opportunities across a broad range of plausible worldviews (rather than taking a view on how impact varies across these cause areas). As a research team, we think we can add most value within a cause area, whereas donors can decide for themselves which cause areas best align with their worldview.

Our choice of these three cause areas (global health and wellbeing, animal welfare, and reducing global catastrophic risk) has been informed by global priorities research from organisations like Open Philanthropy, the Global Priorities Institute, and (in the past) the Centre for Effective Altruism.

There are some promising philanthropic cause areas that we did not (yet) include (such as climate change). We intend to keep evaluating new cause areas and evaluators to add further recommendations, provided we find a strong enough case exists that, from a sufficiently plausible worldview, a donor would choose to support those cause areas over other options.

How we chose the evaluators we looked into

We explain the reasons for choosing each evaluator within each evaluation report. Among other reasons, in 2023, these choices were informed by:

  • A survey we conducted among 16 effective giving organisations (made up primarily of the larger national fundraising organisations listed here), on which evaluations would be most useful for them.
  • Where our donors give — we wanted to prioritise evaluators whose funds and charity recommendations our donors were currently supporting the most, so that the results would be most useful to them.
  • Our previous selection of “trusted evaluators” — we wanted to prioritise evaluators whose research had informed our previous recommendations.
  • Our decision to evaluate exactly two evaluators per cause area — we wanted to be able to compare where feasible, but also wanted to prioritise the quality of our initial investigations over the quantity of them, given the limited time we had available (which was also the preference of the 16 other effective giving organisations we surveyed).

The choice of which evaluators to prioritise affects our overall recommendations for 2023. For example, because we have not yet evaluated Founders Pledge, we have not used its research to inform our recommendations so far. This lack of comprehensiveness is one of our project’s initial key limitations. We try to partially account for this by highlighting promising alternatives to our recommendations on our donation platform, and providing resources for donors to investigate these further.

Our process for our 2023 evaluations

As discussed above, a key goal for our evaluations project was to decide which evaluators to rely on for our recommendations and grantmaking. We were additionally interested in providing guidance to other effective giving organisations, providing feedback to evaluators, and improving incentives in the effective giving ecosystem.

For each evaluator, our evaluation aimed to transparently and justifiably come to tailored decisions on whether and how to use its research to inform our recommendations and grantmaking. Though each evaluation is different — because we tried to focus on the most decision-relevant questions per evaluator — the general process was fairly consistent in structure:

  1. We began with a general list of questions we were interested in, and we used this list to generate some initial requests for information.
  2. After receiving this information, we tried to define decision-relevant cruxes of the evaluation: the questions that, when answered, would guide our decisions on whether and how to rely on the evaluator for our recommendations and grantmaking. These differed for each evaluator. In some cases, they were quite broad (e.g., “Do the Animal Welfare Fund’s review and grading processes look like they are tracking marginal cost-effectiveness?”), and in others, this was a set of more concretely defined research questions (as was the case for Happier Lives Institute).
  3. We shared these cruxes along with some additional (more specific) questions with evaluators, asking for feedback — in some cases, changing our cruxes as a result.
  4. We then investigated these cruxes — asking further questions and iterating on our overall view — until we felt confident that we could make a justifiable decision. We intermittently tried to share our thinking with evaluators so that we could receive feedback and new information that would help with our investigation.

Limitations of our evaluations

Our 2023 evaluations had various limitations, which are detailed in each evaluation report.

Several limitations apply to the project as a whole, some of which we’ve discussed above as well:

  • We didn’t (yet) investigate several promising evaluators, such as Founders Pledge or Giving Green, in addition to deciding not to complete our investigation of Happier Lives Institute. Relatedly, we didn’t (yet) investigate several promising cause areas, such as climate change or effective giving meta organisations.
  • More generally, this project will not allow us to cover the entire charity space: there aren’t evaluators in every promising cause area, and evaluators are generally far from able to cover all promising charities within the cause areas they focus on. (However, we think this project is the most efficient way for us to cover as large a part of the high-impact charity space as possible in a high-quality manner, and we provide donors with various promising alternative options if they want to have more choice and delve deeper themselves.)
  • The quality of our recommendations is highly dependent on the quality of the charity evaluation field in a cause area, and hence inconsistent across cause areas. For example, the state of charity evaluation in animal welfare is less advanced than that in global health and wellbeing, so our evaluations and the resulting recommendations in animal welfare are necessarily lower-confidence than those in global health and wellbeing.

There are also a few limitations that were present across all or most evaluations we conducted this year:

  • In many cases, we had conflicts of interest that we had to navigate. We don’t think these fundamentally challenged our position to do the evaluations — and generally think some conflicts of interest are nearly unavoidable in the still very small charity evaluation space — but we do think they matter and that it is important to be transparent about them.
  • Many evaluators were not set up to be externally evaluated, and we didn’t always have full access to all the relevant information.
  • We generally did not have extensive in-house domain expertise in the evaluation areas we looked into. We tried to partially account for this by asking external reviewers for input, and by keeping most of our evaluation focused on evaluation and grantmaking aspects where only limited subject-specific expertise was needed.
  • We were highly time-constrained in completing this first iteration of our project. On average, we spent about 10 researcher-days per organisation we evaluated (i.e., two researchers spending one full workweek), of which only a limited part could go into direct evaluation work — a lot of time goes into planning and scoping, discussion on the project as a whole, communications with evaluators, and so on.

Given these limitations, we aimed to:

  • Focus our efforts where they would be most useful, by prioritising considerations that were relevant to our recommendation decisions.
  • Only make decisions we thought we could justify, and highlight our uncertainties where we didn’t feel confident enough to make a decision.
  • Be as transparent as possible in our reasoning in each report, without breaching any confidentiality agreements we made and limited by our time constraints.

Even with our efforts to take an approach that prioritises transparency, justifiability, and usefulness, we appreciate there still are significant limitations to our evaluations, and see the first iteration of this project as a minimum-viable-product version which we look forward to improving on in future iterations. However, as mentioned above, we think doing these evaluations represents a significant improvement to the previous situation, in which there were no independent evaluations of evaluators’ work we (or donors and other effective giving organisations) could rely on.

Bottom line

Rather than evaluating individual charities, since 2023, Giving What We Can evaluates which third-party expert evaluators donors can best rely on to maximise their impact. This allows us to make even higher-quality fund and charity recommendations to donors with a wide variety of values and starting assumptions. We think this represents a big improvement over how we previously chose which evaluators to work with — based on rough heuristics— even if it still has limitations. It has also facilitated the launch of our cause area funds, which present a reliable default option for donors who want their money to be allocated according to our latest research.

Over time, we want to expand to more cause areas and evaluators, go more in-depth where it's useful, and keep refining our process based on feedback. Most importantly, we'll keep focusing on empowering donors and collaborating with evaluators to help donors have the biggest impact. We're grateful to all the evaluators who worked with us on this project so far, and look forward to continuing to improve together.

For those who would like to see our current giving recommendations, check out our best charities page. For the full selection of programs Giving What We Can supports, see our donation platform.