Eliz Kilich

Some questions on the cognitive benefits of deworming

A promising study on the long-term effects of deworming

Currently Giving What We Can supports two charities working on deworming – the well-established 

, which primarily targets Schistosomiasis. and the promising but less established 

, which aims to enable countries to implement deworming programs relating to Soil Transmitted Helminths.

In October 2014 GiveWell suggested that a new unpublished 

, by Croke (2014, 'The long run effects of early childhood deworming on literacy and numeracy: Evidence from Uganda'), provided promising data on the long-term effects of deworming, indicating that it represented a major update in this field of research. The cluster randomised trial compared the literary and numeracy test scores of children 7-8 years after mass deworming (22%-66% of the children in the parishes) for soil-transmitted helminths of pre-school children in Eastern Uganda parishes to the scores of a control population which had not undergone mass deworming during this time period. The author found improved test scores in those treated with deworming, and concluded that this strengthens the evidence for cognitive benefits for mass deworming in areas of high worm prevalence. Whilst appreciating the strengths of the study, as highlighted by Jake Marcus in 

, I analysed some of the limitations of the study.

Cluster vs. Individual Randomised control trials

One of the main criticisms of Croke’s study lies in the study design adopted - a cluster randomised trial. A cluster randomised trial is used when one cannot randomise individual participants, so instead randomises groups. For instance, a cluster randomised trial might look at the effects of different teaching styles (A & B) on school classes. This is appropriate because school teachers are not individual tutors but can only provide for the collective when giving a lesson. Since the single intervention affects a large group and it cannot be specifically directed at individuals, the important outcome is the effect on the whole class.

The outcome being measured in the Croke study is collective improvement in cognitive ability, which may seem reasonable. However the use of cluster randomisation is less intuitive here because children are individually being dewormed. Therefore randomising individuals would be a better study design.

On the other hand, cluster randomisation could account for any spillover effect - deworming having additional benefit to children in close proximity to those dewormed. Individually randomised trials cannot take into account spillover effect, so they could underestimate the effect of deworming. Further, randomising individuals would have been substantially more expensive.

Nonetheless, by being unable to follow up participants directly as individuals and instead relying on population-based data, the study is subject to a greater propensity for confounding factors to influence the results.The propensity for confounding in cluster randomised trials is based on baseline differences and on the hidden effects of unmeasured changes to the groups during the study. For instance, the two groups may differ substantially in baseline characteristics which could affect outcome. In the example above, if children in class A are more clever than those in class B, finding that class A improved after a different teaching style is confounded by the children being more amenable to learning.

Croke tries to control for potential confounders by showing that there is no difference in socioeconomic status amongst groups. But there is a flaw in this method. The socioeconomic data (which is collected independently from the study and only includes 22 of the 48 parishes in the original study) does not apply to the original sample (intended for deworming or control) at the beginning of the study. So it only shows that socioeconomic differences were not present at the end of the study (2010) when the cognitive ability studies were performed. Whilst it is plausible that there were not socioeconomic differences at the beginning of the study, this has not been directly shown in his paper.

When considering the data Croke acknowledges the limits of a cluster randomisation and so applies analysis to show that his data is “robust” to obvious potential confounding factors. Croke attempts to show “robustness” by analysing the data where certain factors were controlled, such as restricted water access, along with other measures. This involves checking if the results still stand despite standardising a potential confounding variable between the two groups.

For instance, if one was looking at death rates between two places in the country they may find that in place A this is much greater than place B. However this could be because of the confounding factor that the majority of people living in A are much older than B. Therefore one can standardise the data to compare all those in the same age range categories to test if that result still stands despite the differences in average age being controlled in post data analysis.

In Croke’s study, one way in which robustness of data is illustrated is by showing that the youngest and oldest children in the test classes would have smaller differences in test scores than the intermediate age groups, as they would not have been in the population eligible for deworming 7 years earlier. Whilst intuitive that those at the extremes of age would have had reduced number of deworming protocols, and so may have less differences in cognitive ability, this has actually not been evidenced. The graph provided shows no confidence intervals and did not indicate any statistics used to compare the subgroups, so cannot be assessed appropriately.Even if those statistics were used, the study was not powered to detect differences in this subgroup population. In addition, standardising the data cannot account for confounders that are not tested for and may be less obvious.

Further to these limitations, the study assumes that the populations surveyed during deworming and 7-8 years later are comparable,the latter containing the population that was originally dewormed. However, 

 revealed that a total of 3.1 million out of 23 million persons (13 percent) were enumerated outside their district of birth and hence classified as internal migrants.3 This data suggests that the population sampled in the parishes randomised for deworming or control may not correspond to the populations from which the cognitive function data were collected.

Another issue is that the endpoints for the study – differences in standard error of Maths and English tests - are limited measures of cognitive ability. Further, there is no reasonable suggestion as to why deworming would greater effect Maths tests than English tests, as the results indicate. It is also unclear what these differences actually correspond to in absolute terms, on a practical level for the populations analysed. In addition, other studies have shown increased school attendance, which this study fails to show. The reason for this inconsistency is unclear.

Finally, crucial to the analysis is an absence of evidence for combination deworming - there is no data on whether Schistosomiasis deworming occurred in the interim between Soil Transmitted Helminth deworming and cognitive testing of the populations, or on what other public health interventions occurred.

In conclusion, the study has significant limitations. But this should be seen in the context of the wider difficulties of researching the effects of mass deworming. Cognitive benefits are difficult to measure, especially over significant periods of time. This is made even more difficult when looking at substantial benefits to groups (such as spillover effects), as studying group data introduces various confounding factors. Functional outcomes and the ability of improved cognitive abilities to dramatically affect the prospects of a child in this context are difficult to demonstrate, as abundant difficult to measure factors contribute. These difficulties explain why controversy in the field of deworming is rife. This study has significant limitations with data subject to confounders and therefore it may not be as “promising” as we hoped. Further blog posts regarding how we might re-evaluate the evidence will be posted soon.

Hilleman Laboratories: A potential answer to the inequity in vaccine need and vaccine supply?

Vaccines have delivered enormous health gains by eradicating some of the worst diseases confronted by humanity. As a result Giving What We Can has been looking into biomedical research projects as potentially high-leverage opportunities for donors. As part of our investigation we spoke with Hilleman Laboratories, a research group operating in new Delhi, India that tries to adapt vaccines to make them suitable for widespread use in the developing world. We concluded that thanks to seed capital from some large pharmaceutical companies they did not have any need for additional funding from our members. However, Dr Davinder Gill, Hilleman Laboratory's Chief Executive Officer kindly made time to talk to Eliz Kilich of Giving What We Can about their research and what 'funding gaps' remain in vaccine research.

What is necessary for a biomedical project to be worthwhile?

I think there are three essential components to a project being worthwhile. Firstly, there must be a public health need. Secondly, it is essential that the appropriate resources and networks are available and can be utilised effectively. The project needs to be financially viable. The third component to a successful project is organisation. The skills, competency, experience and potential outreach of the organisation are critical. There must be a clear goal – a point of completion.

Why have we struggled to produce some vaccines in the past despite substantial investment?

Investment has mostly taken place in the developed world. This does not reflect the areas where the burden of diseases such as TB, malaria and HIV are highest. Therefore vaccines have been developed largely for the needs of the developed world. The issues of access and affordability have not been addressed. There is a clear difference in equity in the developed world and developing world.

In terms of why investment has not directly yielded results is largely because the research is inherently “risky.” There is no real guarantee of success. HIV, malaria, TB are extremely complex diseases meaning we are still unsure which strategy to use when developing vaccines.

Which diseases or approaches are currently neglected?

This includes some of the world’s tropical diseases such as Dengue, Chagas Disease and Leishmaniasis. Respiratory and diarrhoeal diseases are also a major health burden in the developing world with few affordable vaccines available. This will include pneumococci, rotavirus and other gastroenteritis-causing organisms.

In terms of approaches that are neglected, I would say the technical aspects, such as altering the formulation, packaging and delivery method, have been overlooked in the past. The current methods are not optimized for the developing world. For instance, Rotavirus Vaccine licenced in 2006 is in liquid form and needs to be refrigerated in order to thermostabilise the vaccine. If delivery methods can be improved the costs could be kept very low.

What are the main goals of Hilleman Laboratories and what makes it such a good project?

Our goal is to bridge the translational gap between research and development of vaccines to suit the needs of the developing world. We have a unique and strong 50:50 partnership between the Welcome Trust and Merck. Also, by being based in India we have a strong infrastructure to leverage, because the country has a long tradition of actually making vaccines. At the moment we are still in preclinical research but aim to test our products in clinical trials by next year.

What benefit would the vaccines you plan to develop offer?

The main obstacle to the access of vaccines in the developing world is pricing. The vaccines for such diseases (as mentioned above) cannot be afforded by the developing world. Despite organisations such as GAVI subsidising the prices from $50 to about $5 a dose, this is still unaffordable for the broader needs of the developing world. Hopefully, the optimised vaccine will reduce the cost to $1 a dose or less. This saved procurement cost would be added to the saved distribution cost of vaccines by reducing wastage.

Vaccines need to be kept cold otherwise their effectiveness can be reduced a lot and their shelf life is shortened. Vaccines cannot be adequately stored in the cold in tropical climates where transport is difficult. Therefore a thermostable vaccine would impact to reduce the cost dramatically. These reduced expenses for delivery and disposal would reflect in the price of the vaccine. By being manufactured in the developing world the operational cost would also be kept low, allowing the price of the vaccine to be affordable.

What are the pitfalls such research can fall into?

The obstacles include timing. It is important that stakeholders, that will be crucial in the implementation of the vaccine, are brought into the development and conversations early on. Not being fast enough to deal with local agencies, government and distribution could hinder progress. This is because we would be in a situation where technically the problem of vaccine design is solved but we would have no one to take on the vaccine to adapt and implement it.

What is the future for Hilleman Laboratories?

Hilleman Laboratories has a strong and committed mission to address the major needs in global health. As a strong partnership and as a global company we are well-connected to global health policy makers and agencies. Our collaborative model, network and technical innovation provides us with the resources to make our long term goals successful.

Eliz Kilich

Articles

Some questions on the cognitive benefits of deworming

Hilleman Laboratories: A potential answer to the inequity in vaccine need and vaccine supply?