Office Discussions: Is stunting reduction the best way to measure impact?

Since that I’ve joined Max Foundation as part of the impact measurement workgroup, I have focused my research on cost-effectiveness. It is safe to say that I leave with more questions than answers. This is a critical reflection of the literature surrounding the cost-effectiveness of interventions to reduce stunting in relation to Max Foundation’s programmes. 

Is stunting reduction the best way to measure Max’s impact as well as its cost-effectiveness?

Although stunting reduction in programme areas is an overarching goal of the Max Foundation, how much does measuring children under 5 reflect Max’s work? I found that Max’s interventions help improve child health in many ways beyond simply growth. In fact, if anything, it seems that growth is one of the less important health indicators when carrying out such a cocktail of interventions. Here is how I came to understand this issue and what potential avenues Max could explore to overcome this challenge. 

When diving into the topic of stunting reduction and cost-effectiveness, nutrition-specific interventions are often mentioned. Most nutrition focused interventions actively provide a specific nutritional supplement to children or pregnant women. In turn, this facilitates the straightforward measurement of the impact and cost effectiveness of a programme. This is clearly harder to achieve with integrated, cross-sector programmes. As a result, and despite the fact that integrated interventions are already implemented by some NGOs and recommended by many academics, they have largely been overlooked by cost-effectiveness studies.  

When it comes to integrated interventions as carried out by Max Foundation, pinpointing the impact on stunting of one particular activity is impossible and would actually be against the NGO’s rationale. However, cost-effectiveness analyses of programmes are often prospective calculations using real-life budgets stunting reduction rates produced by academic research. Applying these methods to Max’s interventions would mean merely adding up the effects linked to CLTS (Community Led Total Sanitation) interventions, complementary feeding education, homestead gardening promotion and growth monitoring (to name a few of the interventions Max foundation implements). One would therefore be likely to overestimate the impact (in terms of cost effectiveness) of Max Foundation in its Nutri-WASH programme. How then to measure the impact of Max?  

I would like to reflect on stunting as an indicator of child health. Stunting, measured as the height-for-age z-score of a child, is widely used across the sector as a key determinant of the overall health of children under 5, as well as an indicator for future well-being and cognitive development. Indeed, low height is not bad per se, what is problematic about it is its link to lower performance in school, lower wages and higher morbidity rates. Stunting measurement therefore acts as a good summary indicator and a predicator of human capital.  

However, a strict single focus on one measurement comes with the risk of hiding important trends. An important example in this respect is the trial in Early Childhood Development (ECD) analysed by Walker et al. (2005). Through psychosocial stimulation, it was found that the cognitive impairment in children induced by stunting was actually reversed, so much so that both cohorts of children had the same IQ by age 17. This example underlines the effect that a sole focus on stunting can have on the impact measurement of a child health intervention.  

So, what other health factors are positively impacted by Max Foundation’s work? The cocktail of interventions carried out by Max Foundation has the potential to decrease neonatal and infant mortality, especially through breastfeeding promotion, childhood diarrhoea reduction, maternal and infant anaemia reduction but also cognition impairment and much more. In turn, such improvements yield reduced healthcare costs for the beneficiaries as well as increased levels of schooling. Max’s work also involves a strong female empowerment component, which is not captured in any of the above indicators.   

So, although all of these intermediary indicators point to higher chances of healthy growth, positive development within these indicators does not necessarily mean infants will be taller (higher HAZ). If these improvements are not entirely captured in stunting reduction rates, it would underestimate the improved well-being of children in the intervention areas of Max Foundation. 

What have others done?  

FANTA, a USAID project, tries to circumvent these challenges by listing the different outcomes and benefits stemming from their integrated programming. Although they keep stunting as the central health concern to be addressed by the FANTA project, they also recognise and highlight the multiple benefits of such interventions both as intermediaries but also beyond stunting.

Borghi et al. (2002) accounts for the cost-effectiveness of a hygiene promotion intervention by calculating three different costs-per-outcome; namely the cost-per case of diarrhoea averted, the cost-per-consultation averted, cost-per-death averted and the cost-per-hospitalisation averted). Splitting the cost-effectiveness of a program in “costs-per-outcome” is a clear way of showing the cost-effectiveness of individual outcomes while also capturing their impact.

Are DALYs the best measure of cost-effectiveness?

Turning to dryer methodological considerations, I have to admit that I am still puzzled by the use of  Disability-adjusted life years (DALYs) in the measurement of cost-effectiveness. Both their calculation and meaning remain unclear. Issues include disability weights of different diseases arbitrarily imposed by the WHO to be universally applied. There also seems to be a general opacity of the techniques used to come to results presented in studies. In fact, even the aforementioned FANTA impact report was published without including DALYs and clearly stated that they were searching for a way to translate the results into this obscure scientific language. In this sense, DALYs seem out of reach for most of the development sector. I argue that this makes DALYs a technocratic tool, understood solely in tight developmental and scientific circles.  

Furthermore, using DALYs to compare the cost-effectiveness of interventions hides the goal of an intervention and its health outcomes. When analysing the mean cost-per-DALY averted for various interventions in the maternal and children health sector, I found that breastfeeding support programmes have, on average, one of the lowest cost-per-DALY averted. Providing clean delivery kits and training traditional birth attendants (TBAs) was over 8 times more expensive per-DALY averted. Very little can be inferred from these results, the notion that training TBAs is more expensive than promoting breastfeeding, is a rather intuitive assertion. Most interventions compared in the literature are ‘very cost-effective’ according to the WHO CHOICE threshold (An intervention is considered ‘very cost-effective’ when costing less than three times a country’s GDP per capita). If most interventions for maternal and child health reported in the literature are very cost-effective, then more needs to be done to compare them than the mere use of DALY estimates. 

I interpret my confusion around this concept as stemming from its use beyond its primary aim. Indeed, DALYs were developed to count the burden of different diseases in the world, and thereby help governments prioritise between different policies for the reduction of the most burdensome diseases. DALYs were not intended for NGOs to assess their impact on the ground.  Considering that they are prone with large uncertainties, it seems that DALYs work better as aggregated sums of global trends: giving rough estimates, rather than use at the micro-level for specific diseases and complex interventions.  

What is still needed?

I would conclude in suggesting that more research could be done to develop appropriate methodologies to measure the wide impact integrated adaptive approaches yield on the ground. RCTs ‘gold standards’ and simplistic hegemonic ‘best practices’ in cost-effectiveness should be challenged and new approaches, such as the one adopted as part of the FANTA intervention, should be put forward. These should pave the way towards cost-effectiveness as a useful tool accounting for the complex realities NGOs face in implementing interventions.  

 

 

Back to news overview