A review of research methods used to study specialised nutritious foods
By Kristine Caiafa, Maria Wrabel, Devika Suri, Ye Shen, Shelley Walton, Beatrice Lorge Rogers and Patrick Webb.
All authors are affiliated with the Food Aid Quality Review (FAQR) project at the Friedman School of Nutrition Science and Policy at Tufts University.
The authors are grateful to the U.S. Agency for International Development (USAID) and the USAID Office of Food for Peace (FFP) of the Bureau for Democracy, Conflict and Humanitarian Assistance (DCHA) (AID-OAA-C-16-00020) for funding this activity, and to Breanna Langlois for her invaluable contributions to the development of this manuscript.
What we know: Research on specialised nutritious foods (SNFs) has increased in the past two decades, but this has not resulted in commensurate advances in policies and practice.
What this article adds: The goal of this exercise was to identify key factors in SNF study methodologies that could be strengthened to develop a more rigorous evidence base. A search of the literature and clinical-trial registries was conducted to identify studies using SNFs to influence anthropometric outcomes, and information about the research methods used was collected. Among the 114 studies identified (89 published and 25 ongoing), impediments to a robust evidence base included research bias, heterogeneous study design and insufficiently reported study details. A list of specific actions was developed to be taken by global agencies, research funders, researchers and practitioners to build a higher-quality evidence base for translating research on SNFs into policy and practice. Supplemental tables are included in an online version of this article.
Specialised nutritious foods (SNFs), which include lipid-based nutrient supplements (LNSs), ready-to-use therapeutic foods (RUTFs), ready-to-use supplementary foods (RUSFs), fortified blended foods (FBFs), micronutrient powders (MNPs) and locally produced analogs of these products, are food products specially formulated to treat, prevent or mitigate undernutrition. Scientific research on SNFs has expanded rapidly in the past two decades, driven by an intent to improve nutrition outcomes. While much has been learned, global practice standards for using SNFs as a class remain elusive. The challenges inherent in studying these products and a lack of aggregate emphasis on study quality has generated an evidence base considered in recent reviews to be of low or moderate quality (Webb, 2015; Lazzerini et al, 2013; Schoonees, 2013).
An excellent model for how to move forward can be found in the methods used to develop standards for the management of acute malnutrition (WHO, 2013; WHO, 2012). These global policies were made possible by corralling a robust evidence base, largely through the use of systematic reviews. Similar evidence synthesis for SNFs would require high-quality studies; i.e., using designs that are appropriate to the research question and which mitigate risk of bias and threats to validity, and that are collectively similar enough in study characteristics so that findings are comparable. Research generalisability, or applicability to larger populations from which a study sample is drawn, is also critical to this type of evidence synthesis.
The aims of this review are threefold: first, to identify common methods used in a sample of SNF research; second, to highlight the methods that influenced quality, comparability and generalisability; and third, to propose actions for a stronger evidence base.
This review included studies based on two factors: 1) the study used an SNF in at least one intervention arm, and 2) the study specified at least one anthropometric outcome. Studies were not excluded based on characteristics of participants, comparisons or study design. Studies published between January 1 2011 and April 1 2017 were identified through searches conducted in English of PubMed and Web of Science in April 2017. These were compiled and screened for inclusion by one analyst; first by title, then abstract, then content. A second analyst independently reviewed any studies in question, first by abstract and then by content. Eighty-nine publications were identified for this review.
Ongoing trials were identified in December 2017 using REFINE (Research Engagement on Food Interventions for Nutritional Effectiveness; www.REFINEnutrition.org), a public platform that maps SNF research. Twenty-five ongoing studies were identified by one analyst, who recorded information from trial registrations and publicly available study protocols.
In preparing this manuscript, the authors followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist (Higgins and Altman, 2008) to the extent applicable.
Findings and discussion
Description of studies
Overall, we reviewed 89 publications and 25 ongoing studies (114 studies total). Based on terminology used in each study, predominant outcomes of interest among studies treating acute malnutrition (n=60) were recovery (as defined by the study) and weight gain. Other studies included stunting as a categorical variable (61%; n=33); height/length-for-age (included in 72%; n=39); wasting as a categorical variable, defined by either mid-upper arm circumference (MUAC) or weight-for-height z-score (WHZ) (44%; n=24); weight-for-age (59%; n=32); underweight as a categorical variable (31%, n=17); linear growth (39%, n=15); birth outcomes (17%, n=9); and biomarkers of undernutrition (20%, n=11). Most studies (87%, n=99) targeted children under five years of age, with focus on multiple sub-groups in that age range.
Eighty per cent (n=92) of studies were intervention studies, in which investigators assigned (randomly or not) participants to receive a specific intervention. Of these, most (92%, n=88) had multiple study arms. Twenty per cent (n=22) of studies were observational, in which investigators observed the outcomes of participants who had received an intervention not assigned by the investigators.
Studies typically explored one of three questions:
- Is a given SNF effective at improving nutrition-related anthropometric measures?
- Which SNF is comparatively more effective than other SNF(s) at improving nutrition-related anthropometric measures?
- Do complementary activities in addition to SNF improve nutrition-related anthropometric measures?
To effectively answer these questions, researchers must be transparent about research design limitations and interpret findings only in a way that the study is designed to answer. For example, single-armed research (which accounted for 18% of all studies) can offer valuable hypothesis-generating information, but is relatively limited in answering the above questions.
We assessed study methods against the forms of bias included in the Cochrane Collaboration’s study quality evaluation protocol (Higgins and Altman, 2008; Ryan and Hill, 2016; Schünemann et al, 2013):
Selection bias – Among intervention studies (n=92), 42% (n=39) passively recruited participants (i.e., drew from community screenings or enrolled self-selecting individuals), 39% (n=36) actively recruited participants from a target community, and 8% (n=7) randomly sampled participants from communities. Although passive recruitment is decidedly convenient, it can introduce selection bias insofar as some segments of a target population may be unintentionally but systematically excluded due to exogenous characteristics, leading to baseline differences in the groups that are compared. Consequently, enrolled participants may also not be representative of the underlying population (which also impedes generalisability). Active recruitment, in which study investigators directly recruit participants, can reduce this bias.
Performance and detection bias – Of multi-armed studies, less than half (47%, n=41) used a form of blinding or blinded outcome assessors or data analysts (43%, n=38). Not blinding the treatment, enumerators or analysts can lead to awareness of the treatment group that might cause altered outcomes (due, for example, to more careful SNF preparation, adherence to feeding regimens or weight measurements). To this point, differences in SNF packaging, appearance, taste, texture and preparation should be as discreet as possible, and participants, researchers and staff should be blinded to the treatment and non-treatment groups to the extent possible. At a minimum, data analysts should be blinded to intervention groupings.
Most multi-armed studies did not note whether or how intervention implementation differed between arms. To mitigate performance bias, or differences between study arms relating to how interventions are carried out, study arms should be implemented using the exact same methods, except for the elements designed to differ. For instance, SNFs should be distributed in the same time increments (e.g., monthly) and enumerators should be identically trained and supervised.
Attrition bias – To avoid bias, study attrition and differential rates of attrition between arms should be expected and accounted for in study planning and analysis. Most publications reported some degree of attrition (52 out of 68 publications reported baseline and endline sample sizes); however, only three explicitly reported differential attrition. Determining whether attrition rates differ based by age, sex, SNF or distance from treatment centre can inform whether there was any bias related to those who remained enrolled compared to those who did not.
Reporting and publication bias – Twenty-one per cent of all studies did not state pre-specified research outcomes, introducing potential bias regarding whether outcomes have been selectively reported to alter the conclusions drawn. Reports should: i) state all pre-specified primary and secondary outcomes; ii) examine those outcomes; and iii) present all pre-specified outcome data, regardless of positive, negative or null results. Reporting and publication bias can be further mitigated by reporting every research activity to a trial registry. Keeping information updated upholds high standards of transparency and accountability to participants.
Study comparability was impeded by inadequately detailed study design information, as well as inconsistent parameters and definitions. A subset of studies on severe acute malnutrition (SAM) treatment illustrates the lack of comparability in this sample. Among these (n=28), there were six similar (but not identical) target age groups: “under 6 months,” “6-23 months,” “6-24 months,” “6-59 months,” “6-60 months,” and “6 months to 5 years.” A child treated from “6 months to 5 years” could mean that child receives the intervention from six months until their fifth birthday or until any timepoint within their fifth year; without more explicit cut-off information or commonly-agreed definitions, it is impossible to know. Outcome variables, while also generally aligned, were not equivalent. Although 57% (n=16) identified the primary outcome as “per cent of children recovered from SAM,” different definitions of “recovery” were used: seven studies based their determination of recovery on MUAC, compared with nine that used WHZ.
To enhance comparability, it would be useful for all studies to report on all relevant outcomes. Direct comparability is impossible; differing resource constraints, geography, infrastructure, political environments and social factors require (and even favour) flexibility in SNF programming choices. Nevertheless, research leaders are well positioned to establish a set of “reference cases” like those developed for economic evaluation of health technologies in low- and middle-income countries (Wilkinson et al, 2016). For instance, for studies considering the effect of an SNF on wasting recovery, there would be a standardised set of research designs with acceptable definition and indicators for “recovery”. The intention should be to reach a minimum level of research alignment for study comparability, while still enabling creative scientific inquiry.
Generalisability requires representative sampling of the population of interest. Sample size should always account for variability in the target population and be large enough to draw conclusions about that population with a specified level of confidence. Random selection and assignment of participants allows researchers to verify that observed effects are not a result of differential characteristics of the groups being studied. In this sample, random assignment was common: of intervention studies with multiple study arms (n=88), 92% (n=81) randomly assigned interventions to either clusters (42%, n=37) or individuals (51%, n=45).
Ideally, a study investigating the effects of an SNF on the incidence of moderate acute malnutrition (MAM) recovery in a specific country would randomly select children with MAM from across the country. In practice, however, such defined sampling frames do not exist and implementing an experimental intervention at the national level would not be feasible – and it would be impossible to monitor these programmes closely. Cluster randomisation, in which interventions are assigned to clusters rather than individuals, is an approach that allows clinic and community-centred feeding programmes to continue to operate with minimal disturbance. Participants can be sampled in a way that is representative of the target study population in terms of demographic, geographic and other relevant community characteristics; blinding can be addressed; and the risk of treatment crossover is greatly reduced (Friis et al, 2015).
Reporting methods predetermine the quality and utility of an evidence base. In this sample, study methods were often not reported with the capacity for replicability, comparability and external assessment of quality. Twenty-five per cent (n=28) of publications did not provide SNF nutrient composition; 72% (n=82) of all studies did not specify SNF dosage; and 69% (n=79) did not state dose distribution frequency (e.g., weekly, monthly, etc.). Of published intervention studies, 94% (n=85) did not report study-arm sample sizes at both baseline and endline and 26% (n=66) of publications did not report statistical power calculations. As information about these important design elements is critical to assess bias and replicate studies, researchers should always use accepted checklists when preparing reports1.
Reporting also has implications for systematic reviews, which are widely used tools for decision-making. To be included in a systematic review, a study must first include the relevant search terms, then meet narrowly defined inclusion criteria assessed through reported study methodology. We observed inconsistent terminology for common potential search terms, like ready-to-use foods (RUF) and lipid-based nutrient supplements (LNS). To avoid study exclusion due to nominal differences, norm-setting bodies (such as the United Nations organisations, academic institutions, and prominent, consortium-based communities of practice such as the State of Acute Malnutrition and the Food Security and Nutrition Network) should standardise the language around SNFs and nutrition, beginning with product terminology.
Despite the many challenges inherent to SNF research, there are opportunities to support a stronger research practice and evidence base. A global body responsible for coordinating SNF research would be ideally placed to develop and institutionalise specific guidance on SNF research methods, supporting both researchers and funders in their efforts to produce rigorous evidence. In the absence of such a body, we suggest a list of actions as a starting point for discussion and revision (Box 1). We hope that this is a first step for decision-makers in this community to take this effort forward and own a more active role in coordinating SNF research to inform global policy.
For more information, please contact Maria Wrabel at email@example.com
Box 1: Actions for SNF research
Actions for normative agencies
- Establish reference cases for commonly studied topics in food aid research to improve how studies are conducted and reported.
- Establish standards for terminology and corresponding abbreviations.
Actions for research funders
- Require research awardees to identify all the components listed under Study reporting, below.
- Require research awardees to follow relevant reporting checklists.
- Require research awardees to register trials in a trial registry and keep the information up-to-date.
Actions for researchers and research staff
Study design and analysis
- When possible, actively recruit study subjects.
- Always blind data analysts to the intervention group. To the extent possible, blind participants, researchers, intervention implementers and outcome assessors to the different interventions and intervention groups.
- Anticipate attrition and account for it in the sample size calculation. Record attrition overall, across and within study arms, and investigate differential attrition rates by participant and intervention characteristics.
- Identify all outcomes of interest at the outset of the study and investigate all of them.
- Use the same intervention methods in all study arms.
- Use random selection and assignment of study participants. When appropriate, consider randomised cluster-based sampling.
- Register every trial with a trial registry. Keep the information up-to-date.
- Follow accepted research-reporting checklists.
- Describe the study with the intention for replicability, comparability and external assessment of quality. Include:
- Study details: year(s) trial took place; country in which trial took place; context (emergency/protracted emergency/development; rural/urban/semi-urban); target population; nutrition problems studied; all planned outcomes of interest; outcome indicators.
- Study design: intervention (randomised controlled trial, cluster-randomised control trial, non-randomised study, etc.)/observational (cross-sectional, retrospective, prospective)/ descriptive; inclusion of control group or comparator; all blinding used.
- Participant selection and sampling strategy: inclusion criteria; exclusion criteria; active or passive recruitment; randomisation strategy (clustered, individual, non-randomised); total sample size; sample size per intervention arm; sample size and power calculations.
- Intervention assignment: parallel; factorial; single group; crossover.
- Intervention design: SNF studied; nutrient composition of SNF; dose provided; frequency of product distribution; total number of distributions over the course of the intervention; duration of intervention; implementation methods used to carry out study arms, noting differences; descriptions of how foods were prepared or how instructions were given to prepare foods.
- Analysis: statistical methods used; assessment of baseline comparability across study arms; detectable effect sizes and an explicit comparison to actual sample size; desired and actual sample size; attrition rates overall, across and within study arms; possible explanations for attrition.
- Outcomes: results for all pre-specified outcomes.
1CONSORT 2010 checklist www.consort-statement.org/media/default/downloads/consort 2010 checklist.pdf; STROBE checklist www.strobe-statement.org/fileadmin/Strobe/uploads/checklists/STROBE_checklist_v4_combined.pdf; MOOSE checklist www.elsevier.com/__data/promis_misc/ISSM_MOOSE_Checklist.pdf and SQUIRE www.squire-statement.org/index.cfm?fuseaction=page.viewpage&pageid=471
Friis H, Michaelsen KF, Wells JC. Choice of Design and Outcomes in Trials among Children with Moderate Acute Malnutrition. Food Nutr Bull. 2015 Mar;36(1_suppl1):S35–40.
Higgins JP, Altman DG. Assessing risk of bias in included studies. Cochrane Handbook for Systematic Reviews of Interventions: Cochrane book series. 2008:187-241.
Lazzerini M, Rubert L, Pani P. Specially formulated foods for treating children with moderate acute malnutrition in low- and middle-income countries. Cochrane Database Syst Rev [Internet]. John Wiley & Sons, Ltd; 2013; Available from: http://doi.wiley.com/10.1002/14651858.CD009584.pub2
Ryan R, Hill S. How to GRADE the quality of the evidence [Internet]. 2016. Available from: http://cccrg.cochrane.org/sites/cccrg.cochrane.org/files/public/uploads/how_to_grade_revising_1_ december_2016.pdf
Schoonees A, Lombard M, Musekiwa A, Nel E, Volmink J. Ready-to-use therapeutic food for home-based treatment of severe acute malnutrition in children from six months to five years of age. In: Volmink J, editor. Cochrane Database of Systematic Reviews [Internet]. Chichester, UK: John Wiley & Sons, Ltd; 2013. Available from: http://doi.wiley.com/10.1002/14651858.CD009000.pub2
Schünemann H, Brozek J, Guyatt G, Oxman A. Handbook for Grading the Quality of Evidence and the Strength of Recommendations Using the GRADE Approach [Internet]. The GRADE Working Group; 2013. Available from: http://gdt.guidelinedevelopment.org/app/handbook/handbook.html
Webb P. How Strong is Our Evidence for Effective Management of Wasting? A Review of Systematic and other Reviews. Food Nutrition Bulletin [Internet]. SAGE PublicationsSage CA: Los Angeles, CA; 2015; 36(1_suppl1):S65–71. Available from: http://journals.sagepub.com/doi/10.1177/15648265150361S111
WHO. Guideline: Updates on the management of severe acute malnutrition in infants and children. Geneva: World Health Organization; 2013.
WHO. Technical note: supplementary foods for the management of moderate acute malnutrition in infants and children 6–59 months of age. Geneva, World Health Organization, 2012.
Wilkinson, Thomas, Mark J. Sculpher, Karl Claxton, Paul Revill, Andrew Briggs, John A. Cairns, Yot Teerawattananon et al. “The international decision support initiative reference case for economic evaluation: an aid to thought.” Value in Health 19, no. 8 (2016): 921-928.
More like this
FEX: Food aid for nutrition: A landscape review of current research and implications for future studies
View this article as a pdf Lisez cet article en français ici By Maria Wrabel, Kristine Caiafa, Beatrice Lorge Rogers and Patrick Webb All authors are affiliated with...
FEX: The “ComPAS” trial combined treatment model for acute malnutrition: study protocol for the economic evaluation
Research snapshot1 Acute malnutrition is currently divided into severe (SAM) and moderate (MAM), based on level of wasting. SAM and MAM currently have separate treatment...
FEX: Defining and treating “high-risk” moderate acute malnutrition using expanded admission criteria (Hi-MAM Study): A cluster-randomised controlled trial protocol
View this article as a pdf By Natasha Lelijveld, David Taylor Hendrixson, Claire Godbout, Alyssa Los, Jukka M Leppänen, Aminata Koroma and Mark Manary Natasha Lelijveld...
FEX: Treatment of moderate acute malnutrition using food products or counselling: A systematic review
View this article as a pdf By Natasha Lelijveld, Alexandra Beedle, Arghanoon Farhikhtah, Eglal E. Elrayah, Jessica Bourdaire and Nancy Aburto Natasha Lelijveld is a Teaching...
Hi, We are developing one programme one nutrition sensitive social protection programme on maternity cash transfer with intensive behaviour change communication. Below were...
Summary of research* Location: Global. What we know: There is global momentum to bring down levels of undernutrition. Wasting and stunting frequently co-exist, but are often...
FEX: Use of educational videos to improve maternal breastfeeding knowledge and practices in Ethiopia
View this article as a pdf By Ritu Rana, Hatty Barthorp, Liya Assefa, Alemayhu Beri and Mary T Murphy Ritu Rana is a GOAL Nutrition Research Advisor (Management of At-risk...
FEX: A series of three related published papers share findings regarding quality of care, treatment outcomes and cost-effectiveness of uncomplicated SAM treatment delivered by community health workers (CHWs) in rural Mali.
By Eleanor Rogers, Karen Martínez, Jose Luis Alvarez Morán, Franck G. B. Alé, Pilar Charle, Saul Guerrero, Natalie Sessions and Chloe Puett Quality of...
Summary of research1 Location: Global What we know: Water, sanitation and hygiene (WASH) interventions are frequently implemented to improve health and reduce infectious...
Blog post: Food Assistance and Nutrition- a recent presentation of our research findings in Pakistan
Hello, I am Bridget Fenn, ENN's long standing lead research investigator. I recently I gave a virtual presentation about the REFANI to participants at a research conference in...
FEX: Social Return on Investment (SROI) assessment of a Baby-Friendly Community Initiative in urban poor settings, Nairobi, Kenya
By Sophie Goudet, Caroline W. Wainaina, Teresia N. Macharia, Milka N. Wanjohi, Frederick M. Wekesah, Peter Muriuki, Betty Samburu, Paula L. Griffiths and Elizabeth W....
Summary of research1 Location: Ethiopia What we know already: Health, nutrition and water, sanitation and hygiene programmes impact on child nutrition. Evaluation of impact...
FEX: MAM and SAM cases reduced through a stunting prevention programme in Malawi and the associated costs averted
View this article as a pdf Lisez cet article en français ici By Jackson Mason-Mackay, Arghanoon Farhikhtah, Mica Jenkins, Trust Takudzwa Mlambo, Emma Lefu and Nancy...
en-net: Stratified randomization
Hi, Can someone (especially: epidemiologist) check my though on Stratified Randomization for one study? Thanks in advance. Our team is going to implement a one nutrition...
Summary of panel discussion1 This session was moderated by Eva Leidman, of Centers for Disease Control and Prevention (CDC). Presentations were given by Mary Hodges of Helen...
FEX: The REFANI Project in Pakistan: adapting research to a multi-sectoral programme for impact measurement
By Zvia Shwirtz, Bridget Fenn, Riccardo Mioli, Ghulam Murtaza Sangrasi and Maureen Gallagher Zvia Shwirtz is currently the REFANI Communications and Research Uptake Officer,...
FEX: Scalable integrated early child development programme using a conditional cash transfer programme in Colombia
Summary of research1 Location: Colombia What we know: Poverty and the associated poor nutrition and lack of psychosocial stimulation are associated with poor child...
View this article as a pdf By Mark Manary, Donna Wegner, D Taylor Hendrixson, Rebecca Roediger, Meghan Callaghan Mark Manary is one of the world's foremost experts in...
FEX: Research Snapshots
The following provides a short summary of each of these important research studies. A fuller summary of each can be found online at www.ennonline.net/fex Inpatient management...
Your help We are thinking of conducting a study looking at the impact of our food security interventions (cash transfer, small scale agriculture ) on under five nutrition...
Reference this page
Kristine Caiafa, Maria Wrabel, Devika Suri, Ye Shen, Shelley Walton, Beatrice Lorge Rogers, Patrick Webb (). A review of research methods used to study specialised nutritious foods. Field Exchange 62, April 2020. p41. www.ennonline.net/fex/62/specialisednutritiousfoods