Methodological challenges for operational research in the humanitarian context
By Stephanie Stern and Melchior de Roquemaurel
Stephanie Stern leads the Action Against Hunger LAB project which aims to reinforce the impact and uptake of knowledge. Before joining Action Against Hunger she worked for the Strategy & Analysis Department of Save the Children International and was a research fellow at IRIS, the French think tank on international relations and strategic affairs. Her work focused on the transformation of the humanitarian system.
Melchior de Roquemaurel coordinated the Action Against Hunger Research for Nutrition Conference #R4NUT.
This article summarises contributions from Yves Martin-Prével, Institute for Research on Development (IRD); Kate Golden, Concern Worldwide; Timothy Williams from the SPRING project; Victoria Sibson, UCL; Myriam Aït-Aissa, ACF, and plenary discussion during the ACF research conference, Paris, 2016.
The ACF research conference, Paris, 2016 included a panel discussion on the operational challenges of research in humanitarian contexts. This article summarises the session. The discussion was moderated by Yves Martin-Prével of the Institute for Research on Development (IRD). Panellists were Kate Golden from Concern, Timothy Williams from the SPRING project, Victoria Sibson from UCL and Myriam Aït-Aissa from Action Against Hunger.
The panellists shared some of the operational challenges they have experienced in a number of recent research studies (see Box 1) and suggested recommendations to improve the overall quality and efficiency of research on nutrition in humanitarian contexts. The studies in question focused on stunting and/or severe acute malnutrition and/or chronic undernutrition, involving experimental and observational designs. Discussions largely centred on experiences with randomised controlled trials (RCTs) in complex settings.
View a video of this panel debate here.
Box 1: Research experience of panellists
In 2015 UCL and Concern collaborated to test the effectiveness of an early seasonal randomised controlled trial (RCT) to prevent acute malnutrition in rural Niger. This was one of three studies by the ACF-led consortium REFANI (www.actionagainsthunger.org/refani).
Concern also conducted an RCT on a ‘community resilience to malnutrition’ integrated programme in Chad.
Action Against Hunger research department has a multidisciplinary team and currently has a portfolio of six RCTs and seven observational studies recently completed/in progress.
The Strengthening Partnerships, Results, and Innovations in Nutrition Globally (SPRING) project is USAID’s flagship, multi-sector nutrition project focused on reducing stunting and anaemia among children in the first 1,000 days. SPRING has conducted operations and implemented research and evaluations in over 10 countries to guide USAID nutrition programmes and contribute to the global evidence base.
Challenges related to RCTs
RCT methodology is often used for nutrition research in humanitarian contexts and is acknowledged as the ‘gold standard’ in terms of quality of evidence. Nevertheless, implementing an RCT in a humanitarian crisis context poses many challenges.
Humanitarian programmes operate within strict and often circumscribed and short-term, donor-driven timeframes, whereas scientific research often requires a lengthier period for implementation.
In some contexts, RCTs are not feasible: a well-designed observational study may be better than a poorly implemented RCT. Nevertheless, RCTs are feasible in many humanitarian contexts, even in volatile situations. Critically, researchers must remain mindful of the context and ‘expect the unexpected’, which typically impacts the length and cost of the research. For example, there may be security issues during randomisation, a pipeline breakdown in nutritional products, or sudden difficulty in population access. An acute conflict or natural disaster may occur on top of a chronic emergency situation. From a research perspective, with the right attitude and preparedness, these are manageable situations and sometimes research consequences may be limited, e.g. an event may impact on the control and intervention groups equally. However, researchers must often engage in lengthy discussions with operations teams concerned with the challenges and risks of implementing research in volatile contexts. In a multi-sector research project, these conversations are multiplied by the number of sectors involved. Good communication with all parties regarding the project’s purpose and objectives can help minimise adverse impacts.
RCTs are important but can be a ‘risky’ investment for non-governmental organisations (NGOs). From the outset, funding can limit the scope of investigation, such as limiting study arms to a control and one intervention. Where no impact is found in a trial, programme teams may be disheartened by the lack of a clear ‘positive’ or immediately actionable result. This is particularly true when extraordinary efforts by the programme team have been required to carry out an RCT in challenging contexts. To help address this, a mixed-methods approach, including other elements such as qualitative research, monthly surveillance and process evaluations alongside RCTs, can help ensure practical learning takes place that can be applied even if the headline results are less ‘exciting’.
Conducting experimental studies on nutrition requires investment in robust context analysis to document impact pathways or process evaluations, given that nutrition is context-specific, multi-sectorial and related to seasonality. It is important to implement qualitative methodologies, capture seasonal features and take the necessary time for full analysis of all monitored indicators, involving experts from the field, operations and academia.
Funding timing and flexibility is an ongoing challenge in humanitarian contexts. Programme and research funding sources are typically distinct, with different timeframes and donor requirements. For example, the REFANI project research element was funded by DFID, while programme activities were funded by ECHO. Programme funding was not fully committed until several months after the research funding was secured and study preparations needed to start, including hiring staff. This required both the research and programme partners to plan flexibly and to spend research funds before the trial was 100% guaranteed to happen. Single-source funding that covers both research activities and the intervention(s) would allow better coordination of activities. It would also facilitate study designs that could answer the most relevant questions, rather than just those permitted by the current programme design.
Managing the unexpected in research requires donor flexibility. Delays mean extended deadlines are often required. Having a financial envelope for unplanned events is key in the implementation phase, empowering researchers with the flexibility and reactivity needed to respond quickly to change.
Challenges with human resources
Establishing international and national staff teams is not straightforward. Cultural sensitivity is important but is seldom taken into account or given the priority it needs. Capacity development of national staff is often a ‘tick-box’ exercise without proper investment; agencies and donors need to take it seriously to have an impact. Also, we often fail to appreciate that capacity development is two-way; national staff have a wealth of contextual knowledge that is critical to the implementation of field research.
Experiences from the REFANI Project in Niger and ACF research have found that recruiting national staff with the necessary research skills and experience is challenging. This has led to operational/support staff not being hired as planned and overburdening some existing staff, who became responsible for both research and programme activities.
A number of lessons were learned. First, it is important to be realistic when recruiting research staff where capacity is known to be limited, i.e. appoint international staff or include capacity-building for national staff if time and budget permit. Secondly, recruit adequate numbers of extra personnel to support the implementation of research activities. Thirdly, having a part-time database manager is invaluable, given the need for remote data management in many contexts. Finally, paying casual-hire enumerators bonuses for undertaking all data collection rounds (e.g. baseline, mid-term and endline studies) saves time and money, helps build capacity (e.g. through refresher training) and supports data quality. Staff turnover at field level is common; for REFANI, the oversight and continuity of an HQ-based research coordinator was a successful element in the research implementation.
Monitoring and indicators
During the Concern research studies, many personnel hired as study staff had different expectations about the quality and types of data collection required, given their experiences of working for NGOs rather than academic institutions (which have more rigorous standards for data quality). External parameters such as deteriorating security can constrain access for international research staff, which limits opportunities for researchers to provide support and oversight. For example, during the UCL REFANI study, data collection was undertaken with increasingly limited access for UCL’s international research staff due to deteriorating security. To compensate, the team conducted relatively lengthy trainings and hired consultants to provide support as trainers and supervisors. Tablets were used for data collection which posed some practical challenges but at the same time facilitated real-time access to recently collected data at office level in Tahoua, Niamey and London. This proved invaluable for remote data management.
Having a comprehensive theory of change regarding nutrition-related issues is key to understanding research findings. To be truly effective, research needs to go beyond simple quantitative frequencies to learn which programme components contribute to improved nutrition outcomes; how and why; and whether the results are generalisable. Rigorous quantitative methods, complemented by qualitative research, are necessary to answer these questions. SPRING has successfully used mixed-methods research in several countries, but finding time and resources to fully analyse data, especially qualitative data, remains challenging.
Project indicators, while important for accountability, have limitations: they may under-emphasise or fail to capture key factors which can have direct impact and could benefit decision-making. Outcome indicators are unique to each country, making cross-country comparisons difficult. At country-level, however, they do allow for tailoring research and evaluation to local needs. This can help build ownership and investment, since the indicators measure what is directly relevant to countries.
Research challenges usually relate to project management, particularly regarding data collection and monitoring. Quality data collection and management can be lacking in the nutrition and health sectors. It is therefore essential to invest in good monitoring and evaluation systems and link this with observational research, resulting in strong data being embedded in programmes. This requires working with operation teams, better use of the field data documented routinely or through audit, and opening discussions around this. Investing in a good MEAL (Monitoring, Evaluation, Accountability, and Learning) system is key. Action Against Hunger, for example, is working on a tool (NEAP: Nutritional Evaluation Assessment; http://bit.ly/2ktDtuW) to improve the assessment of nutritional outcomes in programmes at field level.
Control groups present an ethical dilemma, particularly in resource-constrained settings. The idea of targeting an intervention based on random selection rather than on need presents operational agencies with a real quandary. In Concern’s experience, ways were found to leverage a control or comparison group while maintaining what was considered an acceptable level of accountability to the communities with which they work.
In Chad, the same intervention was provided to the control group, albeit three years later. In Niger, a comparison group – not a full control group – was used, whereby both study arms received the same total amount of cash but over six months versus four months. In Somalia, the control group (an internally displaced persons (IDP) camp close to the IDP camp receiving the cash intervention) was not randomly assigned. It was identified after it had not been prioritised to receive cash following an independent targeting process. The study team had also devised an alternative study design and analysis plan in the event that the control camp did become a target for cash; as it turned out, this did not happen during the three-month study period.
Research data sharing is also a challenge; a fundamental question is whether data should be accessible to all. Open data is transforming research methods and data treatment.
Multi-sector approaches and partnerships and international partnerships
The implementation of longitudinal studies increasingly requires multidisciplinary approaches and the creation of international (and national) spaces to enable the necessary connections and partnerships. Formal structures where NGOs and academics convene are rare. Creating a formal forum for partnerships to develop can be critically important. Even where financial and time constraints limit this type of collaboration, we must try to capitalise more on sharing past experiences and greater investment in multi-sector approaches.
Detailing the operational challenges of conducting research in humanitarian contexts is important. It is also important to invest in longitudinal and multidisciplinary studies to help understand the causes of undernutrition and the means to manage it. These studies should be complemented by in-depth observational studies. Such ambitious projects should be managed by consortiums of NGOs and academics, supported by donors willing to invest and innovate.
More like this
Bridget Fenn, the REFANI Pakistan Principal Investigator, talks about the results of REFANI at the ACF hosted Research for Nutrition Conference 2017. What is REFANI? The...
View this article as a pdf By Kevin PQ Phelan Kevin PQ Phelan is Nutrition Advisor at ALIMA, the Alliance for International Medical Action based in Paris. Before joining...
FEX: The REFANI Project in Pakistan: adapting research to a multi-sectoral programme for impact measurement
By Zvia Shwirtz, Bridget Fenn, Riccardo Mioli, Ghulam Murtaza Sangrasi and Maureen Gallagher Zvia Shwirtz is currently the REFANI Communications and Research Uptake Officer,...
By Sarah Style Sarah Style is part of the ENN team working with UNHCR on the Anaemia Control, Prevention and Reduction Project In April 2011, the International Lipid-Based...
Summary of review1 Location: Multi-country review What we know already: Current WHO guidelines recommend routine antibiotics for all children with severe acute malnutrition...
Summary of research1 Location: Global What we know: Water, sanitation and hygiene (WASH) interventions are frequently implemented to improve health and reduce infectious...
Summary of research1 Location: DRC, Uganda and Nigeria What we know: Ill children are at increased risk of malnutrition due to the infection-malnutrition cycle. What this...
Concern Worldwide is a Dublin-based non-governmental, international, humanitarian organization dedicated to reducing suffering and eliminating extreme poverty in the world's...
SAM inadequately addressed in the Lancet Undernutrition Series Dear Editor, In 2003, The Lancet captured and focused attention on saving children’s lives with the publication...
FEX: Recommendations for multi-sector nutrition planning: Cross-context lessons from Nepal and Uganda
By Amanda Pomeroy-Stevens, Heather Viland and Sascha Lamstein Lisez cet article en français ici Amanda Pomeroy-Stevens is a Research and Evaluation Advisor on the...
By Amanda Pomeroy-Stevens, Alexis D'Agostino, Madhukar B Shrestha and Abel Muzoora View this article as a pdf Amanda Pomeroy-Stevens is Research and Evaluation Advisor on the...
Summary of panel discussion1 This session was moderated by Eva Leidman, of Centers for Disease Control and Prevention (CDC). Presentations were given by Mary Hodges of Helen...
FEX: A cluster RCT to measure the effectiveness of cash-based interventions on nutrition status, Sindh Province, Pakistan
By Bridget Fenn View this article as a pdf Bridget Fenn is an epidemiologist with a background in nutrition. She is currently a consultant for the Emergency Nutrition Network...
Summary of unpublished paper1 The ENN have recently completed a review with funding from CIDA of the published evidence for the impact and cost-effectiveness of 6 key...
By Alix Harou, Marjolein Moreaux and Leanne Dougherty View this article as a pdf Alix Harou has a background in national and international public health and currently works...
By Stephanie Stern Stephanie Stern leads the Action Against Hunger LAB project which aims to reinforce the impact and uptake of knowledge. Before joining Action Against...
Name: SPRING (Strengthening Partnerships, Results and Innovations in Nutrition Globally) Established: 2011 (5 year USAID funded project) Address: JSI Research & Training...
Blog post: Research on multi-sectoral programming: reflections on a cash and WASH, nutrition integrated approach
Cliquez ici pour lire en français Some time ago I had the opportunity to attend a regional event for sharing multi-sectoral nutrition approaches, organized by ACF and...
The post is for a Research Associate at the Institute for Global Health at University College London (UCL). It is funded by the Department for International Development via...
Summary of research1 Background ENN produced a report, through the MQSUN+ programme, that synthesises existing evidence and stakeholder opinion on what works to prevent...
Reference this page
Stephanie Stern and Melchior de Roquemaurel (2017). Methodological challenges for operational research in the humanitarian context. Field Exchange 54, February 2017. p57. www.ennonline.net/fex/54/opchallengeshumresearch