Menu ENN Search

Robust evidence for an evidence-based approach to humanitarian action

Research

By Mike Clarke, Jeroen Jansen and Claire Allen

Mike Clarke is one of the founders of Evidence Aid and is now voluntary Research Director and Chair of the Board of Trustees. He has extensive experience in the evaluation of health and social care and is Director of the Northern Ireland Hub for Trials Methodology Research at Queen’s University, Belfast. He has worked on some of the largest randomised trials in specific areas of health and on dozens of systematic reviews in a wide range of areas, with a strong interest in increasing capacity for the conduct and use of systematic reviews. He facilitates trainings on systematic reviews for Evidence Aid.

Jeroen Jansen is the Director of Evidence Aid. He has been engaged in a wide variety of sectors, issues and working environments, mostly in management positions. He worked for Médecins Sans Frontières (MSF) in Afghanistan, Liberia, Darfur and several other contexts. He has been successfully employed in the private sector, managing the deployment of humanitarian aid and working as a department head for MSF in the UK. Formerly an engineer, he later obtained a Masters in International Human Rights Law and worked for Marie Stopes International.

Claire Allen is the part-time Operations Manager for Evidence Aid. She worked for Cochrane for 18 years and has a B.A. (Hons) in Communication, Media and Culture. Claire’s background is in administration and finance. Her main tasks relate to the overall operations of Evidence Aid, including management of information to ensure that evidence is available in a readily accessible format.

The increasing demand for ‘value for money’, proof of impact and effectiveness in the provision of humanitarian aid requires that decisions and activities become more evidence-based. Reliable and robust evidence will help those making decisions and choices, and those developing policies and standards, to know which interventions work, which don’t work and which remain unproven. For those interventions that work, people need to know how effective they are and for whom, so that they can choose the most appropriate and effective intervention in a specific context. Only with reliable and robust evidence is it possible to maximise the impact, efficiency (‘value for money’) and effectiveness of humanitarian action and ensure more good than harm is done.

Evidence Aid 1 engages with those guiding the humanitarian sector to inspire and enable them to apply a more evidence-based approach to their decisions and actions. Through this engagement, and by working with others who generate, disseminate and apply evidence, a large and growing audience is revealed that supports a more evidence-based approach to humanitarian action. A momentum is building, similar to what has happened in the healthcare sector since the latter decades of the twentieth century (Clarke, 2015), generating a demand for robust evidence. One of the challenges is that this audience has a wide range of interpretations regarding what reliable and robust evidence entails.

When it comes to identifying and using findings of research to decide what is likely to do more good than harm, the healthcare sector recognises the need for evidence to come from a synthesis of similar studies, often in a systematic review. These evidence syntheses provide users with a critically appraised summary of the relevant research on a topic, allowing the existing studies to be compared, contrasted and combined to provide the knowledge needed to resolve uncertainties. In this way, systematic reviews provide the vehicle by which evidence from earlier research can be brought together in ways that minimise bias, avoid undue emphasis on individual studies, maximise the power of research that has already been done, and minimise waste from unnecessary duplication or inadequate uptake.

Systematic reviews begin with a focused question and clear eligibility criteria, then seek out and appraise the relevant studies and compare, contrast and, where relevant and possible, combine their findings. They provide decision-makers and others making choices with a summary of the available evidence, which they can consider alongside other information, such as local values and resources, before taking action. An up-to-date, systematic review allows well-informed decisions to be taken quicker and eases the evidence-gathering burden for people who need to take these decisions. The value of systematic reviews is widely recognised in healthcare, and the concept of drawing on the totality of evidence when making decisions is neither new nor outlandish when explained to practitioners, patients, policy-makers and the public. This should be no different for disasters and other humanitarian emergencies (Gerdin et al, 2014). Just as happened in the healthcare sector several decades ago, the growing need and support for an evidence-based approach in the humanitarian sector should result in a growing need for robust evidence and an increase in the investment required to generate and disseminate this evidence.

Despite a growing momentum for an evidence-based approach in the humanitarian sector, progress seems limited by a strong sense of tradition, an antipathy to change and continuing misapplication of ‘expert opinion’ or ‘best practice’. Key documents in the sector, such as guidelines and policies, continue to be predominantly based on ‘expert opinion’ and ‘best practice’, although this is changing as key influencers that straddle the humanitarian and healthcare sectors, such as the World Health Organisation (WHO), emphasise the need to underpin their guidance with systematic reviews. (See, for example, the ongoing work on a guideline for major radiation emergencies (Carr et al, 2016) and a systematic review of accessibility in the home for an upcoming WHO guideline on healthy housing (Cho et al, 2016)). This does not imply an unthinking adoption of evidence synthesis as a recipe for decision-making. There are situations where the use of ‘expert opinion’ or ‘best practice’ is justified, just as an evidence-based approach, based on robust evidence, should only be applied when and where appropriate. Nevertheless, it is important that the humanitarian sector recognises the limits of ‘expert opinion’ and ‘best practice’ and the value of an approach based on evidence synthesis.

Expert opinion is a valuable tool when applied appropriately, but applied inappropriately it can cause considerable harm, getting in the way of effective action or promoting the use of ineffective or harmful actions. William J. Sutherland and Mark Burgman (2015) described the pitfalls of applying expert opinion rather well in the journal Nature. They assert that the “accuracy and reliability of expert opinions is … compromised by a long list of cognitive frailties (Tversky & Kahneman, 1982). Estimates are influenced by experts’ values, mood, whether they stand to gain or lose from a decision (Englich  & Soder, 2009), and by the context in which their opinions are sought. Experts are typically unaware of these subjective influences. They are often highly credible, yet they vastly overestimate their own objectivity and the reliability of their peers (Burgman  et al, 2011).” This does not render ‘expert opinion’ useless, but demands a certain degree of care when engaging experts and a need for awareness of the tools available to counter these pitfalls.

An example of the appropriate use of expert opinion is in determining the gaps in the evidence base and prioritising the filling of those gaps. The identification of gaps in evidence in the humanitarian sector is important for minimising unnecessary overlap of activities and waste of resources. Evidence Aid held a priority setting meeting in 2013, bringing together those who influence and guide the humanitarian sector, and published the output in the journal PLOS Currents: Disasters (EAPSG, 2013).

Thirty high-priority research questions were identified under ten themes that could be addressed by systematic reviews in the area of planning for or responding to natural disasters, humanitarian crises or other major healthcare emergencies. Some of these gaps have already been taken up by the Humanitarian Evidence Programme2 and others, and relevant systematic reviews will appear in coming months and years. As these new reviews are done, they will be added to the more than 250 systematic reviews that are already freely available from the Evidence Aid resources.3

Another concept frequently applied in the context of decision-making in humanitarian response is ‘best practice’. On The Guardian Global Development Professionals Network page, an anonymous blogger asserts that “best practices are those things that we’ve somehow managed to figure out actually work, and work well” or “is something that we know works and is worth repeating”4 The variety of definitions of concepts such as best practice is one of the common problems in the humanitarian sector. However, potentially even more problematic is the fact that the blogger, like many others in the sector, does not reveal what their criteria are for something to ‘work’ and how it was established that any particular practice adheres to these criteria. As with ‘expert opinion’, the application of a transparent methodology to determine what ‘works’ would overcome most of the hurdles, allowing specific interventions, actions and strategies to be proven to be effective or efficient to the satisfaction of decision-makers.

Proper evaluations can provide evidence of the impact of a certain project or practice. Key to this is the application of an appropriate methodology, such as promoted by the Active Learning Network for Accountability and Performance in Humanitarian Action (ALNAP)5. Nevertheless, an evaluation of one project only reveals something about the impact in that particular context and does not allow us to derive strong conclusions about the likely effects of a similar project in a different context. Just as the successful treatment of one patient or the positive findings of one clinical trial in the healthcare sector do not provide any certainty that it will work for others, we need a synthesis of the evaluations of similar projects to determine the likelihood of success in another place and time. Recognition of this in the healthcare sector helped the drive towards evidence synthesis and systematic reviews as a means to bring together all the available evidence.

Evidence syntheses, or systematic reviews, of reliable evaluations should be a key source of knowledge for all decision-makers who want to answer the question: What is likely to happen if we do this, rather than something else? In the humanitarian sector, decisions, choices and policies impact on the health and wellbeing of thousands, if not millions, of people and those responsible have a duty to ensure that the evidence they use is reliable and robust. This requires the use of appropriate methodologies; firstly to evaluate humanitarian action, and then to bring together the findings of those evaluations. It will mean that the best possible use is made of what happened in the past to predict what will happen in the future, and ensure that humanitarian action does what those who fund it and those who implement it want it to do: prevent and alleviate the suffering of people in need in humanitarian and disaster risk reduction contexts.

The next two courses An introduction to systematic reviews in the humanitarian sector, led by Mike Clarke, are to be held on 16 November in Washington DC, USA and on 30 November in Oxford, UK. The cost is £225/person. For more information, contact: Claire Allen, email: callen@evidenceaid.org or visit http://www.evidenceaid.org/

References

Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L, Wintle B, Fidler F, Rumpff L, Tweedy C. (2011) Expert status and performance. PLoS ONE 6(7): e22998. doi:10.1371/journal.pone.0022998.

Carr Z, Clarke M, Ak EA; Schneider R, Murith C, Li C, Parrish-Sprowl J, Stenke L, Cui-Ping L, Bertrand S, Miller C. (2016) Using the GRADE approach to support the development of recommendations for public health interventions in radiation emergencies. Radiation Protection Dosimetry (first published online 12 August 2016) doi: 10.1093/rpd/ncw234.

Cho HY, MacLachlan M, Clarke M, Mannan H. (2016) Accessible home environments for people with functional limitations: a systematic review. Int J Environ Res Public Health 13: 826. doi: 10.3390/ijerph13080826.

Clarke M (2015). History of evidence synthesis to assess treatment effects: personal reflections on something that is very much alive. JLL Bulletin: Commentaries on the history of treatment evaluation.  J Roy Soc Med 109:154-63. Doi: 10.1177/0141076816640243. www.jameslindlibrary.org/articles/history-of-evidence-synthesis-to-assess-treatment-effects-personal-reflections-on-something-that-is-very-much-alive/)

EAPSG Evidence Aid Priority Setting Group. Prioritization of themes and research questions for health outcomes in natural disasters, humanitarian crises or other major healthcare emergencies. PLoS Curr Disasters 2013 October 16. Edition 1. doi: 10.1371/currents.dis.c9c4f4db9887633409182d2864b20c31

Englich B, Soder K. (2009) Moody experts – How mood and expertise influence judgmental anchoring. Judgement and Decision Making Vol. 4, No. 1, February 2009, pp. 41-50. journal.sjdm.org/71130/jdm71130.pdf

Gerdin M, Clarke M, Allen C, Kayabu B, Summerskill W, et al. (2014) Optimal evidence in difficult settings: improving health interventions and decision making in disasters. PLoS Med 11(4): e1001632.doi:10.1371/journal.pmed.1001632. journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1001632

Sutherland WJ and Burgman M. Policy advice: Use experts wisely. Nature 2015 Oct 15;526(7573):317-8. doi: 10.1038/526317a. www.ncbi.nlm.nih.gov/pubmed/26469026

Tversky A and Kahneman D. (1982) In: Kahneman D, Slovic P, Tversky A (editors) Judgement Under Uncertainty: Heuristics and Biases. Cambridge University Press pp.23-30.

 

Read more...

More like this

FEX: Evidence Informed Decision-Making Process on Nutrition in Ethiopia

Tesfaye Hailu has an MSc in Human Nutrition. He has worked for the Ethiopian Public Health Institute (EPHI)1 for the last eight years as a researcher. He is also a member of...

FEX: Donors in humanitarian action: changing roles, trends and issues

changing roles, trends and issues Summary of a published review1 The changing role of donor governments in the management of humanitarian assistance is the subject of a...

FEX: The use of evidence in humanitarian decision making

Summary of study1 Location: Ethiopia, DRC and Philippines What we know: Decision making in humanitarian response requires timely information and analysis and there are...

en-net: Nutrition specialist (f/m) - Consultant

Médecins Sans Frontières (MSF) is an international, independent, medical humanitarian organisation that delivers emergency aid to people affected by armed conflict, epidemics,...

en-net: Nutrition specialist (f/m) - Consultant

Médecins Sans Frontières (MSF) is an international, independent, medical humanitarian organisation that delivers emergency aid to people affected by armed conflict, epidemics,...

en-net: Humanitarian Evidence Program hosts discussion on evidence at DFID

We invite our London-based colleagues to join us on April 14, 2016 for a discussion on humanitarian evidence, co-hosted by Oxfam and the Feinstein International Center....

FEX: Links between needs assessment and decisions in food crisis responses

Summary of published review1 Workers wade into the Sea to collect WFP food at the southern Somali port of Merka A recently completed study under the WFP Strengthening...

FEX: Aid effectiveness and Vulnerability Assessment Framework: determining vulnerability among Syrian refugees in Jordan

By Hisham Khogali, Lynette Larsen, Kate Washington and Yara Romariz Maasri Hisham Khogali is an independent consultant with 19 years of experience in a range of humanitarian...

FEX: Nutrition, resilience and the genesis of AGIR

By Jan Eijkenaar Jan Eijkenaar has been ECHO's Technical Assistant in support of a resilience approach and the roll out of the AGIR Alliance from September 2012 to April 2015,...

FEX: Designing an inter-agency multipurpose cash transfer programme in Lebanon

By Isabelle Pelly Isabelle Pelly was Save the Children's Food Security & Livelihoods Adviser in Lebanon until September 2014, and co-chair of the Lebanon Cash Working Group....

FEX: The cost of a knowledge silo: A systematic re-review of water, sanitation and hygiene interventions

Summary of research1 By Michael Loevinsohn Michael Loevinsohn is a research fellow at the Institute of Development Studies, UK. An ecologist and epidemiologist, he has worked...

FEX: Research Snapshots

The following provides a short summary of each of these important research studies. A fuller summary of each can be found online at www.ennonline.net/fex Inpatient management...

FEX: Ready-to-use therapeutic food and the WHO list of essential medicines

By Aurélie du Châtelet, Anne-Dominique Israel, Elise Rodriguez, Wisdom Dube, Laetitia Battisti, Magali Garcia, Coline Morin and Natalie Sessions. Aurélie...

Management of Acute Malnutrition in Infants under six months (MAMI)

Donors: OFDA, Irish Aid Collaborators: LSHTM, University of Washington, KEMRI, WHO, UNHCR, UNICEF, Save the Children, ACF, Goal and independent experts. ENN project lead:...

FEX: Technical briefs on MAM

The CMAM Forum (www.cmam.org) has produced a series of technical briefs on moderate acute malnutrition (MAM) written by expert authors: Management of MAM: current knowledge...

FEX: ENN’s perspective on the nutrition response in the Syria crisis

By Carmel Dolan, Marie McGrath and Jeremy Shoham Unless otherwise stated, referenced articles feature in Field Exchange 48. While the ENN's role is first and foremost to...

FEX: From the editor

Queing for cash transfer Four of the six field articles in this issue of Field Exchange endeavour to demonstrate some form of intervention impact. The programmes are all very...

FEX: Briefing on the Bihar Child Support Programme, India

By Oxford Policy Management (OPM) India Oxford Policy Management (OPM) is a development and research based consultancy company that has 30 years' experience is providing...

FEX: Complementary food hygiene: An overlooked opportunity in the WASH, nutrition and health sectors

Summary of research1 Location: Global What we know: Poor food hygiene may account for a substantial proportion of diarrhoeal diseases and contribute to malnutrition among...

en-net: WFP looking for consultant for development of guidance document ‘Integration of Nutrition in VAA’

1. Background The SADC Regional Vulnerability Assessment and Analysis (RVAA) Programme was established in 2005 with an aim to “strengthen national and regional vulnerability...

Close

Reference this page

Mike Clarke, Jeroen Jansen and Claire Allen (2016). Robust evidence for an evidence-based approach to humanitarian action. Field Exchange 53, November 2016. p21. www.ennonline.net/fex/53/robustevidence