Menu ENN Search

Improving impact evaluations

Summary of published research1

The ENN has long-championed the need for more evidence based learning in food and nutrition emergency programming and in particular more rigorous impact studies. The report summarised below by the Centre for Global Development (CFGD) sets out a broad agenda for impact assessment across all areas of social development and proposes institutional mechanisms for bringing this about (Ed)

In 2004, the Centre for Global Development (CFGD) convened the Evaluation Gap Working Group in order to find out why rigorous impact assessment of social development programmes are relatively rare and to develop proposals to stimulate more and better impact evaluations.

Many governments and organisations are taking initiatives to improve the evidence base in social development policy but investment is still insufficient while the quality of evaluation studies is mixed. While these institutions do well in their normal data collection and evaluations tasks related to monitoring inputs, improving operations, and assessing performance) they largely fail to build knowledge that require studies outside normal budget and planning cycles. Lacking are impact studies that show the intervention has impacted the condition that the programme sought to alter, e.g. health status, income generation, etc. While the knowledge gained from rigorous impact studies is in part a public good, the cost of producing such studies are borne by individual institutions or agencies.

Even when impact evaluations are commissioned, they frequently fail to yield useful information because they do not use rigorous methods or data. A systematic review of UNICEF estimated that 15% of all its reports included impact assessments but noted that many were unable to properly assess impact because of methodological shortcomings. Similarly a review of 127 studies of 258 community health financing programmes found that only two studies were able to derive robust conclusions about impact.

The CFGD report concluded that there are too few incentives to conduct good impact evaluations and too many technical, bureaucratic and political obstacles. However, tolerance for evaluation gaps is waning and donor countries are increasingly concerned that international financial assistance should generate results.

Concern about the evaluation gap is widespread as demonstrated by the many initiatives underway. However progress will be slow and investment insufficient without greater effort. The Evaluation Gap Working Group recommends that the full range of stakeholders should both reinforce existing initiatives and collaborate on a new set of actions to promote more and better impact evaluations.

Governments and agencies should reinforce efforts to generate and apply knowledge from impact evaluations of social programmes. This includes strengthening overall monitoring and evaluation systems; dedicating resources to impact evaluation; ensuring collaboration between policymakers, project managers, and evaluations experts; improving standards for evidence; facilitating access to knowledge; and building capacity in developing countries to conduct rigours evaluations.

Progress is likely to be much faster if some countries and agencies collectively commit to increase the number of evaluations and adhere to high standards of quality. In one form of commitment, similar to a contract, each organisation would agree to do its part, while another form would see organisations support a common infrastructure to carry out joint functions. The working group identified the following characteristics of a successful new initiative:

The Evaluation Gap Working Group developed consensus that some entity ('council') - whether a committee, standards-based network secretariat or other organisation - is needed as a focal point for leading such an initiative. Council functions were identified as follows;

Other council functions might include;

Some Working Group members were concerned that such a fund would divert financial resources from current impact evaluation efforts. Others argued that giving the council adequate funds to commission impact evaluations was essential to address the central concerns set out in the analysis.

The report also discusses how to constitute the council to best provide collectively beneficial services. Suggestions included an interagency committee, a network, secretariat or independent organisation. The authors recognise that the choice will depend on assessing the relevant tradeoffs and the institutional design should ultimately be guided by the structure that will best fulfil a range of aims, including high technical standards, independence and legitimacy, operational efficiency and international leadership.

Show footnotes

1When Will We Ever Learn: Improving Lives through Impact Evaluation. Policy Recommendations from the CGD Evaluation Gap Working Group, May 2006 http://www.cgdev.org/content/calendar/detail/7829/

More like this

en-net: POLICY COORDINATOR, SIERRA LEONE (IPA/JPAL PARTNERSHIP)

Job Location: Sierra Leone Contract Length: 1 year Start Date: ASAP Job Description Immunization is often claimed to be one of the most cost-effective methods to improve...

FEX: From the editor

Queing for cash transfer Four of the six field articles in this issue of Field Exchange endeavour to demonstrate some form of intervention impact. The programmes are all very...

FEX: Improving the assessment and attribution of effects of development assistance for health

Summary of research1 Location: Global What we know: It is difficult to relate Overseas Development Assistance for health (DAH) and health outcomes; there are multiple funding...

en-net: CMAM Forum Technical Adviser – Consultancy (up to 50 days to end 2014)

Background: Community-based Management of Acute Malnutrition (CMAM) has been adopted by over 60 countries to help combat acute malnutrition in children under five years and...

FEX: Letter challenging conclusions and analysis of Lancet Undernutrition Series, by Fiona Watson and Carmel Dolan

Challenging conclusions and analysis of the Lancet Undernutrition Series Dear Editor, We welcome the opportunity provided by the Lancet to highlight the “desperately...

FEX: Did the Rwandan Evaluation Change anything?

A report by the Joint Evaluation Follow-up Monitoring and Facilitation Network (JEFF) The speed of onset and scale of the Great Lakes emergency which unfolded in April 1994...

FEX: Methodological challenges for operational research in the humanitarian context

By Stephanie Stern and Melchior de Roquemaurel Stephanie Stern leads the Action Against Hunger LAB project which aims to reinforce the impact and uptake of knowledge. Before...

FEX: Food assistance for nutrition: Evidence Summit

By Lindsey Green Ongoing food crises around the world underscore the need for effective food assistance (defined as in-kind food aid and/or cash and voucher programming). As...

FEX: Introduction to the special issue

The most recent Lancet series on maternal and child undernutrition (Bhutta et al, 2013) calculated that even with 90% coverage of specific nutrition interventions (addressing...

en-net: VACANCY - Senior Manager, Monitoring, Learning and Research at GAIN

Senior Manager, Monitoring, Learning and Research Salary: Competitive Location: Geneva The Global Alliance for Improved Nutrition (GAIN) is driven by the vision of a...

FEX: Briefing on the Bihar Child Support Programme, India

By Oxford Policy Management (OPM) India Oxford Policy Management (OPM) is a development and research based consultancy company that has 30 years' experience is providing...

FEX: Action to improve infant and young child feeding in nutrition and child health programmes

A mother prepares porridge from soy-fortified bulgur in Guatemala Proceedings of meeting1 Over 50 researchers, programme implementers, infant feeding experts and...

FEX: Technical brief on the cost of malnutrition

Summary of research1 Location: Global What we know: Poor nutrition carries a significant economic burden at individual, national and global levels and prevents poverty...

en-net: Maternal and Child Health and Nutrition Advocacy Specialist, FANTA III Project #3073

FHI 360 is a nonprofit human development organization dedicated to improving lives in lasting ways by advancing integrated, locally driven solutions. Our staff includes experts...

en-net: Maternal and Child Health and Nutrition Advocacy Specialist, FANTA III Project #3073

FHI 360 is a nonprofit human development organization dedicated to improving lives in lasting ways by advancing integrated, locally driven solutions. Our staff includes experts...

FEX: Robust evidence for an evidence-based approach to humanitarian action

Research By Mike Clarke, Jeroen Jansen and Claire Allen Mike Clarke is one of the founders of Evidence Aid and is now voluntary Research Director and Chair of the Board of...

FEX: Valid International

In late June this year, ENN interviewed Kate Sadler, Director of Programmes at Valid International (VI), and followed up by speaking to Steve Collins, one of the founders of VI...

FEX: The State of the Humanitarian System 2018 report

The fourth 'State of the Humanitarian System' report1, covering the three years from 2015 to 2017, is based on a review of over 200 evaluations of humanitarian action,...

Blog post: Reflections from ENN - maximizing on the value and potential of the GNC and SUN Movement

ENN has returned from the Global Nutrition Cluster (GNC) annual meeting in Amman, Jordan, where it seemed that everyone was talking about the humanitarian development nexus as...

FEX: Improving Training in Nutrition in Emergencies

Summary of workshop report1 An international workshop on improving training in emergency nutrition was held in Nairobi on November 6th and 7th 20082. The workshop brought...

Close

Reference this page

Improving impact evaluations. Field Exchange 29, December 2006. p5. www.ennonline.net/fex/29/impactevaluations