Menu ENN Search

Improving impact evaluations

Summary of published research1

The ENN has long-championed the need for more evidence based learning in food and nutrition emergency programming and in particular more rigorous impact studies. The report summarised below by the Centre for Global Development (CFGD) sets out a broad agenda for impact assessment across all areas of social development and proposes institutional mechanisms for bringing this about (Ed)

In 2004, the Centre for Global Development (CFGD) convened the Evaluation Gap Working Group in order to find out why rigorous impact assessment of social development programmes are relatively rare and to develop proposals to stimulate more and better impact evaluations.

Many governments and organisations are taking initiatives to improve the evidence base in social development policy but investment is still insufficient while the quality of evaluation studies is mixed. While these institutions do well in their normal data collection and evaluations tasks related to monitoring inputs, improving operations, and assessing performance) they largely fail to build knowledge that require studies outside normal budget and planning cycles. Lacking are impact studies that show the intervention has impacted the condition that the programme sought to alter, e.g. health status, income generation, etc. While the knowledge gained from rigorous impact studies is in part a public good, the cost of producing such studies are borne by individual institutions or agencies.

Even when impact evaluations are commissioned, they frequently fail to yield useful information because they do not use rigorous methods or data. A systematic review of UNICEF estimated that 15% of all its reports included impact assessments but noted that many were unable to properly assess impact because of methodological shortcomings. Similarly a review of 127 studies of 258 community health financing programmes found that only two studies were able to derive robust conclusions about impact.

The CFGD report concluded that there are too few incentives to conduct good impact evaluations and too many technical, bureaucratic and political obstacles. However, tolerance for evaluation gaps is waning and donor countries are increasingly concerned that international financial assistance should generate results.

Concern about the evaluation gap is widespread as demonstrated by the many initiatives underway. However progress will be slow and investment insufficient without greater effort. The Evaluation Gap Working Group recommends that the full range of stakeholders should both reinforce existing initiatives and collaborate on a new set of actions to promote more and better impact evaluations.

Governments and agencies should reinforce efforts to generate and apply knowledge from impact evaluations of social programmes. This includes strengthening overall monitoring and evaluation systems; dedicating resources to impact evaluation; ensuring collaboration between policymakers, project managers, and evaluations experts; improving standards for evidence; facilitating access to knowledge; and building capacity in developing countries to conduct rigours evaluations.

Progress is likely to be much faster if some countries and agencies collectively commit to increase the number of evaluations and adhere to high standards of quality. In one form of commitment, similar to a contract, each organisation would agree to do its part, while another form would see organisations support a common infrastructure to carry out joint functions. The working group identified the following characteristics of a successful new initiative:

The Evaluation Gap Working Group developed consensus that some entity ('council') - whether a committee, standards-based network secretariat or other organisation - is needed as a focal point for leading such an initiative. Council functions were identified as follows;

Other council functions might include;

Some Working Group members were concerned that such a fund would divert financial resources from current impact evaluation efforts. Others argued that giving the council adequate funds to commission impact evaluations was essential to address the central concerns set out in the analysis.

The report also discusses how to constitute the council to best provide collectively beneficial services. Suggestions included an interagency committee, a network, secretariat or independent organisation. The authors recognise that the choice will depend on assessing the relevant tradeoffs and the institutional design should ultimately be guided by the structure that will best fulfil a range of aims, including high technical standards, independence and legitimacy, operational efficiency and international leadership.

Show footnotes

1When Will We Ever Learn: Improving Lives through Impact Evaluation. Policy Recommendations from the CGD Evaluation Gap Working Group, May 2006 http://www.cgdev.org/content/calendar/detail/7829/

More like this

en-net: POLICY COORDINATOR, SIERRA LEONE (IPA/JPAL PARTNERSHIP)

Job Location: Sierra Leone Contract Length: 1 year Start Date: ASAP Job Description Immunization is often claimed to be one of the most cost-effective methods to improve...

FEX: Improving the assessment and attribution of effects of development assistance for health

Summary of research1 Location: Global What we know: It is difficult to relate Overseas Development Assistance for health (DAH) and health outcomes; there are multiple funding...

FEX: Lessons learned from evaluations of the impact of WFP programmes on moderate acute malnutrition in the Sahel

View this article as a pdf By Jonas Heirman, Mica Jenkins and Jennifer Rosenzweig Jonas Heirman leads impact evaluation activities for the World Food Programme (WFP). Jonas...

en-net: CMAM Forum Technical Adviser – Consultancy (up to 50 days to end 2014)

Background: Community-based Management of Acute Malnutrition (CMAM) has been adopted by over 60 countries to help combat acute malnutrition in children under five years and...

FEX: Did the Rwandan Evaluation Change anything?

A report by the Joint Evaluation Follow-up Monitoring and Facilitation Network (JEFF) The speed of onset and scale of the Great Lakes emergency which unfolded in April 1994...

FEX: From the editor

Queing for cash transfer Four of the six field articles in this issue of Field Exchange endeavour to demonstrate some form of intervention impact. The programmes are all very...

FEX: Methodological challenges for operational research in the humanitarian context

By Stephanie Stern and Melchior de Roquemaurel Stephanie Stern leads the Action Against Hunger LAB project which aims to reinforce the impact and uptake of knowledge. Before...

en-net: VACANCY - Senior Manager, Monitoring, Learning and Research at GAIN

Senior Manager, Monitoring, Learning and Research Salary: Competitive Location: Geneva The Global Alliance for Improved Nutrition (GAIN) is driven by the vision of a...

FEX: A Review of the advances and challenges in nutrition in conflicts and crises over the last 20 years

Abbreviated version of unpublished paper Food distribution at the ICRC kitchen in Tonj. By Frances Mason and Anna Taylor This paper is a shortened version of the complete...

en-net: Nutrition Information Management Officer - Rapid Response Team

Nutrition Information Management Officer 6 month contract position, beginning September 1, 2014 The Rapid Response Team (RRT) of the Global Nutrition Cluster (GNC) is composed...

en-net: Updated posting for Nutrition Information Management Officer - Rapid Response Team

Nutrition Information Management Officer 6 month contract position, beginning September 1, 2014 The Rapid Response Team (RRT) of the Global Nutrition Cluster (GNC) is composed...

en-net: Maternal and Child Health and Nutrition Advocacy Specialist, FANTA III Project #3073

FHI 360 is a nonprofit human development organization dedicated to improving lives in lasting ways by advancing integrated, locally driven solutions. Our staff includes experts...

en-net: Maternal and Child Health and Nutrition Advocacy Specialist, FANTA III Project #3073

FHI 360 is a nonprofit human development organization dedicated to improving lives in lasting ways by advancing integrated, locally driven solutions. Our staff includes experts...

FEX: Letter challenging conclusions and analysis of Lancet Undernutrition Series, by Fiona Watson and Carmel Dolan

Challenging conclusions and analysis of the Lancet Undernutrition Series Dear Editor, We welcome the opportunity provided by the Lancet to highlight the “desperately...

FEX: Introduction to the special issue

The most recent Lancet series on maternal and child undernutrition (Bhutta et al, 2013) calculated that even with 90% coverage of specific nutrition interventions (addressing...

FEX: Conclusions to 'For Food Crisis to Fair Trade' (Special Supplement 3)

Enormous progress has been made over the past few years in food security and livelihoods programming in emergencies. Assessments are increasingly making use of the livelihoods...

FEX: From the editor

This Field Exchange special issue on Urban Food Security and Nutrition aims to provide some insights into the learning and experience of a broad range of agencies, and...

en-net: Action Against Hunger seeks a Nutrition Assessment Technical Advisor for the Tech RRT


Job title: Nutrition Assessment Technical Advisor (Tech RRT)
Department: Programmes
Organization: Action Against Hunger Canada
Direct Supervisor:...

FEX: Technical brief on the cost of malnutrition

Summary of research1 Location: Global What we know: Poor nutrition carries a significant economic burden at individual, national and global levels and prevents poverty...

en-net: Associate Director Investments

Contract: Permanent
Hours: Full time (37.5 per week)
Location: Central London with a requirement to travel internationally (approximately 6 weeks/year)

Close

Reference this page

Improving impact evaluations. Field Exchange 29, December 2006. p5. www.ennonline.net/fex/29/impactevaluations

(ENN_2190)

Close

Download to a citation manager

The below files can be imported into your preferred reference management tool, most tools will allow you to manually import the RIS file. Endnote may required a specific filter file to be used.