Review of survey methodology in emergencies
Summary of published research1
Spinning the stick to select clusters during a UNHCR survey in Bangladesh
A recent paper set out to identify common methodological errors in nutrition and mortality surveys conducted in humanitarian emergencies, to examine trends over time and to provide recommendations on how to improve surveys in future.
The sample of surveys was selected from 948 reports of nutrition surveys received by the Nutrition Information in Crisis Situations (NICS) between October 1993 and April 2004 from 34 countries. Of these, 17 countries were selected using a random number generator and all of the survey reports in these countries were reviewed for analysis. Survey reports were evaluated for validity of sampling methodology, precision of estimates, quality of measurements and calculation of prevalence of acute malnutrition and mortality rates.
Three hundred and sixty eight survey reports conducted by 33 non-governmental organisations (NGOs) and international agencies in 17 countries were eventually evaluated.
Criteria for sampling validity were met for 85.9% of surveys. All of the random sample surveys that used sample sizes of 450 children for random and systematic sampling and 900 children for cluster sampling were sufficiently precise. However, cluster surveys that sampled <900 children were insufficiently precise - half of these were conducted in Sudan with many of them occurring in 1999. The vast majority of surveys correctly included children 6-59 months or 65-110 cm. Measurements met the quality criterion in 57.1% of surveys, the remainder that did not meet criterion due mainly to missing information. Regarding oedema, 8.4% of surveys did not include oedema in the definition of acute malnutrition. Three-quarters (76.1%) of the surveys provided the percentage of oedematous children. Incorrect interpretation of results occurred in 15.5% of surveys. Overall, 42.4% of surveys met the criteria for correctly calculating prevalence of acute malnutrition.
Just over three-quarters (76.6%) of surveys were both valid and precise, 51.3% were valid, precise and met the quality of measurement criteria. Finally, 35.3% were valid and sufficiently precise, met the criteria for quality of measurement, outcome definition and calculation.
Measles vaccination coverage could be assessed in 57.1% of surveys. Most (72.3%) used card examination and history of vaccination. Just over half (52.4%) of nutrition surveys had an associated mortality survey. Of these, 81.3% assessed Crude Mortality Rate (CMR) and under-five mortality rates (U5MR) while only 18.1% assessed U5MRs.
Among the 158 surveys that assessed CMRs, 55.1% met criteria for sampling validity and the same percentage met precision criteria, 38.6% of surveys were both valid and precise and only 3.2% were valid, precise and met the calculation criteria.
The proportion of nutrition surveys that met criteria for sampling validity, precision, measurement, definition and calculation rose significantly from 11.1% in 1993-4 to 51.7% in 2003-4. The implementation of CMR surveys associated with nutrition surveys increased significantly over the years but the proportion of CMR surveys that met criteria for sampling validity, precision and calculation did not differ.
The lack of validity of the mortality surveys was mainly due to a lack in specificity in the reports as to how household selection occurred. The lack of precision in the CMR surveys was most likely because the sample sizes often used the same sample size calculated for nutrition surveys. Sample sizes for each key variable in a survey must be calculated independently. Many reports lacked structured methodology and results sections and had insufficient detail to allow for proper analysis and evaluation.
The quality of calculations depended upon the software used for analysis. Approximately 33% of calculations performed with EpiInfo 6 were incorrect because oedematous children had not been correctly taken into account. No comprehensive guideline to analyse nutrition surveys using EpiInfo 6 was available before 2004 when a manual that extensively describes the procedure was released.
Only approximately half of surveys assessed measles vaccination. All nutrition surveys among children should include a measles vaccination coverage component and should clearly disaggregate in the report the percentage according to both history and vaccination card as well as card alone.
In the study, precision of nutrition and mortality surveys was analysed assuming a design effect of 2.0 for cluster-sampled nutrition and mortality surveys. However, recent research shows that a design effect of some nutrition and non war-related mortality surveys was below 2.0 and closer to 1.5. If a design effect of 1.5 had been used, the proportion of nutrition and mortality surveys for which the sample size was sufficient would have been higher. Conversely, war-related mortality surveys might have a higher design effect and might require a higher sample size.
The study concludes that several factors are essential to the strengthening of the quality of nutrition and mortality surveys. Appropriate guidelines are now available and need to be widely disseminated; the increased availability and access to the internet in remote areas has improved their distribution. However, the most crucial factor is the availability of well-trained and experienced staff.
The authors cite a number of initiatives to improve and standardise survey methodology, e.g. FAO, through the Food Security Analysis Unit (FSAU) for Somalia since 1994 and the Emergency Nutrition Coordination Unit (ENCU) in Ethiopia. Furthermore, an international inter-agency initiative was launched in 2002 to Standardize Monitoring and Assessment of Relief and Transitions (SMART).
There are still a number of methodological issues in nutrition and mortality surveys in humanitarian emergencies that need further study. These include, but are not limited to, the mapping of the affected populations for first stage cluster sampling and the selection of households for the second stage, documenting different design effects among various outcomes in different situations, and different methodologies to record deaths, births, and migration in mortality surveys. Furthermore, there is a need to consider standardised age distribution in order to compare over different situations for mortality surveys. The introduction of the recently released WHO growth standards suggests important differences in the diagnosis of wasting compared to the National Centre for Health Statistics (NCHS) growth reference and will thus make comparison of trends in malnutrition in different regions more complicated.
Finally, there is a need for a central and accessible site on the internet, not only for survey reports but also the actual data. This repository will help to ensure quality of surveys and aid in future research.
1Proudhon C and Spiegel P (2007). A review of methodology and analysis of nutrition and mortality surveys conducted in humanitarian emergencies from October 1993-April 2004. Emerging Themes in Epidemiology 2007, Vol 4:10
FAO Food Security Analysis Unit (FSAU): http://www.fsausomali.org
More like this
FEX: A method for estimating mortality rates using previous birth history
By Mark Myatt, Anna Taylor and W. Courtland Robinson Mark Myatt is a consultant epidemiologist and senior fellow at University College London. His areas of expertise include...
FEX: Cross-sectional assessment of retrospective mortality in humanitarian emergencies
Summary of published research1 A scene from Port au Prince, Haiti, following the earthquake 6 days earlier. The rates and causes of mortality are critical indicators of the...
FEX: Is there a systematic bias in estimates of programme coverage returned by SQUEAC coverage assessments?
View this article as a pdf Lisez cet article en français ici By Mark Myatt and Ernest Gueverra Mark Myatt is a consultant epidemiologist. His areas of expertise...
en-net: KAP surveys
Hi, I am trying to find out is there is a standard methodology for doing a KAP? Also, is a KAP the same as a KPC? Please can you let me know - also if there are any good...
en-net: MUAC and Mortality relationship
I have found a nutritional survey report conducted in Sudan with GAM of 27.0% and SAM of 7.5%, while the U5MR IS 1.19/10 000/day using ENA for SMART software. Does this result...
en-net: clear definition of Small Scale SMART Survey
Now days, the term "Small scale SMART survey, SSSS" is becoming common. In principle, is it different from that of normal SMART survey procedure? To make it clear, does SSSS...
FEX: New method for assessing acute malnutrition in nomadic pastoralist populations
By Anne-Marie Mayer, Mark Myatt, Myriam Ait Aissa and Nuria Salse Anne-Marie Mayer is a technical consultant for this project and carried out the first field test in Mali with...
en-net: Cluster Designs to Assess the Prevalence of Acute Malnutrition by Lot Quality Assurance Sampling: A Validation Study by Computer Simulation
The Food and Nutrition Technical Assistance Project II (FANTA-2) would like to announce the release of "Cluster Designs to Assess the Prevalence of Acute Malnutrition by Lot...
FEX: Can height-adjusted cut-offs improve MUAC’s utility as an assessment tool?
By Michel Van Herp, An Verwulgen, Bérengère Leurquin, and Pascale Delchevalerie Michael Ven Herp, Bérengère Leurquin, An Verwulgen & Pascale Delchevalerie Michael Ven Herp is...
en-net: Deciding on sampling design: cluster sampling or stratified random sampling
Hello, we are doing a baseline study for the evaluation of a programme. The programme is going to be implemented in 6 provinces and 18 counties. We want to have information by...
en-net: Anthropometric survey target group: 0-59 months vs. 6-59 months and implications
Greetings EN-NET, When I learned to do anthro surveys years ago, I was taught to sample children 6-59 months old. The rationale includes this group being most vulnerable to...
FEX: Assessment of the PROBIT approach for estimating the prevalence of acute malnutrition from population surveys
Summary of research1 Location: N/A What we know already:Prevalence of GAM is normally estimated using two stage cluster sampled surveys using the SMART method. The PROBIT...
en-net: Sample size for KAP survey
We are planning a KAP baseline survey which will be followed by an endline KAP survey. When calculating the sample size, should I use the following formula? n = D [(Z? + Z?)2...
en-net: Sample size for rapid nutritional assessment of U5 using MUAC
Hi all, We would like to conduct a rapid nutritional assessment in an area affected by flood three times in the last quarter. Could anyone suggest about the size of the...
FEX: Nutrition programme coverage: implementation strategy and lessons learnt from the Sahel trip in Chad
By Hermann Ouedraogo, Ismael Ngnie-Teta, Mahamat Bechir, Esaie Diongoto Domaya, Adama N'Diaye, Paola Valenti Hermann Ouedraogo, MPH, PhD, is Nutrition Cluster Coordinator with...
In looking at materials on CSAS, there are several references to S3M but I can't find anything that actually explains it. Does anyone have any resources for that? Hi Mark, I...
en-net: Second stage sampling
We are doing a SMART survey in province X, which has 20 districts. We plan to have a representative sample in each district, then a weighted provincial prevalence. Our...
en-net: When sampled cluster doen't have enough HH demanded by sample size because of urgent and sudden population movement
Experienced from my current SMART survey, we faced one issues as HH from one sample cluster doesn't have enough HH to met the sample size demanded for each cluster. The target...
en-net: developing a methodology for a highly security compromised area. need guidence
Hi Mark, Salute We are in process of designing a nutrition survey in area which is highly security compromised, infact active war zone in many places. The survey is to...
en-net: Prévalence de la malnutrition aigue selon 3 critères (Z-score Poids-Taille et / ou PB et / ou œdème)
Dear Expert, As the gold standard method, SMART survey permits to estimate accurately and with precision Acute Malnutrition prevalences. ENA software permits to produce SAM...
Reference this page
Review of survey methodology in emergencies. Field Exchange 31, September 2007. p9. www.ennonline.net/fex/31/surveymethodology