Appraisal of the CMAM information system in Northern Nigeria using a ‘toolbox’ approach
By Cora Mezger, Veronica Tuffrey, Gloria Olisenekwu, Charles Umar, Simeon Nanama and
Cora Mezger currently leads the Oxford University Statistical Consulting Service. She was previously the Official Statistics team leader at Oxford Policy Management Ltd and holds a DPhil in Economics from the University of Sussex.
Veronica Tuffrey is a public health nutritionist with over 20 years’ experience as a consultant in international nutrition and lecturer in research methods. She holds a master’s degree and PhD from the London School of Hygiene and Tropical Medicine (LSHTM).
Gloria Olisenekwu is a Survey Coordinator with Oxford Policy Management Nigeria. She has led and been part of numerous data-collection exercises in West Africa.
Charles Umar currently manages the Child Health and Nutrition Project implemented by Nutrition International in Nigeria. He has over 15 years’ experience as a public health professional.
Simeon Nanama is the Chief Nutrition for UNICEF Nigeria. He holds a PhD in International Nutrition from Cornell University and a Doctorate in Food Technology from the University of Ouagadougou, Burkina Faso.
Assaye Bulti manages routine nutrition programme data and surveys for UNICEF Nigeria. He holds an MSc in Epidemiology from LSHTM and has a public health background.
The authors gratefully acknowledge the support that all in-country partners gave to this assessment and the Children’s Investment Fund Foundation for its financial and technical input.
What we know: Information systems for community-based management of acute malnutrition (CMAM) programmes often run parallel to government health information systems and are vulnerable to organisational and behavioural weaknesses.
What this article adds: A quality assessment of the government-led CMAM information system in Northern Nigeria was undertaken in 2017. A mixed-methods approach (primary data collection from multiple levels and secondary data analysis) was applied in a sample of nine health facilities in Sokoto state to assess the quality of data and data sources; data collection; management and analysis processes; organisational inputs (data entry, software and supervision); and behavioural (staff) inputs. Significant weaknesses in the system were identified, which were translated into concrete recommendations (reported elsewhere) and shared with stakeholders for action. Challenges encountered included loss of paper-based documentation and incomplete records at facility-level, and lack of buy-in from senior officials. The authors conclude that the ‘toolbox’ approach used provides a successful method of thoroughly assessing and identifying areas for improvement in information systems that could be replicated in other similar contexts.
Nigeria has the second-highest burden of stunted children in the world, as well as poor rates of global acute malnutrition (GAM), with prevalences of 8.7% in the North-East zone and 8.3% in the North-West zones (NNHS, 2018). Community-based Management of Acute Malnutrition (CMAM) was introduced in Nigeria in 2009 and subsequently scaled up to cover 12 of the northern states. The programme is managed by the Government of Nigeria, with technical support from the United Nations Children’s fund (UNICEF) and funding from various partners, including the Children’s Investment Fund Foundation (CIFF). By 2014, an estimated 37% of severe acute malnutrition (SAM) cases were reached through the CMAM programme (Banda et al, 2014). According to data provided by UNICEF for 2017, approximately 95,000 children were in treatment in around 700 stabilisation centres and outpatient therapeutic programme (OTP) facilities in any given week.
Monitoring of the CMAM programme operates in parallel to the government Health Management Information System (HMIS), as is common practice globally. Initially, monitoring was based on a predominantly paper-based system, but in 2017 this was partially upgraded to a digital system using smartphones to help standardise and speed up data collection and management. Underlying both systems is a set of paper forms filled in at facility level. In the digital system, weekly aggregates are texted by SMS from facility level directly to federal level. In the paper-based system, facility-level monthly aggregated data are transferred by hand to the local government authority (LGA)-level Nutrition Focal Person, from where they are transferred to the State Nutrition Officer to be entered into a spreadsheet. From here, data are transferred to other state-level government users and UNICEF.
In 2017, CIFF and UNICEF jointly requested a review of the quality of the data generated through the mixed paper-based and SMS-based system in the OTP component of CMAM, as well as an assessment of how the data is used. The primary aim was to generate learning to inform improvements to the existing information management system. Both the paper-based and digital systems were running in parallel at the time this study was implemented in November 2017, as shown in Figure 1; strengths and weaknesses of both were compared to assess the feasibility of gradually phasing out the paper-based system. This article describes and reviews the assessment process with a view to informing other similar studies. A summary of the study findings is available elsewhere.1
Figure 1: CMAM information system data flow
A mixed-methods approach was adopted, including fieldwork for primary data collection and a desk-based secondary data analysis and document review.
Dimensions of quality considered
Most studies assessing public health information systems limit their analysis to just three dimensions: completeness, accuracy and timeliness of information produced (Chen et al, 2014). We chose to adopt a broader definition of data quality to assess not just the quality of the information itself, but also how relevant and accessible it is to users to enable its successful use. To this end, five additional dimensions were added to the study design: reliability, integrity, confidentiality, relevance and accessibility.
A conceptual framework was developed against which components of the system could be assessed (Figure 2). This was based on the Performance of Routine Information System Management (PRISM) framework, developed by Measure Evaluation Project, which was further developed to include all aspects (Belay and Lippeveld, 2013). The conceptual framework encompasses data and data sources; data collection, management and analysis processes; organisational inputs (including data entry forms and software, staff training and supervision); and behavioural inputs (including staff confidence and motivation).
Figure 2: Conceptual framework
Toolbox of assessment methods
The study team developed a toolbox of methods that could be used to measure the quality of all components of the conceptual framework (Box 1). Table 1 demonstrates the application of these methods to each stage of the conceptual framework.
Box 1: Toolbox of methods used to appraise the information system
Primary methods of data collection:
- Semi-structured interviews with actors involved in collecting, processing and analysing data and those involved in/responsible for quality assurance. These included the head nurse/CMAM-in-charge and health workers in facilities; the nutrition or monitoring and evaluation officers at LGA level; the State Nutrition Officer at state level; and UNICEF staff at sub-national and federal levels. Interviews covered inputs and processes described in the conceptual framework.
- Semi-structured interviews with current and potential users of CMAM data (beyond those also involved in the data generation).
- Observations of implementation of protocols at each level in the data flow using checklists.
- Verification of records at facility level by recounting source forms for a specified sample time period (mostly the month of July 2017) to produce quantitative results on accuracy (from OTP cards and ready-to-use therapeutic food (RUTF) stock cards).
Secondary data analysis (desk-based activities):
- Analysis of monthly facility-level data in terms of source data characteristics, as well as key performance indicators;
- Review of documentation (forms, guidelines, training material and CMAM protocols)
Table 1: Data collection tools used at each stage of the conceptual framework
The study initially considered using a random two-stage sample of 75 facilities so that results could provide statistical inference to a larger geography. However, it was decided that it would be more useful to carry out an in-depth analysis of a much smaller sample of just nine facilities across three LGAs in one state. The underlying assumption was that any weaknesses in the inputs and processes discovered would be linked to the way that the entire system was designed and implemented, and findings would therefore have system-wide implications. Budgetary considerations also played a role. Sokoto State, three LGAs and nine facilities were purposively selected to provide diverse geographical representation, balanced with logistics such as the day of the week on which the CMAM clinic was held.
The main fieldwork took place in November and December 2017. A team of three field workers coordinated closely with local UNICEF and Ministry of Health staff on logistics and approvals. The fieldworkers developed an in-depth knowledge of the study protocols by actively participating in the development of all tools, jointly reviewing and desk-testing all aspects, and participating in a pre-test and pilot. Health facilities were informed that they would be visited and informed of study objectives shortly beforehand. This helped to gain their cooperation and trust, while minimising the chances of deliberate changes to data collection and transmission processes or data. More information about the study was provided when obtaining consent for each of the activities.
For each health-facility assessment, the team arrived prior to the start of the clinic to first interview the CMAM in charge, count RUTF stock for the verification of RUTF consumption on the day of the visit, and retrieve OTP cards for clients not attending that day for verification. During the day, the team split up to cover both the OTP card recount component and to observe data capture during admission and treatment. After the clinic, the team conducted semi-structured interviews or focus group discussions with health workers, counted RUTF stock, finalised verification from OTP cards in use, took electronic scans of data collection forms and registers, and observed all end-of-day tallying and data-transfer activities. Fieldwork at LGA, state and federal levels also involved both semi-structured interviews and observation of data capture and transfer.
The desk-based analysis involved reviews of monthly, paper-based information and weekly SMS-based information provided by UNICEF. The data collected at facility and LGA levels were compared to both paper and digital datasets. Findings from observations and interviews were also compiled. Detailed results were presented and discussed for each of the inputs, processes and outputs depicted in the conceptual framework and subsequently consolidated into an overall finding and set of recommendations for each quality dimension. A three-level rating system was used throughout, as described in Table 2. Headline findings are presented in Table 3. Findings related to data quality aspects will be reported in full in a peer-reviewed journal.
Table 2: Rating (‘smiley face’) system used to judge performance of the CMAM information system
Table 3: Main findings of the assessment
Challenges and lessons learned
The in-depth approach applied to a small sample of purposively selected facilities proved to be an appropriate and informative design choice. The range of activities we undertook generated a wealth of information across the various components of the conceptual framework and levels of data production. Analysis of this information provided a clear picture of consistent strengths and weaknesses in the design and operation of the CMAM information system and in the use of data. It is unlikely that findings observed were due to chance. However, the approach did mean that quantitative results from the verification component were not statistically representative of all CMAM operations in Northern Nigeria. These results must therefore be reported with caution in future; for instance, with respect to a likely under-reporting of defaults and deaths.
A practical challenge encountered during fieldwork was the difficulty of accessing the OTP cards for data verification. This was due to inadequate space and systems for storage in the facilities and resulting risk of damage and loss of documents. These challenges were picked up in the assessment in the analysis of ‘processes’. Loss of source documents also directly influenced the verification of figures through recounts. Such problems can be minimised by choosing a recent reference period to reduce the likelihood of loss or damage by the time of assessment. Triangulation was performed between OTP cards and entries into the CMAM registration book to provide further information on the extent of loss of OTP cards.
A challenge to the verification of performance indicators was the inconsistent use of the ‘outcome’ field on OTP cards, the main source of data for discharge information (a key CMAM performance indicator for reporting against Sphere Standards). The outcome category was left blank on the OTP card in 64% of cases and could therefore not be recounted to the level of disaggregation required by the study protocols. To avoid having to rely on an aggregate analysis across all discharges, the team decided to use additional OTP card information (mid-upper arm circumference, weight and RUTF given) to determine the most ‘likely’ reason for discharge. Even after applying this method, it was still not possible to determine likely reason for discharge in 3% of cases. These cases were coded as a separate category (‘discharge reason unclear’) in the verification data.
While findings on inputs, processes and outputs mostly built a consistent picture of the information system, consolidation of results into a single, easily understandable message for each quality dimension was sometimes challenging. For instance, completeness and timeliness of data analysed at the output level highlighted challenges with submission of data and resulting deficiencies in these quality dimensions. However, at the input level, observations and interviews showed that great emphasis is given to these quality dimensions as part of in-built data-quality assurance procedures.
A lesson learned from the fieldwork was the importance of working with a small, well trained team to prevent introduction of errors during the verification exercise. A small-scale pre-test and pilot of instruments was also crucial to ensure the correct application of methods. Following the pilot, adjustments were made to the sequence of field activities and the design of semi-structured and focus group discussion questions to improve the process.
While the topic of data use was well covered during interviews with staff also involved in data production (with a 100% response rate in this group), the response rate for other users and potential users of data was mixed. Interviews were successfully undertaken with two of the three non-governmental organisations contacted, but attempts to arrange interviews with three government officials at state and federal levels were unsuccessful. Given that “performance” of an information system comprises both the production of good-quality data and use of information for decision-making, it is essential that this area is well assessed. The lesson learned is to identify, connect with and brief key data users about the study well ahead of fieldwork to ensure their buy-in to the process.
Data-quality assessments in health and nutrition often use secondary data analysis, or sometimes large-sample verification approaches, to provide precise and statistically robust estimates of accuracy and completeness. However, other complementary approaches are needed to understand the reasons behind issues identified and assess additional important dimensions of data quality and use. Our study methods enabled the assessment of the entire data production and use process of the CMAM programme in Northern Nigeria, providing rich and contextualised information on the strengths and weaknesses of the whole system. Findings clearly indicate which elements of the system at which level/s need improvement and have been translated into concrete recommendations tailored to all stakeholders involved.2 Given similarities in the way CMAM information systems and other health information systems are set up and operated in different countries, this study approach and lessons learned may prove useful to those implementing similar assessments of the drivers of data quality and use.
In the case of the CMAM information system in Nigeria, study recommendations led to several follow-up actions. In Sokoto, findings were shared with CMAM stakeholders, health workers were retrained and CMAM monthly meetings were strengthened to share best practices. The final report served as a background document for the assessment of the UNICEF emergency nutrition response in 2019. Findings were also factored into recommendations made to the Nutrition Sector in the northeast of the country; these are currently being translated into a sector strategic plan and triggered a larger verification exercise implemented by the Government (2018-2019) in all 12 states that built on the protocols developed for this study.
For more information, please contact Cora Mezger at firstname.lastname@example.org
1Report of findings is available at: https://www.opml.co.uk/files/Publications/a1468-nigeria-nutrition/ms2-opm-cmam-data-systems-final-report.pdf?noredirect=1 Findings related to data quality aspects will also be published in a peer-reviewed journal.
2Recommendations are shared in the full report available at: https://www.opml.co.uk/files/Publications/a1468-nigeria-nutrition/ms2-opm-cmam-data-systems-final-report.pdf?noredirect=1
Banda et al. (2014) SLEAC Survey of CMAM program Northern States of Nigeria.
Belay and Lippeveld (2013). Inventory of PRISM Framework and Tools: Application of PRISM Tools and Interventions for Strengthening Routine Health Information System Performance, Measure Evaluation, p.5. www.measureevaluation.org/resources/publications/wp-13-138
Chen et al (2014). A review of data quality assessment methods for public health information systems. International journal of environmental research and public health;11(5):5170-207.
National Bureau of Statistics Nigeria (2018). National Nutrition and Health Survey (NNHS) 2018. 2018. www.unicef.org/nigeria/reports/national-nutrition-and-health-survey-nnhs-2018.
More like this
FEX: Monitoring systems for the management of severe acute malnutrition programmes in northern Nigeria
View this article as a pdf Research snapshot1 Each year in Nigeria, more than two and a half million children under five years of age are affected by wasting. Since 2016,...
View this article as a pdf Lisez cet article en français ici By Alison Donnelly, Joanne Chui and Arja Huestis Alison Donnelly has more than 10 years of experience...
By Sasha Frankel, Mark Roland and Marty Makinen Sasha Frankel worked for the Results for Development Institute as a Senior Programme Associate focusing on health financing and...
By Maureen Gallagher, Karina Lopez, Stanley Chitekwe, Esther Busquet & Saul Guerrero Maureen Gallagher is the Technical Coordinator for ACFInternational in Nigeria since July...
By Martin Eklund and Tsinuel Girma Martin Eklund holds an MSc in Clinical Nutrition from the University of Copenhagen. His specialisation is community-based management of...
By Lily Schofield, Selome Gizaw Lalcha and Terefe Getachew Lily Schofield has worked in many countries in Africa and Asia as a nutrition consultant. She has been involved in...
This is a summary of a Field Exchange field article that was included in issue 64. The original article was authored by Alison Donnelly, Joanne Chui and Arja Huestis. Alison...
By Maureen Gallagher, Saúl Guerrero, Ifeanyi Maduanusi and Diego Macías Maureen Gallagher is the Senior Nutrition & Health Advisor Action Against Hunger US based...
By Ekram Mustafa El-Huni Ekram Mustafa El-Huni is WFP's Head of Programmes in Beirut, Lebanon. She has worked in a variety of roles with WFP at the headquarters, regional...
A case study based on the UNICEF Action Framework for improving the diets of young children during the complementary feeding period. Please scroll down for links to download...
FEX: From the editor
Successful homestead gardening in Satkhira, Bangladesh The year 2013 promises to be an important year for the nutrition aid community and those whom it serves. Since...
Summary of study1 Children and their caregivers enrolled in the BSFP In spite of national and international efforts to manage the devastating impact of the conflict in Darfur...
Marijka van Klinken, Matthew Parnaby, Paulina Acosta and Julien Chalimbaud Marijka van Klinken was the NCA Analyst conducting this study. She holds an MSc of Public Health...
A case study based on the UNICEF Action Framework for improving the diets of young children during the complementary feeding period. Please scroll down for links to download...
Summary of evaluation1 A flood affected community in Khyber Pakhtunkhwa province CMAM has been implemented in Khyber Pakhtunkhwa province of Pakistan, with scale up in 2010...
Terms of Reference Evaluation of Concern Ethiopia's National CMAM programme (4 Regions) including the Bale (5 Woreda) emergency project implemented for the period...
FEX: Nutritional response in north-eastern Nigeria: Approaches to increase service availability in Borno and Yobe States
View this article as a pdf Lisez cet article en français ici By Sanjay Kumar Das, Dr Sule Meleh, Dr Umar Chiroma, Bulti Assaye and Maureen L Gallagher Sanjay Kumar...
FEX: Delivering treatment of SAM at scale: lessons learned and forward looking for the Nigeria CMAM programme
By Amy Mayberry and Claire Harbron (CIFF), Arjan De Wagt (UNICEF) and Dr. Chris Osa Isokpunwu (FMOH) Amy Mayberry is on the Children's Investment Fund Foundation's (CIFF)...
By Tewoldeberhan Daniel, Tarig Mekkawi, Hanaa Garelnabi, Salwa Sorkti and Mueni Mutunga View this article as a pdf Dr Tewolde is a Nutrition Specialist with UNICEF Kenya...
FEX: Barriers to access for SAM treatment services in Pakistan and Ethiopia: a comparative qualitative analysis
Summary of research1 Location: Ethiopia and Pakistan What we know: Community-based sensitisation can overcome lack of awareness that impedes access to SAM treatment, but is...
Reference this page
Cora Mezger, Veronica Tuffrey, Gloria Olisenekwu, Charles Umar, Simeon Nanama and Assaye Bulti (). Appraisal of the CMAM information system in Northern Nigeria using a ‘toolbox’ approach. Field Exchange 62, April 2020. p34. www.ennonline.net/fex/62/cmaminformationsystemnorthernnigeria