Appraisal of the CMAM information system in Northern Nigeria using a ‘toolbox’ approach

Toolbox of assessment methods

The study team developed a toolbox of methods that could be used to measure the quality of all components of the conceptual framework (Box 1). Table 1 demonstrates the application of these methods to each stage of the conceptual framework.

Box 1: Toolbox of methods used to appraise the information system

Primary methods of data collection:

• Semi-structured interviews with actors involved in collecting, processing and analysing data and those involved in/responsible for quality assurance. These included the head nurse/CMAM-in-charge and health workers in facilities; the nutrition or monitoring and evaluation officers at LGA level; the State Nutrition Officer at state level; and UNICEF staff at sub-national and federal levels. Interviews covered inputs and processes described in the conceptual framework.
• Semi-structured interviews with current and potential users of CMAM data (beyond those also involved in the data generation).
• Observations of implementation of protocols at each level in the data flow using checklists.
• Verification of records at facility level by recounting source forms for a specified sample time period (mostly the month of July 2017) to produce quantitative results on accuracy (from OTP cards and ready-to-use therapeutic food (RUTF) stock cards).

Secondary data analysis (desk-based activities):

• Analysis of monthly facility-level data in terms of source data characteristics, as well as key performance indicators;
• Review of documentation (forms, guidelines, training material and CMAM protocols) 

Table 1: Data collection tools used at each stage of the conceptual framework  

Sample design

The study initially considered using a random two-stage sample of 75 facilities so that results could provide statistical inference to a larger geography. However, it was decided that it would be more useful to carry out an in-depth analysis of a much smaller sample of just nine facilities across three LGAs in one state. The underlying assumption was that any weaknesses in the inputs and processes discovered would be linked to the way that the entire system was designed and implemented, and findings would therefore have system-wide implications. Budgetary considerations also played a role. Sokoto State, three LGAs and nine facilities were purposively selected to provide diverse geographical representation, balanced with logistics such as the day of the week on which the CMAM clinic was held.

Study implementation

The main fieldwork took place in November and December 2017. A team of three field workers coordinated closely with local UNICEF and Ministry of Health staff on logistics and approvals. The fieldworkers developed an in-depth knowledge of the study protocols by actively participating in the development of all tools, jointly reviewing and desk-testing all aspects, and participating in a pre-test and pilot. Health facilities were informed that they would be visited and informed of study objectives shortly beforehand. This helped to gain their cooperation and trust, while minimising the chances of deliberate changes to data collection and transmission processes or data. More information about the study was provided when obtaining consent for each of the activities.

For each health-facility assessment, the team arrived prior to the start of the clinic to first interview the CMAM in charge, count RUTF stock for the verification of RUTF consumption on the day of the visit, and retrieve OTP cards for clients not attending that day for verification. During the day, the team split up to cover both the OTP card recount component and to observe data capture during admission and treatment. After the clinic, the team conducted semi-structured interviews or focus group discussions with health workers, counted RUTF stock, finalised verification from OTP cards in use, took electronic scans of data collection forms and registers, and observed all end-of-day tallying and data-transfer activities. Fieldwork at LGA, state and federal levels also involved both semi-structured interviews and observation of data capture and transfer.

Data analysis

The desk-based analysis involved reviews of monthly, paper-based information and weekly SMS-based information provided by UNICEF. The data collected at facility and LGA levels were compared to both paper and digital datasets. Findings from observations and interviews were also compiled. Detailed results were presented and discussed for each of the inputs, processes and outputs depicted in the conceptual framework and subsequently consolidated into an overall finding and set of recommendations for each quality dimension. A three-level rating system was used throughout, as described in Table 2. Headline findings are presented in Table 3. Findings related to data quality aspects will be reported in full in a peer-reviewed journal. 

Table 2: Rating (‘smiley face’) system used to judge performance of the CMAM information system 

Table 3: Main findings of the assessment 

Challenges and lessons learned  

The in-depth approach applied to a small sample of purposively selected facilities proved to be an appropriate and informative design choice. The range of activities we undertook generated a wealth of information across the various components of the conceptual framework and levels of data production. Analysis of this information provided a clear picture of consistent strengths and weaknesses in the design and operation of the CMAM information system and in the use of data. It is unlikely that findings observed were due to chance. However, the approach did mean that quantitative results from the verification component were not statistically representative of all CMAM operations in Northern Nigeria. These results must therefore be reported with caution in future; for instance, with respect to a likely under-reporting of defaults and deaths.

A practical challenge encountered during fieldwork was the difficulty of accessing the OTP cards for data verification. This was due to inadequate space and systems for storage in the facilities and resulting risk of damage and loss of documents. These challenges were picked up in the assessment in the analysis of ‘processes’. Loss of source documents also directly influenced the verification of figures through recounts. Such problems can be minimised by choosing a recent reference period to reduce the likelihood of loss or damage by the time of assessment. Triangulation was performed between OTP cards and entries into the CMAM registration book to provide further information on the extent of loss of OTP cards.

A challenge to the verification of performance indicators was the inconsistent use of the ‘outcome’ field on OTP cards, the main source of data for discharge information (a key CMAM performance indicator for reporting against Sphere Standards). The outcome category was left blank on the OTP card in 64% of cases and could therefore not be recounted to the level of disaggregation required by the study protocols. To avoid having to rely on an aggregate analysis across all discharges, the team decided to use additional OTP card information (mid-upper arm circumference, weight and RUTF given) to determine the most ‘likely’ reason for discharge. Even after applying this method, it was still not possible to determine likely reason for discharge in 3% of cases. These cases were coded as a separate category (‘discharge reason unclear’) in the verification data.

While findings on inputs, processes and outputs mostly built a consistent picture of the information system, consolidation of results into a single, easily understandable message for each quality dimension was sometimes challenging. For instance, completeness and timeliness of data analysed at the output level highlighted challenges with submission of data and resulting deficiencies in these quality dimensions. However, at the input level, observations and interviews showed that great emphasis is given to these quality dimensions as part of in-built data-quality assurance procedures.

A lesson learned from the fieldwork was the importance of working with a small, well trained team to prevent introduction of errors during the verification exercise. A small-scale pre-test and pilot of instruments was also crucial to ensure the correct application of methods. Following the pilot, adjustments were made to the sequence of field activities and the design of semi-structured and focus group discussion questions to improve the process.  

While the topic of data use was well covered during interviews with staff also involved in data production (with a 100% response rate in this group), the response rate for other users and potential users of data was mixed. Interviews were successfully undertaken with two of the three non-governmental organisations contacted, but attempts to arrange interviews with three government officials at state and federal levels were unsuccessful. Given that “performance” of an information system comprises both the production of good-quality data and use of information for decision-making, it is essential that this area is well assessed. The lesson learned is to identify, connect with and brief key data users about the study well ahead of fieldwork to ensure their buy-in to the process.

Conclusions

Data-quality assessments in health and nutrition often use secondary data analysis, or sometimes large-sample verification approaches, to provide precise and statistically robust estimates of accuracy and completeness. However, other complementary approaches are needed to understand the reasons behind issues identified and assess additional important dimensions of data quality and use. Our study methods enabled the assessment of the entire data production and use process of the CMAM programme in Northern Nigeria, providing rich and contextualised information on the strengths and weaknesses of the whole system. Findings clearly indicate which elements of the system at which level/s need improvement and have been translated into concrete recommendations tailored to all stakeholders involved.² Given similarities in the way CMAM information systems and other health information systems are set up and operated in different countries, this study approach and lessons learned may prove useful to those implementing similar assessments of the drivers of data quality and use.

In the case of the CMAM information system in Nigeria, study recommendations led to several follow-up actions. In Sokoto, findings were shared with CMAM stakeholders, health workers were retrained and CMAM monthly meetings were strengthened to share best practices. The final report served as a background document for the assessment of the UNICEF emergency nutrition response in 2019. Findings were also factored into recommendations made to the Nutrition Sector in the northeast of the country; these are currently being translated into a sector strategic plan and triggered a larger verification exercise implemented by the Government (2018-2019) in all 12 states that built on the protocols developed for this study.

For more information, please contact Cora Mezger at c.l.mezger@gmail.com

Endnotes