"Monitoring and evaluation is an organized approach to understanding how well programs work and... whether they're achieving what they're suppose to achieve...That information can then be used to strengthen... programs, to better understand what works so you can do better, and also to ultimately demonstrate the impact of the programs and advocate for resources."
-Dr. Sian Curtis [Watch this interview]
Through monitoring and evaluation (M&E), family planning programs gauge their progress and their success. Through research, programs can decide wisely how to develop in new directions and how to solve problems. Without M&E and research, the program manager is working in the dark.
Q. What is monitoring and evaluation?
A. M&E is a process of collecting and analyzing information about how a program is operating. Its goal is to inform program planning and project management.
Monitoring involves collecting and analyzing routine data that measure progress toward program objectives. It can track the use of resources, the delivery of services, and the production of products. Thus, monitoring can detect changes in program performance over time and help explain why those changes occurred. Monitoring helps managers make informed decisions about the efficient and effective use of resources. It also helps managers make mid-course corrections to improve performance. Monitoring should be conducted at every stage of the program. Data should be collected, analyzed, and used on a continuous basis.
Evaluation measures what a program has accomplished over time. These findings can be compared with original objectives. Evaluation also may try to determine how much credit a program or project can take for changes in ultimate outcomes, such as increases in contraceptive use or reduction in unmet need for family planning. Evaluations guide planning for future programs. They also help decide whether ongoing programs need major changes.
While some evaluations may take place at the end of programs, it is important that they be planned at the start. Data for evaluation must be collected throughout the program so that comparisons can be made with later data. Having baseline data collected before a program starts is essential to complete evaluations.
M&E Fundamentals: A Self-Guided Mini Course
Q. Why is M&E important to achieving program success?
A. Monitoring and evaluation gives program implementers objective evidence to:
- Make informed decisions regarding program operations
- Ensure the effective and efficient use of resources
- Assess whether the program is having or has had the desired impact, and how much impact
- Meet organizational reporting requirements
"It's essential to think about M&E from the start. Too often, program managers think about evaluation as a program is coming to an end. In some ways evaluation does happen "at the end," but it is essential to have data collected over the course of the program implementation so that at the end you have something to study in terms of trends."
-- Jane Bertrand, Director, Center for Communication Programs, Johns Hopkins Bloomberg School of Public Health [Watch the interview]
Q. What are indicators, and why are they useful?
A. Indicators are variables that are measured to monitor progress and to assess what works. They are signs or markers that reflect some specific aspect of a program that directly relates to the specific program objectives. Indicators must be stated as a percentage, absolute number, mean/median, or presence/absense. Examples of indicators include:
- Percentage of clinic personnel who have completed a particular training workshop
- Percentage of clinics that experienced stockouts of injectable contraceptives during the past six months
- Number of family planning visits at a specific clinic, or network of clinics
Q. What types of indicators should programs use?
A. Indicators can measure four different aspects of performance:
- Input indicators ask whether resources, such as personnel, equipment, and supplies, are sufficient and whether they are delivered on time
- Process indicators investigate how well program activities, such as staff training and counseling clients, are carried out
- Output indicators check whether expected changes in access to services, quality of care, and service utilization occur
- Outcome indicators examine whether expected changes in contraceptive use and fertility levels occur at the population level
Q. How can a program identify the most appropriate indicators?
A. It is important to select indicators that measure the objectives of the program or intervention. To be useful, M&E data must be manageable, timely, reliable, and specific. Objectives should be SMART, that is, Specific, Measurable, Appropriate, Realistic, and Time bound. Also, the data must not be too expensive or too hard to collect.
The types of data that a program collects depend on its objectives and its budget. Full M&E would use all four types of indicators. Monitoring would make use mostly of the first two types of indicators. Evaluation would focus largely on the last two.
Q. In practice, how can M&E and research guide a program?
A. M&E and research provide objective information that helps managers make good decisions. Each type of data can guide program management. Here are examples:
- Input data can tell whether supplies are coming in on schedule, so that the manager can address any logistics problems quickly.
- Process data include number of trainings, number of Information and Education and Communication materials produced (IEC).
- Output data can sometimes help measure how much of the population is being served, and so a manager can argue for more resources to fill gaps. Together, input data and output data are used to calculate efficiency, so that the manager can invest the budget in the most productive way.
- Outcome data can tell a manager and stakeholders whether changes have occurred in the behavior of the population as a whole (e.g., contraceptive use). This analysis usually requires an evaluation expert.
Q. What is operations research, and how can it help family planning programs succeed?
A. Operations research develops and experiments with various approaches to managing and providing family planning services. Thus, it can test what works best in a particular setting. This approach involves identifying program and policy problems and then designing, testing, and evaluating possible solutions. Interventions are designed with the assumption that, if successful, they will be scaled up and institutionalized. Operations research can even change national policy by providing evidence on the most efficient way to accomplish a given objective.
Q. What is formative research and how can it help family planning programs succeed?
A. Formative research (also called formative evaluation) provides in-depth understanding about the intended audience. Formative research takes place before any program design or implementation. It collects information about the audience; their knowledge, attitudes, and beliefs about health; and the factors that affect their health behavior. Evaluators and researchers can assess the audience’s access to information, services, social support, and other resources. Thus it helps programs to address their audiences effectively.
Q. How can research help programs develop in the right direction?
A. Research conducted before starting a new activity or service is called formative research—because it helps form or guide the new undertaking. Formative research often gathers information from potential clients, such as their needs, their attitudes and preferences, their current use of health care services, and their ability to pay. Formative research also may assess what resources are available to the program. For example, can the program hire health care professionals with the necessary skills? Which radio stations reach the intended audience? Good formative research can take much of the guesswork out of planning new activities.
Case Study
Scaling Up Operations Research in Guatemala
In the 1990s the Population Council conducted a series of operations research studies to improve integrated reproductive health services in the Mayan highlands. In part, the research sought ways to improve the Mayan people’s access to a wider range of contraceptive methods.
Researchers found one reason for high unmet need for contraception among Mayan women: Health care providers did not tell women about reproductive health services when they visited a clinic for curative care. Therefore, the Population Council and the Ministry of Health developed a job aid to help providers identify women’s unmet needs and offer reproductive health services at the same visit. They tested the job aid in several public clinics.
Researchers found that, in the last nine months of 1996, clinics using the job aid had more than twice as many new family planning clients as they had previously. In contrast, clinics that did not have the job aid saw numbers of new family planning clients increase by about one-fifth. These results demonstrated the benefits of using a simple job aid. The Ministry of Health later scaled up the strategy nationwide. (Brambila, 2007)
Each of the 10 elements was chosen based on online survey results and was discussed in an online forum hosted by the Implementing Best Practices (IBP) Knowledge Gateway. Read more about the survey, forum and results in the Forum Synopsis. Learn more about IBP Knowledge Gateway, and join their Initiative here.
Recommended Resources
Publications
Job Aids/Tools
Case Studies
