4.1 Overview of Setting Up the M&E System

4.1.1 Seeing M&E as a System

A well-functioning M&E system manages to integrate the more formal, data-oriented side commonly associated with the task of M&E together with informal monitoring and communication, such as field staff sharing impressions of farmer experiments with their manager over morning coffee. If you can see M&E in this way – as an integrated system of reflection and communication supporting project implementation – then you can plan for and manage it over the entire life of the project. However, M&E is often seen as a statistical task or a tedious external obligation of little relevance to those implementing the project. It is also common to see projects separating the monitoring function from the evaluation function. In such cases, the higher-level, impact-related assessments are sub-contracted and the project focuses only on tracking short-term activities, thus limiting opportunities to learn if they are having a strategic input. Seeing M&E as an integrated support to those involved in project implementation requires:

• creating M&E processes that lead to clear and regular learning for all those involved in project strategy and operations (see Box 4-1);
• understanding the links between M&E and management functions;
• using existing processes of learning, communication and decision-making among stakeholders as the basis for project-oriented M&E;
• putting in place the necessary conditions and capacities for M&E to be carried out.

Figure 4-1 illustrates the M&E system and how it links to other key elements of a project. For M&E to be effective, four core tasks need to be fulfilled:

1. Designing and setting up the system;
2. Gathering and managing information;
3. Reflecting critically (on experience and information) to improve action;
4. Communicating and reporting results.

Although Figure 4-1 shows the four tasks as a sequence, in reality they overlap. For example, a project or component manager may be discussing field observations with staff over morning coffee. In this situation, he or she gathers field observations from different people, they reflect on them together, and then share the information with others all at once. On the other hand, the production of the annual progress report will involve a more structured process with separate steps – information gathering during the year, a participatory project review process where the information is discussed and analysed, and writing the report.

4.1.2 Linking M&E to the Overall Project

The figure below illustrates how the M&E system fits within the project. In Section 2, the idea of "managing for impact" was explained in terms of four elements: guiding the project strategy towards achieving impact, ensuring effective operations, creating a learning environment and setting up and using the M&E system. Figure 4-1 focuses on the M&E element and how it links with the two elements, project strategy and operations. The remaining element – learning environment – is the context that influences how a project and its M&E system are implemented.

Figure 4-1. The M&E system and how it links to the project strategy and operations

1. The project strategy (the plan for what will be achieved and how it will be achieved) is the starting point for project implementation and setting up the M&E system (see Section 3).

2. The strategy is the basis for working out the project operations required to implement activities efficiently and effectively.

3. The completion of project activities leads to a series of actual outputs, outcomes and impacts. Comparing the actual outputs, outcomes and impacts with what was planned in the project strategy and understanding the differences in order to identify changes in strategy and operations is a core function of the M&E system.

4. The M&E system consists of four interlinked parts.

4a. You start setting up the M&E system by identifying information needs to guide the project strategy, ensure effective operations and meet external reporting requirements (see Section 5). Then you need to decide how to gather and analyse this information and document a plan for the M&E system. The process of working out how to monitor and evaluate a project inevitably raises questions about the project strategy itself, which can help improve the initial design. Setting up the M&E system with a participatory approach builds stakeholders’ understanding about the project and starts creating a learning environment.

4b. Implementing the M&E system means gathering and managing information (see Section 6). You can do this through informal as well as more structured approaches. Information comes from tracking which outputs, outcomes and impacts are being achieved and checking project operations (e.g., activity completion, financial management and resource use). After information gathering and management starts, you will need to solve problems or will have new ideas for improving the initial M&E plan.

4c. Involve project stakeholders in reflecting critically (see Section 8). Once information has been collected it needs to be analysed and discussed by project stakeholders. This may happen formally – for example, during the annual project review workshop. Or it may happen informally – for example, by talking with farmers about their ideas during weekly field visits. In these reflections and discussions, you will probably notice information gaps. These can trigger adjustments to the M&E plan to ensure the necessary information is being collected.

4d. The results of M&E need to be communicated to the people who need to use it. Only then can you call the M&E system successful (see Section 6). This includes reporting to funding agencies but is much broader. For example, problems experienced by field staff need to be understood by their manager. Project progress and problems must be shared with project participants so you can identify solutions together. Reports to funding agencies need to balance successes and mistakes and, above all, be analytical and action-oriented. Some of those who are to use the information may have been involved in collecting data and/or analysing part of it. However, you need to plan how to inform those who were not involved.

5. Ultimately, the results from M&E – both the communication processes and the information – will improve the project strategy and operations. Senior management is responsible for seeing to this with the support of M&E staff. Sometimes improvements can be immediate. For example, extension staff may be complaining one day about a vehicle maintenance problem, which the project manager can act on directly. Or there may be a need to change the sequence of certain activities, which the responsible unit manager has the flexibility to do. But sometimes more extensive negotiations may be required between the project director, the supervising ministry, the cooperating institution and IFAD. For example, if a supervision mission notices major problems with an entire project component, such as micro-credit, changes to the loan agreement may be necessary.

So for M&E to work as a tool for managing for impact, project management and the M&E staff need to be clear on how to identify, agree upon and follow up on project improvements. If this process for guiding change is not in place, even a very good M&E system will not have much value for the project.

4.1.3 Key M&E Tasks during the Project Cycle

Considering M&E as a system helps in understanding the range of M&E tasks that different people will need to undertake during the project cycle. The list below looks formidable indeed. But look closely at it and you will probably recognise that you are already implementing many of the tasks as part of your M&E responsibilities. Furthermore, these tasks are specific for each stage of the project and most of them will be shared among a range of people. Note that these M&E tasks are not the sole responsibility of M&E staff or an M&E unit, if the project has these. Rather, they should be seen as functions for which responsibility is to be shared. This makes it critical to give careful thought to whom to involve in each one. Making the M&E system and processes more participatory means sharing these functions. This, in turn, makes shared learning through M&E possible.

Figure 4-2. Key M&E tasks

Each project will need to modify this list to suit its operating conditions. Box 4-2 shows one example of project-specific M&E functions in Guatemala. Another example is from the SAIP project in Bangladesh, where M&E specialists were hired during start-up to work with project management to design and operate the M&E system. The M&E specialists also took responsibility for developing the impact-monitoring methodology and reaching agreement on indicators, with the participation of NGOs and primary stakeholders. Since this project was focusing primarily on participatory development, there was the vital need for a thorough introduction to participatory impact monitoring at an early stage. It was the responsibility of the M&E specialists to draw up and implement a training plan for M&E staff at district and field levels.

Yet another example comes from Benin. At project start-up, the M&E coordinator developed a manual for guiding the project’s monitoring and evaluation process. The manual provides formats for recording revenues from income-generation activities, details on collecting data for the indicators, and information on how field extension agents, NGOs and heads of departments should report. The manual is considered to be an evolving process, influenced by the experiences of people using it. So revising it will be a recurring M&E task at later stages.

4.1.4 Responding to Unplanned Needs and Requests for Information

So far, M&E has been discussed as if it were a process that can be planned entirely ahead of time. However, most project M&E units will often receive sudden demands from the project manager, ministries, steering committees and funding agencies to provide a report on a specific issue, or some other type of information.

When developing the budget, include a budget line for unplanned costs, say a 10% contingency allocation. Leading corporate research organisations typically save part of the research budget and time for projects that do not fit into the established categories. Sometimes up to 25% of the research budget is left open to ideas that do not fit into existing categories. Alternatively, you can top off specific budget lines with resources for unplanned activities. For example, you can add two weeks per year for consultants to deal with focused evaluations or monitoring questions that might be requested during the year. Also plan in extra time for key staff to deal with unplanned M&E-related requests. Three days per month, for example, may seem like a lot. But, in practice, even more days are quite commonly required.

In Cuchumatanes, Guatemala, the M&E unit carries out what it calls "emergent evaluations", which the project director requests when more information is required about certain activities. The M&E unit produces "alert reports" that highlight the need for extra information. Some additional thematic evaluations or research studies are contracted out. Among the emergent evaluations thus far, were case studies about organisational issues, an evaluation of the communal banks programme and an evaluation of environmental impact.

During project design, include a category in the appraisal report such as "sudden opportunities" or "M&E contingency" and identify the upper budget and time limit for unplanned M&E activities. If managers keep track of the time and money spent on unplanned M&E activities and see that the limit has almost been reached, then they can more easily explain to those requesting additional reports or information why it is not possible to respond to all demands.

Back to Top

4.2 Key Steps in Setting Up the M&E System

The six steps involved in designing an M&E system are:

1. Establishing the purpose and scope – why do we need M&E and how comprehensive should our M&E system be?
2. Identifying performance questions, information needs and indicators – what do we need to know to monitor and evaluate the project in order to manage it well?
3. Planning information gathering and organisation – how will the required information be gathered and organised?
4. Planning critical reflection processes and events – how will we make sense of the information gathered and use it to make improvements?
5. Planning for quality communication and reporting – how and to whom do we want to communicate what in terms of our project activities and processes?
6. Planning for the necessary conditions and capacities – what is needed to ensure our M&E system actually works?

A good appraisal report will include an indicative M&E framework that provides enough detail about these questions to enable budgeting and allocation of technical expertise, give funding agencies an overview of how M&E will be undertaken, and guide project and partner staff during start-up. But this will only be indicative and needs to be adjusted and detailed further during the start-up phase (see Box 4-3).

Below, you will find the tasks for each step in taking the outline of an M&E system from an appraisal report and designing the details that make it operational (see Table 4-1). More detailed explanations for each step can be found in Sections 5 through 8.

Table 4-1. Tasks needed when detailing the M&E plan based on a project appraisal report

Careful thought is needed regarding whom to include when detailing the M&E system. In the first years of the PADEMER project, Colombia, for example, the monitoring unit designed the M&E system. One part of this process included a workshop to define the principle indicators. Participants included: the national technical coordination unit, the implementing NGOs, the Mixed Corporation for Micro-Enterprise Development and members of the management committee (ministries of agriculture and economic development and the national planning department).

The more diverse stakeholders you can include in the process, the more all-round clarity you will eventually have as to what is needed from whom and when. But this will take more time – participation inevitably does!

4.2.1 Purpose and Scope of the M&E System

Clear definition of the purpose and scope of the intended M&E system helps when deciding on issues such as budget levels, number of indicators to track, type of communication needed and so forth. The appraisal report will include a brief M&E purpose statement (see Box 4-4), but you will need to revisit this question at start-up with representatives of implementing partners and primary stakeholders.

Whenever you are unclear about a decision on whether to monitor more or less or whether to choose one methodological option or another, you can return to the stated M&E purpose for guidance. Specifying the purpose also helps to make clear what can be expected of the M&E system, as it forces you to think about the nature of the project and the implications for information needed to manage it well.

When formulating the purpose at appraisal or revisiting it during start-up, ask yourself the following question:

• What are the main reasons to set up and implement M&E, for us – as implementing partners and primary stakeholders – and for other key stakeholders?

With a shared understanding of the overall purpose, the next step is to clarify the scope of the M&E system. "Scope" relates to the extent and degree of sophistication of the system. M&E systems can be highly sophisticated, requiring considerable expertise in qualitative and quantitative research methods and extensive information management. They can also be very simple systems that rely largely on discussions with stakeholders and do not try to gather large amounts of data. These different systems will not yield the same results. Each has specific advantages and disadvantages, such as degree of precision and capacity required and labour and cost involved. The sophistication of the M&E system that is appropriate to your situation will depend on your M&E purpose, available resources and M&E expertise. Define the scope of the M&E system by asking:

• What level of funding is potentially available?
• What level of participation in M&E by primary stakeholders and partner organisations is desirable and feasible?
• How detailed does the M&E information have to be, either in terms of quantitative or qualitative data?
• What sort of baseline study is desirable and feasible?
• What are the current M&E capacities among primary stakeholders and partner organisations, and how will this affect the desired levels of M&E?

4.2.2 Performance Questions, Information Needs and Indicators

The most common approach when setting up project M&E is for the M&E coordinator to take each objective and start listing quantitative indicators in the second column of the logframe matrix. This often creates problems. The problems arise not from the quantitative indicators but from the process of jumping directly from objectives to indicators. Many objectives are complex so cannot be summarised in terms of one or a few indicators. Also, while it might be possible for quantitative information to be found that shows if objectives are being met, it does not necessarily explain why and if this can be attributed to the project. Therefore, multiple sources of quantitative and qualitative information are critical to explain what is happening and to look closely at relationships between different pieces of information, rather than single indicators.

Working with performance questions (see Box 4-5) to guide indicator analysis will give you a more integrated and meaningful picture of overall project achievements. Answering these questions requires descriptive analysis and quantitative information. Starting by identifying performance questions makes it easier to recognise which specific indicators are really necessary. Sometimes a performance question can be answered directly with a simple quantitative indicator. However, very often the question can only be answered with a range of qualitative and quantitative information.

Performance questions are needed for each level in the objective hierarchy but also for the project as a whole. For example, you will most probably want to ask some questions about the process of project implementation, such as the quality of relationships between certain target groups and implementing NGOs. Or maybe, "How do project partners feel that the project management unit can improve to enable them to carry out their responsibilities?" You will also want to keep track of unanticipated impacts – for example, "Have any of the project innovations been adopted by people in neighbouring districts who are not the main target group?"

Remember that information needs will shift over time, so performance questions will need revision. In projects set up based on the idea of performance questions, the project appraisal report includes a list of indicative performance questions and indicators. After revising the objective hierarchy with key stakeholders at start-up, you can refine these indicative questions together to fit any revisions made to the project strategy.

For most projects, performance questions will not (yet) exist. To develop good performance questions, you first need to be very clear about the project aims. So the process of finding performance questions with stakeholders will help you further refine the project design together.

In IFAD-supported projects, identifying performance questions and selecting indicators increasingly involves the intended primary stakeholders. The process of participatory performance questions and indicator identification is quite different from one that limits itself to the project team. This issue and others about performance questions and indicators are discussed in more detail in Section 5.

4.2.3 Information Gathering and Organising

Many appraisal reports include a long list of indicators. Yet often little thought is given to the practical implications of gathering the required information, not to mention how it will be used to manage for impact. During start-up, a critical task for all implementing partners is to assess what information can realistically be collected, given available human and financial resources.

For each information need or indicator, you must establish how the information will be collected and organised. For example, monitoring progress on irrigation infrastructure development may require that primary stakeholders and project staff check what infrastructure has actually been constructed and if it is working properly. This is relatively straightforward. However, monitoring the impact of the irrigation development – for example, in terms of changes in household income – requires a different method. One method you could consider is household surveys. This is a fairly time-consuming and expensive monitoring activity and one that does not make primary stakeholder participation easy. A different option might be village-led surveys and open discussions with impact flow diagrams (see Annex D) about how daily life has changed as a result of irrigation. Another example might be monitoring the implementation of an experimental microenterprise development fund. For this, you might choose to hold regular meetings or workshops with the implementing partners and entrepreneurs.

Not only will each indicator require choosing a different method, but for each indicator or information need you will usually present several options. Annex D describes many monitoring and information gathering methods – qualitative and quantitative and individual versus group-based. They range from simple record-keeping forms to agronomic assessments of yield changes, household surveys and participatory workshops. For example, instead of a detailed and extensive household survey on child immunisation, you could hold a focus group meeting with mothers to discuss the extent of immunisation, opinions on how this service is being provided, etc. Each method has specific advantages and disadvantages in terms of cost, reliability of data, skill needed, ability to quantify results and richness of information generated.

Particularly critical at the moment of method selection is knowing who will be involved in collecting, compiling and analysing. The more that the intended users of the methods can be involved in selecting or developing the methods, the more chance there is that they will understand them and use them correctly. If methods are selected by someone not using them, then training users in the methods will be essential.

Gathering data is one thing. But each bit of information also needs to be collated, perhaps summarised and certainly analysed by the right people. This will need to be planned in detail at start-up. Project field staff are often only involved as data collectors and primary stakeholders only as data providers. Seeing monitoring as a learning process implies that analysis and agreeing on actions are undertaken with all levels in the project hierarchy and with partners. As a general rule, data collection and analysis should be undertaken with those to whom the data, analysis and decisions pertain and, therefore, at the relevant level. Field staff would, for example, need to understand about project reach within the administrative division for which they are responsible, while the project director needs to analyse project reach for the entire project area.

See Section 6 for practical aspects of information gathering and management.

4.2.4 Critical Reflection Processes and Events

How can people be involved in making sense of the data generated by M&E processes and in assessing the implications for the project strategy and operations? This aspect of M&E receives no attention in the logframe matrix and is rarely given the attention it needs during project design.

The M&E section in the appraisal report may have outlined the main critical reflection processes and events. But this is rare. Usually during start-up, you need to plan such processes in detail (what will be the focus, who will participate, will they be facilitated or self-managed, how will one process feed into others, etc.) and schedule when they will occur (see Table 4-2). Section 8 deals with critical reflection in detail. Figure 2-3 in Section 2 shows some common reflection events that occur during the life of most projects and how they complement each other.

Table 4-2. Example of the critical reflection schedule for a project

Critical reflection can occur formally and informally. Formally, it can be facilitated during project meetings, workshops with partners and primary stakeholders or as part of external evaluations. Informally, it can occur in ongoing discussions between project stakeholders. There are endless examples of how reflection can be encouraged. For example, in Cuchumatanes, Guatemala, individual learning was encouraged by having core M&E staff participate in ongoing field activities. This gave them a clearer understanding of project operations: the relationships and how primary stakeholder generally accept and view operations. In Tropisec, Nicaragua, the extension agent of the implementing partner meets up with grassroots organisations to analyse current progress and results and to identify possible actions and resource requirements for the coming period. As part of their annual work programme within the project, all technical project staff are expected to share at least one significant learning experience with colleagues.

4.2.5 Communication and Reporting

The appraisal report will usually specify the expected reporting schedule, indicating who should receive a report and when. Often these will be reports for the funding agencies for accountability reasons. However, you also need to communicate M&E findings to many other stakeholders and for different reasons. For example, implications for policy must be shared with government officials, use of funds can be shared with partners and primary stakeholders, and documented lessons learned about the project strategy should be distributed to other projects.

During start-up, develop a detailed idea of your communication strategy. Include not only formal reports but also communication efforts that seek feedback about interim findings, and discuss what actions are needed.

• With representatives of all key stakeholders, develop a precise list of all the audiences, what information they need, when they need it and in which format.
• Spend some time discussing why each of these audiences needs information. To seek feedback for verification? As input for discussions of the implications for project strategy and operations? To clarify their responsibilities? For accountability? To influence and gain more support for field activities?
• Schedule clearly the production of information needs, showing who is to do what by when in order to have the information ready on time.
• Organise the events during which the information is to be communicated and discussed.

Section 6 discusses principles of communication and ways to seek feedback. Annex D includes methods you might find helpful to facilitate communication.

4.2.6 Necessary Conditions and Capacities

In the appraisal report, you will find an indicative budget for M&E and a description of how M&E should or could be organised. But getting the M&E system working also means thinking of appropriate incentives, ensuring you have the right and enough human capacity at hand, and thinking about ways of storing and sharing information. The necessary conditions and capacities for the M&E system to function were introduced in Section 2. They are dealt with in detail in Section 7. Table 4-3 lists key questions that need answers when detailing M&E to make it operational.

Discuss appropriate organisational structures for M&E at start-up. This is critical to the success – or failure – of M&E. It is the moment when negotiations need to reach decisions about each of the partners’ responsibilities and information requirements. IFAD-supported projects often have one of two basic organisational arrangements for M&E:

1. M&E is coordinated by an M&E coordinator or unit within the project management unit (and supplemented by external M&E contracts) to facilitate management’s quick use of information;
2. M&E is carried out by a separate M&E group – for example, subcontracted to a research institute or located within a government department – aiming to provide more objectivity and independent analysis.

Overall, experiences from many projects suggest that M&E is much more effective when those implementing M&E are part of project operations and decision-makers. This can be supplemented in important ways by more external M&E support. And other innovations are emerging that place M&E firmly in the hands of primary stakeholders or field teams (see Box 4-6).

Table 4-3. Questions to guide the detailed planning of M&E conditions and capacities

Once most of the detailed M&E plan has been completed, you can take a fresh look at the M&E budget. Box 4-7 provides a list of likely M&E costs. How you cost M&E depends on whether you allocate resources to specific M&E activities or whether you include M&E in generic categories, such as "staff training", "participatory workshops", etc. Details on budgeting for M&E are discussed in Section 7.

Back to Top

4.3 Documenting the M&E Plan

Projects have three core documents that serve to guide M&E:

1. The M&E framework in the project appraisal report (see Section 3 and Section 4.2, Table 4-1, for a discussion);
2. The project implementation manual;
3. The M&E plan or manual.

4.3.1 M&E in the Project Implementation Manual

Most projects allocate time and resources to develop what is known as a "project implementation (or procedures) manual". This is a set of guidelines with information about financial accounting procedures, procurement procedures, guidelines for staff travel, guidelines for the use of vehicles and other equipment, and other details necessary for the smooth operation of the project.

The detailed M&E plan may be part of the project implementation manual, an annex to it or a separate document. Irrespective of where it can be found, the implementation guidelines and M&E guidelines must be closely linked and, above all, coordinated. Contradictions or ambiguities in the two sets of guidelines must be avoided.

As the detailed M&E plans may contain an overwhelming degree of detail, summaries for all project participants are helpful to keep everyone focused on their responsibilities. A good way to summarise specific inputs is in an M&E timeline for everyone who plays an important M&E role (see Box 4-8). Ideally, these timelines should be integrated within weekly and monthly activity timelines so that M&E becomes an integrated part of activities.

Box 4-8. Working with an M&E timeline