![]() |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
2.1 An Overview of Using M&E to Manage Impact Rural development projects aim to improve the lives of the rural poor. As manager of a project or part of it, do you always know what impact you are having and why? Learning about successes and failures through regular monitoring and critical reflection is fundamental for guiding your intervention towards achieving maximum impact. Monitoring and Evaluation (M&E) is the heart of "managing for impact" (see Box 2-1). "Managing for impact" means you need to respond to changing circumstances and increased understanding by adapting the project so that it will be more likely to achieve its intended impacts. Such adaptation may entail small changes to activities or larger strategic revisions. Each project is being managed for impact under its own set of constraints. Keeping updated on the internal and external constraints will help you have realistic expectations of what can be adjusted and achieved. To manage adaptively, project implementers and managers need to:
Managing for impact is only possible if you have reliable information about the progress of activities and their outcomes, the reasons for success and failure, and the context in which activities are taking place. This information is the output of your M&E procedures. Analysing this information with key stakeholders can support good decisions that improve the project. Only when information helps the project reduce poverty more effectively will an M&E system be worthwhile. Taking poverty seriously has implications for M&E, as RDRS (Rangpur Dinajpur Rural Service) in Bangladesh is aware. With the need to recruit more selectively from severely impoverished groups, it is moving to monitor individual households rather than credit groups. This will allow the organisation to improve its social accountability but also its services to the poor of northern Bangladesh. 2.1.1 The Four Elements of Managing for Impact To know if you are managing for impact, check Figure 2-1 to see if you and other implementing partners are putting in place the four basic elements that will give you the information and insights you need. 1. Guiding the Project Strategy for Poverty Impact - understanding the goals and objectives of the project and then allocating the available resources and guiding relationships between stakeholders to maximise impacts. 2. Creating a Learning Environment - inspiring and helping those involved with the project to reflect critically on progress, to learn from mistakes and to generate ideas for making improvements. 3. Ensuring Effective Operations - planning, organising and checking staff inputs, equipment, partner contracts, financial resources, (bi-) annual work plans, and communications to implement activities effectively and efficiently. 4. Developing and Using the M&E System - designing and implementing information gathering and reflective learning processes to generate insights that help you to improve operations and strategic directions. Figure 2-1. Key elements in managing for impact
Table 2-1 illustrates how an organisation may or may not put these four ideas into practice. It can be used as a checklist by project managers to assess how well the project is doing in terms of managing for impact. Table 2-1. Example of projects that do and that do not manage for impact
2.1.2 Guiding the Strategy for Poverty Impact A typical IFAD-supported project has about seven years during which to have an impact on poverty. At start-up, the project strategy is first revised. This is a critical opportunity for strategic guidance. After this, the project's energy tends to focus on putting capacities and procedures in place; while towards the end, efforts often focus on consolidating impact and phasing out the project presence. This leaves the project stakeholders with three or four years of prime time to take corrective action. To know how best to mobilise project resources and partnerships for reducing poverty, the implementers need to understand the project strategy and redirect it when problems arise. Guiding the strategy is largely about asking the right questions - and getting answers - at the right moments (see Box 2-2). M&E processes are critical for making collaborative decisions about adjusting the project's direction. Every project has an overall logic (see Figure 2-2 and Box 2-3) that describes what is to be achieved, why and how. The project logic starts by describing the situation that a group of stakeholders wishes to improve. The vision about how this situation can be changed should be based on the problems and aspirations of stakeholders, and particularly of the rural poor. This vision justifies the project's existence. Inevitably there will be different perceptions of the key problems and what constitutes an improvement. This is why participatory approaches to planning are so important (see Sections 2.7 and 3.2 for more details on participatory planning).
After making sure you have a clear "objective hierarchy" and the project is underway, guiding the project strategy for poverty impact becomes a matter of steering - continually checking, questioning and correcting. Proof that you have been guiding the project strategy well is when, from year to year, all stakeholders can see an improvement in how activities are implemented, in relationships between stakeholders and in the quality of project outputs. Focusing only on poverty reduction is a long-term goal. So it is not very useful for everyday guidance. Reflections on the project's direction in, for example, monthly or quarterly meetings will focus on activities, outputs and processes, but will also require adjusting assumptions that underpin the project. Each year, a close look at the overall poverty reduction goal is needed to manage for impact. This is when impact monitoring plays a central role. 2.1.3 Encouraging Learning through Critical Reflection and Participation It is the people involved in a development intervention who will make it succeed or fail. Their participation in learning how to improve a project throughout its existence is fundamental. For project and partner staff, this means listening carefully and regularly to the views of different groups - including each other - about what is working and what is not, and hearing reasons for why problems exist and what needs to improve. Learning certainly requires more than only "listening". Opportunities need to be created for staff from the project and implementing partners and primary stakeholders to meet and analyse their experiences with the project. Section 8 offers ideas to encourage reflection and critical analysis. A
good M&E system provides and communicates data to help stakeholder
groups analyse progress. M&E is nothing more - or less - than
an open and critically reflective communication process to strengthen
project partnership. Putting participatory learning at the centre
of good project management requires both data on project activities
and personal accounts of people's experiences. It requires regular
reviews by project staff with primary stakeholders, supported by occasional
inputs from outside specialists. In
many development interventions, people lose motivation if they are
either not invited to participate or the conditions are not created
for their meaningful participation. Participation in M&E is meant
to provide opportunities for people with relevant views about the
project to learn how to improve it. Behind this simple statement lie two questions. First, who has a relevant view? As it is neither practicable nor necessary to include everyone in developing the M&E system, choices need to be made. Second, how can different people best be involved? Different stakeholders' involvement in key M&E tasks needs to be negotiated. Do they want to participate, and under what conditions, in:
A participatory learning approach to M&E needs to be supported by a participatory style of management. If you are going to involve many stakeholders successfully in M&E, then they need to feel that the managers encourage and reward critical reflection. Only then will everyone be able to learn from problems in order to make the project work better. If management has a primarily controlling and punishing function, then those involved will not want to risk innovation. This will greatly limit a project's ability to manage for impact. 2.1.4 Ensuring Effective Operations Ensuring
effective operations requires putting in place the practical and operational
conditions for carrying out project activities efficiently. Operations
are guided by the annual work plan and budget (AWPB) and by regular
meetings with implementers and primary stakeholders. 1.
Staffing: numbers, salaries, capacities, quality of
performance; 2.1.5 Steps in Developing the M&E System Figure 2-1 shows that managing for impact is based on the M&E system. Developing an M&E system involves six steps (see Section 2.5): 1.
establishing the purpose and scope; Each of these design steps needs to be dealt with twice. First, as part of the appraisal report, in which the design team describes a general M&E framework for each of these six elements. Second, during project start-up, the project staff need to transform the general M&E framework into a detailed operational M&E plan. The six steps are discussed in detail in Sections 4 to 8. The outputs of the project M&E system will provide answers to the five questions that guide the project strategy. Looking at the questions in Box 2-2, operational management is more frequent and must focus on the questions of "effectiveness" and "efficiency". More strategic reflections, for example during annual reviews and supervision missions, will look at the questions of "relevance", "impact" and "sustainability".
2.2.1 Key Opportunities to Manage for Impact The four elements of managing for impact are the broad processes that need to happen in any project. But on a daily basis, project and partner staff and primary stakeholders carry out more specific management functions, each of which presents an opportunity to refocus on impacts.
The idea of managing for impact can be implemented quite simply in regular events, such as annual project reviews, quarterly and mid-year partner/staff meetings, and during supervision missions. These ideas are easy for existing projects to implement as part of their current processes and for new projects to plan into their operating procedures. For example, projects increasingly hold annual reviews with primary stakeholders as part of their ongoing self-evaluation process. During such a review, staff, partners and local people will discuss the monitoring data on activities, outputs and outcomes. They will analyse them with respect to project goals to see how activities are - or are not - contributing to poverty reduction. They will also discuss the quality of the project implementation process and of relationships amongst stakeholders. This leads to formulating the next annual work plan and budget (AWPB) and adjusting M&E plans. This self-assessment and development of the AWPB form the basis of the annual progress report, but more strategic issues can also emerge from community-level discussions (see Box 2-4). So an annual review process links all four elements of managing for impact: impact, strategy, operations and M&E. Quarterly and mid-year review and planning meetings could operate similarly to a participatory annual review, perhaps with fewer stakeholders and more discussion on the quality of implementation and relationships. Using monitoring data in discussions can challenge people's impressions and the assumptions and so trigger analysis of what is really happening. Quarterly and mid-year reports could then focus on the achievement of activities, analysis of key achievements and problems, and - most importantly - on agreeing how to improve implementation. Such regular and improvement-oriented self-assessments are proof of a healthy learning environment that focuses on achieving impact by organising and implementing operations effectively. Supervision missions and mid-term reviews are also occasions when all four aspects of managing for impact come together. But a project cannot rely on these alone, as MTRs come too late in a project's life (see Box 2-5) and supervision missions are not always in enough depth or timed appropriately to influence impact achievement. When project implementers take responsibility for their own learning process, they can then take corrective action when it is needed and not when it is too late. Such action involves redressing mistakes, expanding good practices, responding to changes in the context by rethinking activities and processes, and taking up new opportunities. 2.2.2 Knowing Your Information Needs and Planning Learning Opportunities Clarity about your information needs helps in structuring the M&E system and, in particular, knowing how to make the most of events such as half-yearly revisions, MTRs and annual participatory reviews. To check if your M&E system is providing the information you need, refer to the five questions in Box 2-2. If part of the picture is missing for you, then you need to adjust existing M&E processes. Not only managers have information needs. Everyone involved in the project has specific tasks in the project and therefore specific information needs. A project M&E system must consider as many needs as possible. Only then will managing for impact become a participatory learning process. Section 5 discusses some ways to deal with different information needs. For this reason, all projects must have a range of learning events. For instance, the Tropisec project in Nicaragua seeks inputs from 16 different events and information sources (see Section 2.7.2). To use information most effectively for managing for impact, think about the key moments during the project life when strategic decisions are made that enable you to move closer to a poverty reduction impact (see Figure 2-3). Information from M&E will be most useful if it feeds into these moments. Thinking about these key moments as learning opportunities to manage for impact can reveal their value as strategic steering exercises, rather than as obligatory. Keep the level of discussion and type of decision appropriate to the event. For example, an annual review process is usually not the best moment to discuss how to organise the delivery of stationery supplies to village groups. Nor is a weekly staff meeting the appropriate place to agree on the new terms of contracts with partners. Figure
2-3. The sequence of learning opportunities during the project cycle 2.2.3 Recognising and Tracking Operating Constraints in the Project Context Understanding context is critical when it comes to assessing relevance of strategy and activities, anticipating operational problems and judging a project's contribution, and also when designing the M&E aspects of a project. You can start to disentangle the project's contribution by analysing the evolution of project interventions alongside other concurrent external influences that affect primary stakeholders. Continual updating on context also allows you to adapt the project strategy and operations while en route. A systematic and regular assessment of operating conditions helps with anticipating some of the operational issues that could arise. For example, in Nepal, one project manager receives quarterly "political situation monitoring" reports to understand how the project may be affected. These reports include a brief description of that period's critical events - such as safety and insecurity, government actions, demonstrations and dialogues - and their implications. The information is collected via key informants, newspapers, radio and television. It is essential for deciding whether or not to suspend activities in an insecure area. Because contexts vary across projects and during the lifetime of a project, constraints and coping strategies will also vary (see Box 2-6). It is important to adjust project strategy and implementation to the extent that circumstances allow. The first element of managing for impact - guiding the project strategy (see Section 2.3) - may be particularly helpful for dealing with changing contexts. Section 3, on adapting the initial project design, also provides useful ideas.
Table 2-2. Project features with methodological implications
Table 2-4. Project features with implications for the quality of information
Let's turn now to the first of the four elements of managing for impact in more detail - the project strategy and how to guide it. The main output of the formulation phase is a draft strategy for the project. The strategy consists of an objective hierarchy and a description of the necessary implementation arrangements and resources needed. The strategy is the basis for the project appraisal report. For IFAD-supported projects, the project strategy is based on the logical framework approach (LFA) and project objectives are summarised as a hierarchy in a logframe matrix (see Section 3). But irrespective of how the project strategy and objectives are formulated, the ideas in this section remain relevant. By
necessity, any project strategy simplifies reality. It cannot possibly
describe all details of a context and of the intended plan. This means
that the documented strategy is a management tool that requires continual
adjusting to reflect current contexts and changing needs. 2.3.2 The Objective Hierarchy and Assumptions The objective hierarchy is the spine of the project strategy (see Figure 2-4 and Box 2-7). The hierarchy describes how lower-level activities contribute to higher-level objectives, and how these, in turn, help achieve the overall project purpose(s) and goal. Figure
2-4 shows an objective hierarchy with four levels and assumptions
between all levels. Development organisations use many different names
for the levels in the objective hierarchy and even different numbers
of levels (see Section 3).
But all levels can be considered objectives, as they are something
that the project stakeholders want to achieve. This is why the term,
objective hierarchy, is used (although the terms, intervention logic
and narrative summary, are also used). The hierarchy is the first
column of the logframe matrix and shows how the "means"
lead to the "ends" of an intervention.
Getting the logic of the hierarchy clear with the implementing partners, including primary stakeholders, is essential - although this does not necessarily mean using the logframe terminology with them. Simply ensure that there is consensus about what is to be implemented in the short-, mid- and long-term and for what overall purpose and goal. If a hierarchy is not logical, then you may end up implementing many different fragmented activities that do not lead to a clear output. Poor logic can most certainly lead to project failure (see Box 2-8). Or you may be promising to deliver an impact for the rural poor that is totally impossible given your timeframe and budget. Finally, if you are not clear about what you intend to achieve by when, then you will find it very difficult to know precisely what you should be monitoring and evaluating.
Figure 2-4 also shows that a project strategy contains many assumptions. An assumption is any condition outside the direct control of the project that is important for the project to succeed. There are two types of assumptions: external factors and those relating to the internal cause-effect logic (see Box 2-9). You will need to make more assumptions as you move higher up the objective hierarchy, as the project is only one of often many stakeholders contributing to rural poverty reduction and so there are increasing issues outside the direct control of the project. Assumptions need to be identified in initial project design, but this is often poorly done. The assumptions are an important tool for guiding the project strategy. Identifying them helps you know if the project strategy has a reasonable chance of success or is based on unlikely assumptions. Checking them regularly to see which ones are risky for the project, updating them based on better understanding from experiences, and identifying new ones is critical for guiding the project strategy. See Section 3 for ideas on how to work with assumptions as part of your M&E system. 2.3.3 Accountability and Adjusting the Project Strategy Knowing how to adjust the project strategy requires knowing how flexible you can be. There are two ways in which projects are adapted.
Knowing how to adjust the project strategy requires clarity about what project management is accountable for achieving. You should adjust whatever is necessary to be convinced that you can deliver what you are accountable for - and no more or less. A project's control over factors in the project environment that influence the achievement of objectives decreases with each level of the objective hierarchy (see Figure 2-5). At the level of "activities", project implementers have much control. External factors are unlikely to threaten the implementation of activities seriously. But at the level of the "goal", many factors beyond the direct control of implementers will influence impact. At this level, the project is one of many stakeholders contributing to the reduction of rural poverty. So a project's accountability at higher levels also decreases but never disappears entirely. Figure 2-5. The limits of control and accountability in a project
For example, a project might include training activities for farmers. The project can directly control the hiring of a training venue, the preparation of materials, the provision of a qualified trainer and the invitation of suitable participants. It has less control over whether potential participants will attend and considerably less control, if any, over whether the skills the participants learn will actually be used back in the workplace. While the training unit of a project can be held accountable for making the training relevant and accessible, it cannot be held accountable for whether the farmers have all the necessary conditions on-farm to implement the new skills they have learned. Therefore, if you are concerned that the project purpose can only be achieved if certain adaptations are made - and you know that you are accountable for delivering the purpose-level outcomes - then you will need to revise the strategy accordingly (see Box 2-10). You may need to negotiate with funding agencies and supervising organisations for approval of suggested changes. The limits of accountability may need to be negotiated with funding agencies - critical if you are to keep the expectations of impact realistic. To guide the project strategy, you have several management tools. 1. The idea of negotiation and shared decision-making with implementing partners and primary stakeholders may seem so obvious as to be ignored. Building agreement of the need for and types of changes is the basis of strategic guidance, as this increases the chance that changes will be carried out. Negotiations about change will also be needed vis-à-vis the funding agencies and the government structures in which the project works. Not all changes will be possible but with good M&E data to explain why changes are desirable, your manoeuvring space might increase. 2. The objective hierarchy is a valuable focus for regular (semi-annual and annual) review and planning events during which you compare achievements with targets and try to understand why differences occur - which they inevitably will. Many projects focus on activities during progress reviews, but this is not enough to manage for impact. What a project aims to achieve is the purpose and goal level, while outputs and activities describe how it thinks it can do this. A progress review needs to look both at the "how" and the "what" question. Looking only at the activities and outputs, you could conclude these are all going as planned. But you also need to ask, "Where is this leading?" to know if you are on track with the planned outcomes and impacts. This will help avoid wasting time and resources on pointless outputs and activities. 3. Assumptions can be reviewed regularly to check if they are still valid. Identify new assumptions that have emerged and delete those that are no longer relevant. 4. Based on your assessment of problems, successes and revised assumptions, check each level of the hierarchy for relevance and completeness. Add new activities or outputs and delete irrelevant ones in line with your assessment. Adjust targets as necessary. This may require negotiation with funding agencies, particularly at higher levels. 5. An M&E system can provide data for and organise reflection processes around points 1 to 4 above. Box 2-11 shows how one project in Venezuela is assessing achievement of project component outcomes and linking them to an improved strategy.
2.4 Creating a Learning Environment 2.4.1 What Is a Learning Environment? The second element of managing for impact concerns the learning environment that needs to be created if people are to provide strategic and operational guidance by reflecting critically on what is happening. How can you know if your project is actively learning? If you can clearly say "yes, this is happening here" to the following items, then you know that in terms of managing for impact your project is well on its way to having a culture of learning through critical reflection:
A
learning environment can be created through many small changes as well
as more far-reaching events and changes. One project in Tanzania integrates
more than 20 different ways of working in order to stimulate learning
- from the very way in which the project is designed to how fieldwork
happens as well as annual reviews with villagers in the project area.
Critical to this is the attitude of and example set by senior management
and also a dialogue between implementing partners (see Box
2-12). 2.4.2 How Management Styles Affect Learning In one project in Latin America, the project manager asks colleagues to evaluate her performance. This type of management attitude is rare. Yet it sends a clear message of being open to feedback and prepared to learn from colleagues. Few managers are selected on their management skills and attitudes, but all can work on improving the needed skills. Staff in one project identified two essential qualities required by project managers to support monitoring and learning:
While the wider cultural context will affect how a project creates its learning environment, the internal project culture lies within the influence of a project manager. When two projects in the same country were compared, one was designed better but managed the M&E system poorly, while the other had a poorly designed but well managed system and appeared to be doing better. Good management can deal with many problems, even those of poor design. Encouraging
learning does not have to be complex (see Box 2-13).
In Ghana, the World Bank director has an "open door" policy.
If project staff have a quick question, they can call and ask the question
and frequently get verbal clearance and advice immediately. This streamlines
project decision-making because team members do not have to try and
frame the question in a letter, mail it, then wait for a reply. This
has been particularly helpful in procurement and financial management.
An even simpler idea comes from India, where the project director has
a habit of extensive touring to the local development groups. This enables
more qualitative monitoring and direct impressions. The project director
provides a feedback letter to whomever he has visited. Section 8 gives
more ideas on encouraging critical reflection. 2.4.3 Valuing Problems to Avoid Failure You may have had the experience of reading a report about a project that you know well and realised that what you are reading is a very different story than what you know to be true. It is likely that what has not worked well is not being reported. Problems occur on a daily basis in any project. They are not the same as failure. In fact, mistakes can help in avoiding failure - but only if they are used for learning. It is common wisdom that we learn more from failure than success. So is it not strange that everyone tends to overemphasise, and even exaggerate, success while downplaying problems and failure? All organisations and individuals generally want to portray themselves as being successful. The desire for good news is present at all levels of the system, from ministries and funding agencies, down to field reports. It stimulates the ongoing reporting of the "myth of success" (see Box 2-14), which means missing a key learning opportunity. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||