Section 2 - Using M&E to Manage Project Impact

2.1 An Overview of Using M&E to Manage Impact

Rural development projects aim to improve the lives of the rural poor. As manager of a project or part of it, do you always know what impact you are having and why? Learning about successes and failures through regular monitoring and critical reflection is fundamental for guiding your intervention towards achieving maximum impact. Monitoring and Evaluation (M&E) is the heart of "managing for impact" (see Box 2-1).

"Managing for impact" means you need to respond to changing circumstances and increased understanding by adapting the project so that it will be more likely to achieve its intended impacts. Such adaptation may entail small changes to activities or larger strategic revisions. Each project is being managed for impact under its own set of constraints. Keeping updated on the internal and external constraints will help you have realistic expectations of what can be adjusted and achieved.

To manage adaptively, project implementers and managers need to:

understand the project design;
gather and analyse relevant information to make good decisions;
facilitate learning with all key stakeholders;
negotiate required changes.

Managing for impact is only possible if you have reliable information about the progress of activities and their outcomes, the reasons for success and failure, and the context in which activities are taking place. This information is the output of your M&E procedures. Analysing this information with key stakeholders can support good decisions that improve the project. Only when information helps the project reduce poverty more effectively will an M&E system be worthwhile. Taking poverty seriously has implications for M&E, as RDRS (Rangpur Dinajpur Rural Service) in Bangladesh is aware. With the need to recruit more selectively from severely impoverished groups, it is moving to monitor individual households rather than credit groups. This will allow the organisation to improve its social accountability but also its services to the poor of northern Bangladesh.

Box 2-1. Defining monitoring and evaluation

Evaluation, in its broadest sense, simply means "to assess or judge the value or worth of something". In practice, this means that implementers need a questioning attitude for continual assessment. Evaluation events are often more periodic, and ask more fundamental questions about the overall progress and direction of a project. Self-evaluation processes combine well with external evaluations.

Monitoring helps continual self-evaluation by providing data to generate insights through formal and informal processes. Formal monitoring involves gathering data about chosen indicators and performance questions. Informal monitoring is about valuing and sharing impressions from chats with stakeholders and from observations in the field. Monitoring focuses on regular information-gathering and the frequent checking of short-term progress, with analysis about implications for the project.

There is no consensus about terminology in planning and M&E. This Guide does not make an absolute distinction between "monitoring" and "evaluation" because, in practice, the two processes overlap and are part of a systematic participatory learning process. For example, if regular monitoring reveals that things are not going as expected, you might find it necessary to undertake a more thorough, thematic evaluation to understand why and know what changes can be made.

2.1.1 The Four Elements of Managing for Impact

To know if you are managing for impact, check Figure 2-1 to see if you and other implementing partners are putting in place the four basic elements that will give you the information and insights you need.

1. Guiding the Project Strategy for Poverty Impact - understanding the goals and objectives of the project and then allocating the available resources and guiding relationships between stakeholders to maximise impacts.

2. Creating a Learning Environment - inspiring and helping those involved with the project to reflect critically on progress, to learn from mistakes and to generate ideas for making improvements.

3. Ensuring Effective Operations - planning, organising and checking staff inputs, equipment, partner contracts, financial resources, (bi-) annual work plans, and communications to implement activities effectively and efficiently.

4. Developing and Using the M&E System - designing and implementing information gathering and reflective learning processes to generate insights that help you to improve operations and strategic directions.

Figure 2-1. Key elements in managing for impact

Figure 2-1. Key elements in managing for impact

Table 2-1 illustrates how an organisation may or may not put these four ideas into practice. It can be used as a checklist by project managers to assess how well the project is doing in terms of managing for impact.

Table 2-1. Example of projects that do and that do not manage for impact

Elements	Project That Manages for Impact	Project That Does Not Manage for Impact
A focus on poverty	The implementing partners, including primary stakeholders, collect data on impact and meet regularly to consider if impacts can be seen and whether they meet expectations	The project leaves impact assessment to outside organisations and only focuses on monitoring the implementation of activities.
A learning environment	Project and partner staff and primary stakeholders regularly and openly discuss progress and problems. Problems and mistakes are the basis for learning how to work better. People feel safe enough to share their ideas and observations with higher authorities and peers.	Staff focus only on implementing activities, without asking if they are relevant to local poverty needs. Project/Partner organisation staff feel that their ideas and observations are not valued and do not speak up when they see things going wrong. Primary stakeholders are rarely asked their opinions.
Effective operations	Clear systems exist for tracking staff performance, equipment use, etc. People are clear about their responsibilities and deadlines, and the resources they need to do their work are available. The managers allocate sufficient budget and are building capacity where needed.	People are often unclear about what they should be doing, when and where. Vehicles and other resources are poorly coordinated and often not available on time. Much time is wasted in inactivity. No efforts are made to fill capacity gaps.
A supportive M&E system	The project manager has quick and easy access to information about project progress. Staff and partners generate information about outcomes and impact achievements. Information about project progress is shared regularly with all stakeholders in a visually appropriate way. The walls of the project office show up-to-date information, graphs, time charts and pictures about project progress. Studies are undertaken to explain any emerging problems. Project reports make interesting reading, and tell both the good and bad and how improvements will be made.	The project manager has quick and easy access to information about project progress. Staff and partners generate information about outcomes and impact achievements. Information about project progress is shared regularly with all stakeholders in a visually appropriate way. The walls of the project office show up-to-date information, graphs, time charts and pictures about project progress. Studies are undertaken to explain any emerging problems. Project reports make interesting reading, and tell both the good and bad and how improvements will be made. Few people know what the project has achieved to date. There is little evidence about whether all the activities of the project are leading anywhere. M&E is seen largely as an external reporting function. Project reports are uninteresting, are not analytical, and exaggerate successes while not mentioning problems. Little information is shared with project stakeholders.

2.1.2 Guiding the Strategy for Poverty Impact

A typical IFAD-supported project has about seven years during which to have an impact on poverty. At start-up, the project strategy is first revised. This is a critical opportunity for strategic guidance. After this, the project's energy tends to focus on putting capacities and procedures in place; while towards the end, efforts often focus on consolidating impact and phasing out the project presence. This leaves the project stakeholders with three or four years of prime time to take corrective action.

To know how best to mobilise project resources and partnerships for reducing poverty, the implementers need to understand the project strategy and redirect it when problems arise. Guiding the strategy is largely about asking the right questions - and getting answers - at the right moments (see Box 2-2). M&E processes are critical for making collaborative decisions about adjusting the project's direction.

Box 2-2. Five strategic M&E questions to manage for impact

Relevance - Is what we are doing now a good idea in terms of improving the situation at hand? Is it dealing with the priorities of the target groups? Why or why not?
Effectiveness - Have the plans (purposes, outputs and activities) been achieved? Is the intervention logic correct? Why or why not? Is what we are doing now the best way to maximise impact?
Efficiency - Are resources used in the best possible way? Why or why not? What could we do differently to improve implementation, thereby maximising impact, at an acceptable and sustainable cost?
Impact - To what extent has the project contributed towards poverty reduction (or other long-term goals)? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise?
Sustainability - Will there be continued positive impacts as a result of the project after the project funds run out in four or five years? Why or why not?

Every project has an overall logic (see Figure 2-2 and Box 2-3) that describes what is to be achieved, why and how. The project logic starts by describing the situation that a group of stakeholders wishes to improve. The vision about how this situation can be changed should be based on the problems and aspirations of stakeholders, and particularly of the rural poor. This vision justifies the project's existence. Inevitably there will be different perceptions of the key problems and what constitutes an improvement. This is why participatory approaches to planning are so important (see Sections 2.7 and 3.2 for more details on participatory planning).

Figure 2-2. The project strategy and its local context

Figure 2-2. The project strategy and its local context

Box 2-3. From activities - via outcomes - to impacts

Impacts are changes in the lives of the rural poor or, more specifically, improvements in their wellbeing. To reduce poverty, a project team plans concrete activities. But these do not lead directly to impacts. There are many steps en route and many other players involved who influence final impact, so activities need continual steering.

Most planning approaches put two steps between activities and impacts: outputs and outcomes. Outputs are the direct products or services delivered by the project. The outcomes are what happen after outputs are delivered. They are the first signs of impact. Outcomes usually involve behavioural change in people or organisations as a result of the project. For IFAD-supported projects, impacts are both the outcomes and what happens with the rural poor after the outcomes have taken place.

For example, a project output might be "research into and development of an improved farming system" in the project area. The outcome for this is "adoption of the improved farming system by intended primary stakeholders". Then the impact will be that this adoption has improved the financial and food security situation of those who have made the change.

After making sure you have a clear "objective hierarchy" and the project is underway, guiding the project strategy for poverty impact becomes a matter of steering - continually checking, questioning and correcting. Proof that you have been guiding the project strategy well is when, from year to year, all stakeholders can see an improvement in how activities are implemented, in relationships between stakeholders and in the quality of project outputs.

Focusing only on poverty reduction is a long-term goal. So it is not very useful for everyday guidance. Reflections on the project's direction in, for example, monthly or quarterly meetings will focus on activities, outputs and processes, but will also require adjusting assumptions that underpin the project. Each year, a close look at the overall poverty reduction goal is needed to manage for impact. This is when impact monitoring plays a central role.

2.1.3 Encouraging Learning through Critical Reflection and Participation

It is the people involved in a development intervention who will make it succeed or fail. Their participation in learning how to improve a project throughout its existence is fundamental. For project and partner staff, this means listening carefully and regularly to the views of different groups - including each other - about what is working and what is not, and hearing reasons for why problems exist and what needs to improve. Learning certainly requires more than only "listening". Opportunities need to be created for staff from the project and implementing partners and primary stakeholders to meet and analyse their experiences with the project. Section 8 offers ideas to encourage reflection and critical analysis.

A good M&E system provides and communicates data to help stakeholder groups analyse progress. M&E is nothing more - or less - than an open and critically reflective communication process to strengthen project partnership. Putting participatory learning at the centre of good project management requires both data on project activities and personal accounts of people's experiences. It requires regular reviews by project staff with primary stakeholders, supported by occasional inputs from outside specialists.

In many development interventions, people lose motivation if they are either not invited to participate or the conditions are not created for their meaningful participation. Participation in M&E is meant to provide opportunities for people with relevant views about the project to learn how to improve it.

Behind this simple statement lie two questions. First, who has a relevant view? As it is neither practicable nor necessary to include everyone in developing the M&E system, choices need to be made. Second, how can different people best be involved? Different stakeholders' involvement in key M&E tasks needs to be negotiated. Do they want to participate, and under what conditions, in:

designing the M&E system (questions to answer, indicators for data collection, choice of methods and frequency, etc.);
data gathering and synthesis;
analysis for decisions to improve actions;
communication and feedback of results?

A participatory learning approach to M&E needs to be supported by a participatory style of management. If you are going to involve many stakeholders successfully in M&E, then they need to feel that the managers encourage and reward critical reflection. Only then will everyone be able to learn from problems in order to make the project work better. If management has a primarily controlling and punishing function, then those involved will not want to risk innovation. This will greatly limit a project's ability to manage for impact.

2.1.4 Ensuring Effective Operations

Ensuring effective operations requires putting in place the practical and operational conditions for carrying out project activities efficiently. Operations are guided by the annual work plan and budget (AWPB) and by regular meetings with implementers and primary stakeholders.
A project needs detailed annual and half-yearly work plans and budgets for six areas:

1. Staffing: numbers, salaries, capacities, quality of performance;
2. Equipment, goods, office buildings: amounts and qualities of which materials per location, plus processes in place and resources allocated for maintenance;
3. Managing contracts: tracking and guiding the quality of subcontracted parts of implementation, including contracts with communities about their rights and responsibilities;
4. Finances: tracking expenditure in order to (re-) allocate resources as necessary, as well as producing financial audits required by law;
5. Work planning: producing and monitoring the monthly, half-yearly and annual work plans for individual staff members, implementing teams and the project as a whole;
6. Communications: production and dissemination schedule and responsibilities for all communiqu�s, presentations and publications for shared decision-making and accountability.

2.1.5 Steps in Developing the M&E System

Figure 2-1 shows that managing for impact is based on the M&E system. Developing an M&E system involves six steps (see Section 2.5):

1. establishing the purpose and scope;
2. identifying performance questions, information needs and indicators;
3. planning for information gathering and organising;
4. planning critical reflection processes and events;
5. planning for quality communication and reporting;
6. planning for the necessary conditions and capacities.

Each of these design steps needs to be dealt with twice. First, as part of the appraisal report, in which the design team describes a general M&E framework for each of these six elements. Second, during project start-up, the project staff need to transform the general M&E framework into a detailed operational M&E plan. The six steps are discussed in detail in Sections 4 to 8.

The outputs of the project M&E system will provide answers to the five questions that guide the project strategy. Looking at the questions in Box 2-2, operational management is more frequent and must focus on the questions of "effectiveness" and "efficiency". More strategic reflections, for example during annual reviews and supervision missions, will look at the questions of "relevance", "impact" and "sustainability".

2.2 Management and M&E

2.2.1 Key Opportunities to Manage for Impact

The four elements of managing for impact are the broad processes that need to happen in any project. But on a daily basis, project and partner staff and primary stakeholders carry out more specific management functions, each of which presents an opportunity to refocus on impacts.

Leading - Providing vision, strategic direction and inspiration
Planning - Setting and adjusting goals and objectives, and then deciding when to achieve them and what needs to be done, how and by whom, including resource allocation
Organising (internal and external) - Setting up the internal structures and processes for the project to operate, plus dealing with the political system in which the project is set and coordinating stakeholders' roles
Staffing - Employing, supervising, training and motivating those involved
Checking - Ensuring that planned actions have been carried out and resources have been allocated and used appropriately

The idea of managing for impact can be implemented quite simply in regular events, such as annual project reviews, quarterly and mid-year partner/staff meetings, and during supervision missions. These ideas are easy for existing projects to implement as part of their current processes and for new projects to plan into their operating procedures.

For example, projects increasingly hold annual reviews with primary stakeholders as part of their ongoing self-evaluation process. During such a review, staff, partners and local people will discuss the monitoring data on activities, outputs and outcomes. They will analyse them with respect to project goals to see how activities are - or are not - contributing to poverty reduction. They will also discuss the quality of the project implementation process and of relationships amongst stakeholders. This leads to formulating the next annual work plan and budget (AWPB) and adjusting M&E plans. This self-assessment and development of the AWPB form the basis of the annual progress report, but more strategic issues can also emerge from community-level discussions (see Box 2-4). So an annual review process links all four elements of managing for impact: impact, strategy, operations and M&E.

Box 2-4. Changing strategic direction in the Northwest Agricultural Services Project, Armenia

In the initial project design, a community development component was included with the intention of funding community activities and infrastructure. However, once the project began and discussions were held with communities, the project team realised that a more sustainable approach to the project would be to use this funding to stimulate small-scale community enterprise development. If these enterprises turned out to be successful, then the community would be able to afford to invest in their own infrastructure needs, such as water supply and schools. The project team felt that this approach of building greater economic self-reliance would have a more lasting impact. With the agreement of the primary stakeholders, project staff, government and IFAD, the project strategy was changed to make this possible. Now the project needs to check that their assumption that people will invest in their own infrastructure needs is, indeed, a correct one. If this is not happening, then they might need to rethink the project strategy again.

Quarterly and mid-year review and planning meetings could operate similarly to a participatory annual review, perhaps with fewer stakeholders and more discussion on the quality of implementation and relationships. Using monitoring data in discussions can challenge people's impressions and the assumptions and so trigger analysis of what is really happening. Quarterly and mid-year reports could then focus on the achievement of activities, analysis of key achievements and problems, and - most importantly - on agreeing how to improve implementation. Such regular and improvement-oriented self-assessments are proof of a healthy learning environment that focuses on achieving impact by organising and implementing operations effectively.

Supervision missions and mid-term reviews are also occasions when all four aspects of managing for impact come together. But a project cannot rely on these alone, as MTRs come too late in a project's life (see Box 2-5) and supervision missions are not always in enough depth or timed appropriately to influence impact achievement. When project implementers take responsibility for their own learning process, they can then take corrective action when it is needed and not when it is too late. Such action involves redressing mistakes, expanding good practices, responding to changes in the context by rethinking activities and processes, and taking up new opportunities.

Box 2-5. Managing for impact - don't wait for the mid-term review!

Many IFAD-supported projects only start paying attention to M&E at the mid-term review stage. Such a late start with M&E is obviously not desirable. For example, according to the 1997 supervision mission report of one project, monitoring at the government level was not very effective, with the qualitative impacts not being monitored. According to the loan agreement, the government should have established institutional arrangements to monitor and evaluate progress with project impact on primary stakeholders. However, although the project was initiated in 1995, these arrangements were only established by September 1997. Until that time, there was an apparent absence of a proper M&E system in the supervision reports and the memoranda on the project. The only source of impact evaluation was the mid-term status review undertaken by a local research centre.

2.2.2 Knowing Your Information Needs and Planning Learning Opportunities

Clarity about your information needs helps in structuring the M&E system and, in particular, knowing how to make the most of events such as half-yearly revisions, MTRs and annual participatory reviews. To check if your M&E system is providing the information you need, refer to the five questions in Box 2-2. If part of the picture is missing for you, then you need to adjust existing M&E processes.

Not only managers have information needs. Everyone involved in the project has specific tasks in the project and therefore specific information needs. A project M&E system must consider as many needs as possible. Only then will managing for impact become a participatory learning process. Section 5 discusses some ways to deal with different information needs. For this reason, all projects must have a range of learning events. For instance, the Tropisec project in Nicaragua seeks inputs from 16 different events and information sources (see Section 2.7.2).

To use information most effectively for managing for impact, think about the key moments during the project life when strategic decisions are made that enable you to move closer to a poverty reduction impact (see Figure 2-3). Information from M&E will be most useful if it feeds into these moments.

Thinking about these key moments as learning opportunities to manage for impact can reveal their value as strategic steering exercises, rather than as obligatory. Keep the level of discussion and type of decision appropriate to the event. For example, an annual review process is usually not the best moment to discuss how to organise the delivery of stationery supplies to village groups. Nor is a weekly staff meeting the appropriate place to agree on the new terms of contracts with partners.

Figure 2-3. The sequence of learning opportunities during the project cycle

2.2.3 Recognising and Tracking Operating Constraints in the Project Context

Understanding context is critical when it comes to assessing relevance of strategy and activities, anticipating operational problems and judging a project's contribution, and also when designing the M&E aspects of a project.

You can start to disentangle the project's contribution by analysing the evolution of project interventions alongside other concurrent external influences that affect primary stakeholders. Continual updating on context also allows you to adapt the project strategy and operations while en route. A systematic and regular assessment of operating conditions helps with anticipating some of the operational issues that could arise. For example, in Nepal, one project manager receives quarterly "political situation monitoring" reports to understand how the project may be affected. These reports include a brief description of that period's critical events - such as safety and insecurity, government actions, demonstrations and dialogues - and their implications. The information is collected via key informants, newspapers, radio and television. It is essential for deciding whether or not to suspend activities in an insecure area.

Because contexts vary across projects and during the lifetime of a project, constraints and coping strategies will also vary (see Box 2-6). It is important to adjust project strategy and implementation to the extent that circumstances allow. The first element of managing for impact - guiding the project strategy (see Section 2.3) - may be particularly helpful for dealing with changing contexts. Section 3, on adapting the initial project design, also provides useful ideas.

Box 2-6. Enabling and constraining contexts

Guatemala: At project start-up, political and socio-economic changes had led to a resizing of the state and to decentralisation of its functions. The project responded with a new strategy. It involved local actors (especially grassroots organisations) in a continual process of planning and evaluation, which facilitated the gradual decentralisation of project services. So an operating "constraint" was turned into an advantage and led to more participatory forms of monitoring and self-evaluation.
Ghana: In 1996, clients complained that they would only rarely see the intermediaries - the banks that provide credit. After this complaint, the project created a "cycle of training" so that there would be constant follow-up and counselling. This created a new challenge - ensuring sufficient staff at the banks. Although they stipulated three staff members per bank to follow up with the clients, in practice there was often only one because of the government's no-hire policy.

Those involved in designing a project also need to understand contextual issues. Design teams may unknowingly specify certain modes of operations and organisational relationships that cause significant problems for negotiating and implementing good M&E (see Section 4 for more details on designing M&E). Tables 2-2 to 2-4 list project and context features that affect project management and M&E. As a project manager or the person responsible for M&E, you might wish to keep these in mind when deciding what direction to take in developing an M&E system.

Table 2-2. Project features with methodological implications

	How many implementing partners are anticipated?		The more there are, the more time is needed to build capacity and negotiate for an M&E system that meets all information needs and is genuinely shared.

	What aspects of the local culture and government system may help or constrain participatory processes?		The existing local and organisational cultures may help or hinder more participatory M&E efforts.

	Who is the cooperating institution - and what does it say about or require from M&E?		Make sure that its and IFAD's understanding of M&E procedures and requirements do not contradict each other. Ask both to clarify what they want/need.

	How long is the timeframe for the project?		The longer, the more time you have for developing effective M&E with all stakeholders but also the greater the pressure to produce far-reaching impacts and the greater the need to adjust the project en route in reaction to changing contexts.

	How much of the project design is fixed?		Knowing the boundaries for adapting the project enables you to focus your critical-reflections effort on those areas where strategic change is possible.

	Have learning opportunities been designed into the project?		The less this is the case, the more effort is needed to put in place incentives to use M&E as a genuine learning tool to adjust the project strategy and operations

Table 2-3. Project features with communication implications

	How large and decentralised are project operations?		The larger and more decentralised, the more decentralised analysis of M&E data can be and also the more attention needs to be paid to setting up communications flows so that the managers stay informed of what is happening locally.

	How decentralised is the government in the project area?		If more decentralised, then theoretically there is more potential for stakeholder involvement in project M&E, with more options for downward accountability as a form of beneficiary-oriented transparency.

	How hierarchical is the governance tradition in the project country/area?		The form of governance may require or prohibit the involvement of various levels of bureaucracy in M&E and may bring with it more or less bureaucracy.

	How receptive are the local cultures to sharing problems and learning from mistakes?		Cultural-historical contexts will determine the extent to which certain types of information can be criticised and to which people learn from error and share problems.

	How many funding agencies and partners does project management have to report to - and what do they need?		The greater the diversity and number of funding agencies and implementing partners, the more care is needed to keep different reporting/accountability demands manageable.

Table 2-4. Project features with implications for the quality of information

	Is the project in an area of civil unrest?		In areas of conflict, the "start-stop" nature of projects will compromise continuity of implementation and thus of progress assessment, besides making the attribution of impact more difficult due to the large impact of external factors.

	What kind of operational style does the project have?		Different types of projects (more or less decentralised or open-ended) will need to consider different types of indicators to assess the quality of the processes.

	How decentralised and "good-governance" driven is the government in the project area?		The less transparency in a government sector, the more difficult it is to unearth and use highly critical data that emerge from the M&E system.

	What type of development goals does the project strive to achieve?		Depending on the project focus, the impact on poverty alleviation will be more or less direct and will therefore affect how to undertake impact assessment and demonstrate attribution clearly.

	Does the project focus on physical changes or on human capacities - or both?		More quantitative and physical change-oriented project goals versus capacity development and empowerment-focused goals will make other demands on assessing long-term impact in terms of improved lives.

	How expansive is the project?		The larger the geographic coverage of the project, the more conscious effort needs to be made to understand local people's views and to assess above the activity level.

2.3 Guiding the Project Strategy for Poverty Impact

Let's turn now to the first of the four elements of managing for impact in more detail - the project strategy and how to guide it.

The main output of the formulation phase is a draft strategy for the project. The strategy consists of an objective hierarchy and a description of the necessary implementation arrangements and resources needed. The strategy is the basis for the project appraisal report. For IFAD-supported projects, the project strategy is based on the logical framework approach (LFA) and project objectives are summarised as a hierarchy in a logframe matrix (see Section 3). But irrespective of how the project strategy and objectives are formulated, the ideas in this section remain relevant.

By necessity, any project strategy simplifies reality. It cannot possibly describe all details of a context and of the intended plan. This means that the documented strategy is a management tool that requires continual adjusting to reflect current contexts and changing needs.
It is fundamental that the project strategy be made as clear as possible at the onset for - and with - all those involved. In one IFAD-supported project, the staff did not have a clear understanding of what the project was about. The initial project design was poorly formulated and had been based on very limited stakeholder consultation. Since the document was so unclear, people were not motivated to investigate and respond to the problems they encountered. Staff were passively implementing whatever the document said, so were driven by activities rather than impact.

2.3.2 The Objective Hierarchy and Assumptions

The objective hierarchy is the spine of the project strategy (see Figure 2-4 and Box 2-7). The hierarchy describes how lower-level activities contribute to higher-level objectives, and how these, in turn, help achieve the overall project purpose(s) and goal.

Figure 2-4 shows an objective hierarchy with four levels and assumptions between all levels. Development organisations use many different names for the levels in the objective hierarchy and even different numbers of levels (see Section 3). But all levels can be considered objectives, as they are something that the project stakeholders want to achieve. This is why the term, objective hierarchy, is used (although the terms, intervention logic and narrative summary, are also used). The hierarchy is the first column of the logframe matrix and shows how the "means" lead to the "ends" of an intervention.

Figure 2-4. A project's objective hierarchy

Figure 2-4. A project's objective hierarchy

Box 2-7. Definitions for four levels of an objective hierarchy

Goal: The long-term objective, change of state or improved situation to which a development intervention, such as a project or project component, is intended to contribute. For IFAD-supported projects, the goal is some form of poverty reduction. The extent to which the project contributes towards the goal is the impact of the project.
Purpose: The overall objective of the project (or project component), in terms of overall observable changes in performance, behaviour or status of resources that the project (or project component) is responsible for achieving. Standard logframes use one project purpose while IFAD recognises that a complex project may have several purposes.
Outputs: The products, services or results that must be delivered by the project implementers for the project purpose(s) or project component purposes to be achieved.
Activities: The actions taken by project implementers, which are required to deliver the outputs by using inputs such as funds, technical assistance and other types of resources.

Getting the logic of the hierarchy clear with the implementing partners, including primary stakeholders, is essential - although this does not necessarily mean using the logframe terminology with them. Simply ensure that there is consensus about what is to be implemented in the short-, mid- and long-term and for what overall purpose and goal. If a hierarchy is not logical, then you may end up implementing many different fragmented activities that do not lead to a clear output. Poor logic can most certainly lead to project failure (see Box 2-8). Or you may be promising to deliver an impact for the rural poor that is totally impossible given your timeframe and budget. Finally, if you are not clear about what you intend to achieve by when, then you will find it very difficult to know precisely what you should be monitoring and evaluating.

Box 2-8. How project logic of the strategy affects project success¹

*A Successful Project*	Good theoretical [logical] model + Good implementation = Leads to project success
*Theoretical Failure of a Project*	Incorrect theoretical [logical] model + Good implementation = Leads to project failure
*Failure in Implementation*	Good theoretical [logical] model + Failure in implementation = Leads to project failure
*Absolute Failure*	Incorrect/Illogical theoretical model + Failure in implementation = Leads to project failure

Figure 2-4 also shows that a project strategy contains many assumptions. An assumption is any condition outside the direct control of the project that is important for the project to succeed. There are two types of assumptions: external factors and those relating to the internal cause-effect logic (see Box 2-9). You will need to make more assumptions as you move higher up the objective hierarchy, as the project is only one of often many stakeholders contributing to rural poverty reduction and so there are increasing issues outside the direct control of the project.

Assumptions need to be identified in initial project design, but this is often poorly done. The assumptions are an important tool for guiding the project strategy. Identifying them helps you know if the project strategy has a reasonable chance of success or is based on unlikely assumptions. Checking them regularly to see which ones are risky for the project, updating them based on better understanding from experiences, and identifying new ones is critical for guiding the project strategy. See Section 3 for ideas on how to work with assumptions as part of your M&E system.

Box 2-9. Working with assumptions

In Nepal, the Western Upland Development Programme has the following assumptions related to one of its desired outputs, "implementing a micro-finance programme, which includes training, and credit and savings":

market trends and fluctuations do not adversely affect the economic viability of on- and off-farm activities;
project staff are properly trained and motivated;
target groups are willing and able to participate effectively.

Most project designs include assumptions, such as these, that describe external factors needed for a project to succeed. For example, improving irrigation might be based on an assumption of continued water supply from a dam. But the dam could silt up due to poor environmental management or the government could decide to divert the water to other users for political reasons. By identifying those assumptions, project implementers can accept which ones are outside their direct control or see what they can do to reduce external risks.

But equally important - and more neglected - are assumptions about the internal cause-effect logic of the project. For example, an IFAD-supported project in Mali invests in building covered markets in isolated zones as a strategy for poverty reduction. This means that it assumes that the physical presence of these markets will increase household incomes. But who has access to these markets, will these markets be used, and is there produce to exchange in these markets? Will incomes actually increase for those who use the markets to trade? If there is increased household income, who controls how it is spent? And would it be spent on essentials? These are examples of internal logic assumptions that lie behind the simple statement of "marketplaces will reduce poverty in isolated areas".

Making assumptions explicit helps you to check where the objective hierarchy has weak spots and so tells you what to adjust. It is also an important focus for analysing your monitoring data. For example, in an agroforestry initiative in southeast Brazil, the participating farmers were monitoring their investment in on-farm experiments. They were shocked to see how much time they were still spending after several years and how little productivity had increased. This forced everyone to reject the original assumption that poor farmers would get sufficient returns on their labour after two years. The project component on agroforestry was revised with an improved assumption about a slower rate of return for farmers and thus with more realistic levels of anticipated impact.

2.3.3 Accountability and Adjusting the Project Strategy

Knowing how to adjust the project strategy requires knowing how flexible you can be. There are two ways in which projects are adapted.

Knowing how to adjust the project strategy requires clarity about what project management is accountable for achieving. You should adjust whatever is necessary to be convinced that you can deliver what you are accountable for - and no more or less.

A project's control over factors in the project environment that influence the achievement of objectives decreases with each level of the objective hierarchy (see Figure 2-5). At the level of "activities", project implementers have much control. External factors are unlikely to threaten the implementation of activities seriously. But at the level of the "goal", many factors beyond the direct control of implementers will influence impact. At this level, the project is one of many stakeholders contributing to the reduction of rural poverty. So a project's accountability at higher levels also decreases but never disappears entirely.

Figure 2-5. The limits of control and accountability in a project

Figure 2-5. The limits of control and accountability in a project

For example, a project might include training activities for farmers. The project can directly control the hiring of a training venue, the preparation of materials, the provision of a qualified trainer and the invitation of suitable participants. It has less control over whether potential participants will attend and considerably less control, if any, over whether the skills the participants learn will actually be used back in the workplace. While the training unit of a project can be held accountable for making the training relevant and accessible, it cannot be held accountable for whether the farmers have all the necessary conditions on-farm to implement the new skills they have learned.

Therefore, if you are concerned that the project purpose can only be achieved if certain adaptations are made - and you know that you are accountable for delivering the purpose-level outcomes - then you will need to revise the strategy accordingly (see Box 2-10). You may need to negotiate with funding agencies and supervising organisations for approval of suggested changes. The limits of accountability may need to be negotiated with funding agencies - critical if you are to keep the expectations of impact realistic.

Box 2-10. Revising a project strategy

An M&E study in Challa village, China, on the welfare impact of an IFAD-supported credit project highlighted the need for a more holistic approach to poverty reduction. Although credit had helped to increase farm-level productivity because of higher adoption of technological packages, this did not automatically lead to increased income and welfare. The study showed that other factors such as post-harvest losses, marketing and agro-processing hamper incomes. Some factors, such as off-farm income-generating activities, had been incorrectly omitted from the initial project design. So the project strategy needed to change to consider farmers' problems in a more integrated way. Local leaders, other primary stakeholders and experts were brought together so that farmers' problems, needs and priorities could be identified in their complexity, thus improving the strategy.

2.3.4 Guiding the Strategy

To guide the project strategy, you have several management tools.

1. The idea of negotiation and shared decision-making with implementing partners and primary stakeholders may seem so obvious as to be ignored. Building agreement of the need for and types of changes is the basis of strategic guidance, as this increases the chance that changes will be carried out. Negotiations about change will also be needed vis-�-vis the funding agencies and the government structures in which the project works. Not all changes will be possible but with good M&E data to explain why changes are desirable, your manoeuvring space might increase.

2. The objective hierarchy is a valuable focus for regular (semi-annual and annual) review and planning events during which you compare achievements with targets and try to understand why differences occur - which they inevitably will. Many projects focus on activities during progress reviews, but this is not enough to manage for impact.

What a project aims to achieve is the purpose and goal level, while outputs and activities describe how it thinks it can do this. A progress review needs to look both at the "how" and the "what" question. Looking only at the activities and outputs, you could conclude these are all going as planned. But you also need to ask, "Where is this leading?" to know if you are on track with the planned outcomes and impacts. This will help avoid wasting time and resources on pointless outputs and activities.

3. Assumptions can be reviewed regularly to check if they are still valid. Identify new assumptions that have emerged and delete those that are no longer relevant.

4. Based on your assessment of problems, successes and revised assumptions, check each level of the hierarchy for relevance and completeness. Add new activities or outputs and delete irrelevant ones in line with your assessment. Adjust targets as necessary. This may require negotiation with funding agencies, particularly at higher levels.

5. An M&E system can provide data for and organise reflection processes around points 1 to 4 above.

Box 2-11 shows how one project in Venezuela is assessing achievement of project component outcomes and linking them to an improved strategy.

Box 2-11. Self-evaluation for strategic assessment in Venezuela

PRODECOP in Venezuela analyses project component results in five steps.

1. Anticipated outcomes (as planned): assessing for each objective on a scale of 0 (not available) to 5 (excellent) if the objectives were achieved.

2. Unanticipated outcomes: describing the outcomes and assessing whether they made a high, medium or low contribution to the project component.

3. Factors affecting the outcomes: listing the factors and assessing if they were positive or hindering and why.

4. Possible impacts realised: for four levels (primary stakeholders and communities, PRODECOP staff, PRODECOP, other programmes/organisations) listing the possible social, economic, educational and other impacts that might be occurring now.

5. Actions needed to improve success with component objectives: listing all actions that the project will take on itself and all actions that are the suggested responsibility of others (indicating whose responsibility they should be).

The M&E data for this analysis comes from surveys and key informants. The five steps are followed for the three project components: (1) capacity-building for development and civil participation, (2) the financial services and rural finance system, and (3) monitoring and evaluation.

2.4 Creating a Learning Environment

2.4.1 What Is a Learning Environment?

The second element of managing for impact concerns the learning environment that needs to be created if people are to provide strategic and operational guidance by reflecting critically on what is happening.

How can you know if your project is actively learning? If you can clearly say "yes, this is happening here" to the following items, then you know that in terms of managing for impact your project is well on its way to having a culture of learning through critical reflection:

2.5 Ensuring Effective Operations

You might have a great strategy and a very open team that is constantly seeking new challenges. But if you have not organised staffing well, equipment is a mess, and finances are not well kept, then a project cannot have a good annual work plan and budget and is more unlikely to have optimal impact. "Ensuring effective operations" involves putting in place the practical and operational conditions for carrying out project activities efficiently. Operations are guided by the annual work plan and budget (AWPB). The project strategy is the basis for the AWPB. How you carry out the AWPB determines whether or not you are ensuring effective operations.

The topic of ensuring effective operations is not the prime focus of this guide. It is only discussed briefly here in terms of what is needed and how it relates to M&E. See Section 3 for more ideas on the AWPB.

2.5.1 Key Operational Areas

To be operational, a project needs to provide detailed annual and half-yearly plans and reports detailing activities and budget use for six areas.

1. Staffing. This relates to organising the appropriate number of staff needed and their salaries and ensuring their capacities are relevant and are updated. Also critical are processes to assess staff performance. This may include the productivity of staff and partners but should focus more on the quality of their work.

2. Equipment, goods, office buildings. Items include: vehicles, construction equipment and office equipment (including computers and software). This means ensuring that you have enough appropriate operating space at headquarters and outlying project areas and that processes and resources are in place for their maintenance.

3. Managing contracts. All projects subcontract parts of the work - from minor parts to substantial components. For example, some IFAD-supported projects in Latin America work entirely through subcontracting, with many implications for M&E (see Section 1).

4. Financial tracking and audits. All projects are aware of the importance of tracking expenditure and are legally obliged to produce financial audits.

5. Work planning. Monthly, half-yearly and annual work plans are needed for each staff member and key implementing partner, for project components and for the project as a whole.

6. Communications. A schedule is needed that details the types of communication and responsibilities over the next planning period. This includes internal communiqu�s and publications for sharing information with other stakeholders including funding agencies.

2.5.2 Information Needs for Effective Operations

For each of these operational areas, M&E is needed to ensure that resources, processes and quality are adequate. A look at the information tracking systems of most projects will often reveal plenty of data on vehicles, supplies, finances, staffing and so forth. In some projects, tracking such inputs can take up much time of the M&E unit. In FODESA, a project in Mali, all implementation - and the monitoring of implementation - is subcontracted. The M&E unit spends a considerable percentage of its time keeping track of the compliance of an enormous number of contracts.

While much information needs to be gathered and analysed for operational management, this level of monitoring tends to be more straightforward than for the project strategy. Table 2-5 lists the key areas of operation management, the main management tasks and the information needs (also see Section 5).

Operational Management Area	Key Management Tasks	Examples of Information Needs
Staff management	- Developing and monitoring staff work plans - Staff performance appraisal	- Quarterly, half-yearly, annual work plans for each staff member - Performance (quantity and quality) of each staff member

Plant, building and equipment management	- Purchasing and maintaining equipment - Allocating equipment - Finding and maintaining appropriate work space	- Asset register - Vehicle use - Equipment maintenance schedule, standards, responsibilities

Contract management	- Developing contracts - Monitoring delivery of contracts	- Database of all contractors and their contracts - Compliance with contracts (timeliness, quality)

Financial management	- Money allocation to activities and tasks - Monitoring expenditure according to budget - Revising budgets as needed	- General project financial management information - Annual audits

Work planning and activity tracking	- Annual, quarterly, weekly activity planning - Allocation of resources to activities - Checking progress on activities and responding to problems	- Detailed activity, sub-activity, task lists for achievement of outputs - Lists of required resources per activity - Activity and task progress

Communication management	- Keeping stakeholders updated on work and events - Meeting reporting requirements	- Calendar/Responsibilities for deadlines for reports - Database on who should receive what - Database on all documents produced

It is tempting to want to know each detail about operations. But try to limit information to what you "need to know" and avoid what is "nice to know". Also try to prevent all information from flowing up to senior management, as it will clog the decision-making process. Only ask for information to be shared if others really need to know it. Information that you "need to know" about operations should relate directly to the three basic questions for each operational area:

1. What has happened with the money used and the time that people have invested?
2. What is the overall performance of each of these areas - quality of output and quality of process?
3. Is it efficient enough or can we make improvements in how vehicles are used, staff performance, office supply procurement, etc?

Beneficiary contact monitoring can help gain insight into your operations. This requires regular contact with beneficiaries and asking their perceptions about project services and structures. From this you can determine how a project can better meet their needs and demands. You can gather data by maintaining records for each stakeholder in a project. But this is only feasible for clearly targeted stakeholders receiving a specific and measurable input, such as credit, and only for simple types of information. You can also establish a regular schedule of sample questionnaires and surveys. Questions can include: Do primary stakeholders know about services provided by the project? What proportion has used project services at least once? What problems do primary stakeholders feel are priorities to resolve? Finally, informal interviews can help you obtain more direct feedback from the field on success stories and problems. You will need to probe to find out not only what happens but also why. Annex D lists other methods you might find useful.

2.6 Setting Up and Using the M&E System

2.6.1 What Is the M&E System?

Looking back at Figure 2-1, you can see that the four functions of managing for impact require an operational M&E system. The M&E system is the set of planning, information gathering and synthesis, reflection and reporting processes, along with the necessary supporting conditions and capacities required for the outputs of M&E to make a valuable contribution to decision-making and learning. Key project stakeholders need to develop the different elements of the system together if they are all to use the outputs to improve implementation.

Setting up an M&E system involves six steps that need to be dealt with twice - generally at initial design and in detail at start-up:

1. Establishing the purpose and scope - Why do we need M&E and how comprehensive should our M&E system be?
2. Identifying performance questions, information needs and indicators - What do we need to know to monitor and evaluate the project in order to manage it well?
3. Planning information gathering and organising - How will the required information be gathered and organised?
4. Planning critical reflection processes and events - How will we make sense of the information gathered and use it to make improvements?
5. Planning for quality communication and reporting - What, how and to whom do we want to communicate in terms of our project activities and processes?
6. Planning for the necessary conditions and capacities - What is needed to ensure that the M&E system actually works?

These steps can be used when initiating a new project or when revising and expanding the M&E system of an existing project. Existing projects might have planned for some of these M&E elements but miss or undervalue others, so M&E does not perform optimally. This might seem like a lot to remember but in practice is often quite clear. Box 2-15 outlines one project example from India that links several elements into an M&E system.

Box 2-15. Overview of a professional and efficient M&E system in India

The M&E system for the Maharashtra Rural Credit Project consisted of three elements:

1. Input/Output monitoring: regularly to assess the progress of the project and its performance and to respond to any problems that may occur during the project;

2. Impact monitoring and ongoing evaluation: to judge the effect of the project on its primary stakeholders (information collected by the Village Development Councils (VDCs) and supplemented by M&E staff);

3. External evaluation and specific studies: to examine the long-term impact of the project and special issues arising during project implementation.

This M&E system succeeded because:

The overall organisation was sound, with an extensive network of field offices including high-quality M&E staff, particularly at the state level.
With the help of local experts, introductory M&E workshops were given to familiarise the implementing agencies with the monitoring requirements, followed by regular monthly meetings at the district level and periodic sharing workshops. These meetings have contributed to greater clarity and efficiency of the M&E system and the project impacts.
There was regular and focused feedback: the M&E system had a very systematic way of providing feedback to the participating agencies after analysis of information from the field.
In addition to the routine data-gathering, surveys at reasonable, valid intervals (as necessary) were carried out (for example, in-depth assessment of the status of the self-help groups.
The data analysis was systematic, professional and included detailed information, for example, on specific poverty indicators.
There was good involvement of primary stakeholders. A number of meetings were held at village level with relevant officials participating before the VDCs were formed, which could explain the adequate involvement of primary stakeholders in project processes.

2.6.2 From Data to Decisions

A critical attitude is needed to use data for decisions, as shown by this simple yet effective example (see also Box 2-16). The project officer of one IFAD-supported project in Bhadrachalam, India, reviewed the results of its school programmes and found them to be very poor in one zone. On closer examination, she found that the teachers in that zone were not adequately trained. An intensive training programme was introduced and the results showed a significant improvement. This person successfully used data and critical reflection to identify action.

Box 2-16. Small steps to use data for decisions

In Benin, project staff documented problems in the M&E reports. For example, the reports mentioned that certain local groups did not have enough meetings or checks in place to function well. So the project was able to support the groups to change this situation.
In India, an analysis of the monitoring information from a credit extension project revealed that the optimum size of a SHG (self-help group) for effective functioning is 15 women, so the project used this maximum number for all SHGs.
In another Indian project, managers eliminated the difficult agricultural extension messages after noticing a pattern from periodic surveys that certain types of simpler training messages improved agricultural productivity levels.

Most project or partner staff associate monitoring with "data" or many tables of numbers. To monitor well, they feel it is important to spend time perfecting indicators. Thus, many projects end up with long lists of numeric data that staff feel they need to collect even though much of these data rarely influence the direction the project is taking.

Indicators and other information are critical for learning. But on their own they will not provide the understanding that project and partner staff need to guide the project strategy and operations. Getting agreement on data's implications for action is essential to move from data to decisions. Any good M&E system will include clear plans and methods for analysis, communication and critical reflection with the relevant stakeholders. Sections 6 and 8 provide ideas for ongoing reflections and sharing of findings.

Documenting the decisions made after reflecting on M&E data can help to encourage their implementation (see Boxes 2-11 and 2-17). But documents also need to be used by managers to ensure follow-up. In one project, the M&E process clearly identified who was responsible for which changes. However, the reports were never used to check performance and so no one was ever held accountable on the basis of these reports.

Box 2-17. Format used as part of the results-oriented management design demanded by the Ugandan Ministry of Public Service (among others) to propose corrective action in periodic reports (download MSWord format | HTML Format)

Key Issue	Action	Responsible person	Timeframe	Evidence of Completion

2.6.3 Balancing Internal Learning with External Accountability

The focus on M&E to support internal project learning and management does not mean ignoring wider upward and downward accountability. Projects have important responsibilities to primary stakeholders, government agencies, funding agencies and society at large to account for their expenditures, activities, outcomes and impacts. In turn, the supervising and funding agencies must account to their governments and taxpayers for the investments made. This does not have to be complex (see Box 2-18).

Box 2-18. Stories of change for accountability (Davies 1998, see Further Reading)
CCDB is a Bangladeshi NGO with approximately 550 staff. Its main programme involves more than 46,000 people in 785 villages in 10 districts. Participants have access to three types of assistance: group-based savings and credit facilities used to meet the needs of individual households, grant assistance given to the same groups on a pro-rata basis and intended for community-level developments, and skills training mainly for livelihood purposes. The large scale and open-ended nature of these activities posed a major problem for the design of any system intended to monitor process and outcome.

In 1994 a participatory monitoring was tried that involved the deliberate abandonment of indicators. The new approach focused on "stories of change". Every three months, credit groups select and describe the single most significant change that have occurred in their groups in terms of at least three types of changes:

changes in people's lives;
changes in people's participation;
changes in the sustainability of people's institutions and their activities.

These stories are documented simply to enable verification. To avoid having a huge number of stories filtered to the funding agencies, each layer in the project system (extension agent, their supervisors and central office staff) select those they found "most significant" and justify their choice. CCDB has kept up this approach for several years now because, according to the director, it suits their needs to account to donors as well as to understand impacts on participants.

Having "enough" M&E information means being able to say, with confidence, what is happening and why. It also means having enough information for the different information needs of project stakeholders on a "need to know" rather than "nice to know" basis.

A good project M&E system designed to meet the information needs for internal impact- and learning-oriented management will produce the information required for external accountability without much additional effort. The problem is that most projects work the other way round: first they try to do everything needed for reporting and then they invest little time in sorting out their own learning processes.

How do you know if you are investing enough in learning? When M&E is integrated into management, you cannot isolate M&E and track how much time each of the project implementers should be and are spending on learning. However, as a rule of thumb, budgets for M&E-related activities lie between about 2 - 5% of the overall project budget. This includes learning and accountability M&E processes and outputs.

2.6.4 Keep It Simple

A good M&E is sophisticated in the information it generates, yet simple in its construction. It can be as basic as organising participatory AWPB review and planning sessions, alongside recording activities implemented and impact-tracking of a very limited set of key indicators. One project in Bangladesh clustered its six core M&E activities as follows: stakeholder workshops to design project M&E, implementation monitoring of activities, financial monitoring, participatory impact monitoring to strengthen local village organisations, external M&E to assess overall impacts per group, and technical monitoring for specific research questions that might arise.

Another simple construction to consider involves viewing the core M&E system as consisting of three elements, to which other elements can be added as experience grows and needs change:

1. tracking inputs and outputs of operations and activities;
2. organising quarterly discussions with key stakeholders on progress and problems with implementation;
3. annual reflections on core impact questions (see Box 2-19) as the basis for the AWPB.

Note that these are just ideas - and not models to follow blindly.

Box 2-19. Tracking impact with five key questions - the case of Oxfam

Oxfam UK, a large international NGO, encourages the use of five questions to guide reflections:

What key changes have happened in people's lives? How sustainable are they?
What changes in equity and, in particular, gender equity are occurring at different levels?
What has changed in the policies and practices, and ideas and beliefs, of those institutions that affect the lives of people living in poverty?
What has changed in the degree to which people living in poverty have participated in and taken control over programmes, processes and decisions that affect their lives?
How cost-effective have Oxfam and others been in promoting the above changes?

You can answer these questions with different degrees of precision, with the input of different stakeholders and by using a wide range of methods (see Annex D), so you still need to make decisions about how elaborate your M&E system will be. Nevertheless, working with core questions helps to structure the often overly ambitious plans for monitoring large datasets with which projects start (also see Section 5). Conditions for this to be valid - as with all M&E - are that some records be kept of answers, compared from time to time, shared and acted on.

2.7 The Basics of Participatory M&E

2.7.1 Knowing What Participation in M&E Means

For many project staff, participatory M&E is about "getting the community involved" - somehow, sometime, somewhere. For most projects, participation in M&E is another way of saying "let's gather information from local people, using some questionnaires and diagramming methods". In one project, for example, local people are only consulted when M&E staff are collecting data and only approached again when problems arise.

But there is more to participatory M&E than simply changing a few of your data-gathering methods. The NWFP project in Pakistan, funded by the World Bank, took on a more participatory monitoring effort in which communities controlled the quality of sub-projects (see Further Reading). Dropouts of community-based organisations fell from 37% in Phase 1 to none in Phase 3. Costs have been reduced by up to 40% and works are often of better quality than those carried out through government contracting. If participation is to lead to sustained efforts and empowerment, then a common understanding and shared decision-making are needed. This implies seeing joint M&E as part of good governance. An example of participatory M&E based on joint learning and shared decision-making is described in Box 2-20.

Box 2-20. District learning from Chella Village, Tanzania

Tanzanian district officials were asked a challenging question by a local project's water users' association (WUA) during an annual project review meeting: "You say that the project is ours and it is in our [association's] action plan. Why don't you tell us how much cement is needed, for example and what it costs? Why don't you trust us with the cement so that we can keep it ourselves and supply it to the contractor as needed…so we can be responsible?"

The project coordinator was then asked for his consent - which he gave - to let the association supervise the use of the cement. So the association took over the responsibility of overseeing the distribution and use of the cement according to the bills of quantity and the requirements of the contractor. At the end of the rehabilitation process, the association managed to save seven bags of cement due to its "good supervision." The association members pointed out that if the project staff had been supervising the entire contract, there probably would not have been a surplus. The association then proposed selling the excess cement and putting the money gained into the operation and maintenance fund. As a result of an annual review meeting discussion and further testing with a pilot group, the project has changed its contracting procedures to give more construction supervision responsibility to the WUA.

Participatory M&E is not just a matter of using participatory techniques for information gathering in a conventional monitoring and evaluation setting or of organising a single workshop to identify local indicators. It is about radically rethinking who undertakes and carries out the process and who learns or benefits from the findings. One way of thinking about levels of participation in M&E is suggested by Feuerstein (see Further Reading), an evaluation specialist:

Let us apply some of this thinking to an increasingly common situation, the task of setting up a participatory process for the annual project review. You, as manager or M&E officer, need to make budget and staffing decisions about this important learning moment for the project. You have three options:

1. hire a consultant for a couple of months per year to facilitate key stakeholders for a short and focused input to get the job done;
2. hire a consultant for up to six months during the first two years of a project to design a process with project staff for this and facilitate parts of it, and so build staff capacity;
3. sit down with project partners and community members at start-up and develop a joint learning system together for all future annual project reviews.

Although this last option will take longer and require some compromises, you are investing in creating local capacities and ownership. Note that this does mean consciously planning capacity-building activities and realistic timing. Such participation can start as the project starts implementation but seeking wide stakeholder involvement earlier on is better (see Box 2-21).

Box 2-21. Stakeholder participation at project design helps M&E later on

A manager involved in a biodiversity project in Ghana found that participatory decision-making in the design phase greatly improved the quality of subsequent M&E activities. During the design phase process, the team worked with communities to develop project implementation and impact indicators to be included in the project document. She said that by involving community members earlier in the process (in this case in the development of indicators), they understand the justification and strategy of the project better and are more active in implementing M&E.
By contrast, at the start-up of a different project on income-generation, the responsible government departments held a workshop in order to detail the project logframe. Workshop participants (project managers and some potential NGO partners) looked at the operational aspects of the suggested impact indicators. All the project managers indicated their information needs and expectations and were asked to produce a multi-year operational plan. However, the final selection of NGO partners occurred at a later stage. Of those selected, only two had attended the initial workshop. The remaining 12 had not been involved in reviewing the M&E system. This caused problems later in terms of understanding what was needed for the project's M&E.

An inspiring example of how M&E can be participatory, accountable and integrated with planning comes from Colombia (see Box 2-22). The example may seem outside your options, but parts of it may certainly be relevant to your situation. As a manager or M&E officer, you do not have to stop everything you have built up so far and start from zero. Start with a problem you are facing today and use the practical ideas in Sections 4 to 8 of this Guide, to slowly improve how you learn together.

Box 2-22. M&E that strengthens local governance in Colombia

In Colombia, ACIN, an association of indigenous people covering 13 communities, is monitoring and evaluating its own multi-sectoral regional development plan. This involves three cycles: (1) a three-year M&E evaluation cycle of the local development plan supported by (2) an annual evaluation cycle of the small projects that make up the plan and (3) a monthly monitoring cycle of project activities.

The three-year M&E cycle of the overall plan takes place during four assembly meetings. Each assembly is a community meeting of about 300 to 600 people, held over three days. Participants include men and women, adults and youth, children, leaders and government officials, and others who collectively take part in decision-making. Representatives from each social group and each of the six sectors of activities attend the four assemblies to ensure continuity.

The community defines or revisits its vision of the future and establishes development criteria by comparing present and past situations. For example, the present situation is analysed by reproducing data from geographic information systems and small models of the local area. The first assembly is also used to revisit the existing local development plan, identifying the achievements, strengths and weaknesses of each sector based on the goals they had set out to achieve for the next three years.
Expected results are identified by sector, based on the vision and development priorities defined in the first assembly. The expected results are prioritised and grouped into categories, and indicators are selected for each. After the second assembly, a workshop is held by the planning council who reviews the indicators formulated by the assembly and converts them into questions. These questions are then used to develop baseline surveys. The surveys are answered collectively in large-group meetings held in each community. The information is summarised in tables that show expected results, the corresponding indicators, monitored information and achievements attained.
The surveyed information is presented at the third assembly. The indicators are compared against the targets set, and extra targets are identified for the next three years. Participants analyse the information in working committees per sector. The findings of the third assembly are discussed at the community level to ensure that the goals and findings are verified and receive broad-based support from the communities.
In the fourth assembly, the local development plan, goals and activities are reviewed and adjusted based on the evaluation findings and analyses. The assembly, looking at what helped or hindered goal achievement, prioritises goals and identifies strategies to reach prioritised goals. Local development projects are formulated with the help of the council and sectoral community representatives. These projects are then presented to the assembly for approval.

2.7.2 Critical Decisions When Starting with Participatory M&E

You may be starting a new project and have an opportunity to make it participatory from the start. Or you might be planning to make the existing M&E system more participatory. In either situation, four decisions need to be made to develop a version of participatory M&E that suits your situation.

1. Be clear about different people's motivations for getting involved in M&E and do not force them together if they do not fit. Simply provide support so the different systems work and support each other. Many people think that making M&E participatory means that everyone's information needs can be met. But sometimes these information needs are so different that deciding on separate and complementary M&E systems is better than trying to squeeze everything out of one set of indicators or one set of discussions.

Table 2-6 shows one set of motivations from three stakeholder groups involved in an agricultural project in Brazil. After trying to work out one system for everyone, it was clear to the NGO that the information needs and unit of analyses were so different that different M&E processes were needed for each group.

In Nicaragua, the Tropisec project developed different M&E events and reporting mechanisms for different levels and stakeholder groups. For example, the family/grassroots level M&E centred on monthly meetings, biannual planning and evaluation sessions, and baseline surveys. Implementing agencies submit trimester results-oriented reports, lessons learned, local government focal meetings and independent evaluations of sub-contracted components. Finally, project management has similar M&E events to the implementing agencies and also organises, for example, monthly monitoring meetings and impact-evaluation case studies.

Table 2-6. Motivation for agricultural monitoring in Brazil

Participants	Why They Wanted to Monitor and Evaluate the Joint Agricultural Work
Farmers	- to improve management of their own farming enterprise - to activate the interest of other farmers not yet involved in sustainable agriculture

Rural Workers Union	- to evaluate impacts with more certainty and avoid unsubstantiated opinions - to prove to other farmers that sustainable agriculture practices can also benefit them and are worthy of support by organisations - to be able to evaluate better non-agricultural aspects of the union's work - to help with planning, knowing what works and what does not

Local NGO	-to evaluate the local impact of innovations on ecological, social and economic well-being -to report to funding agencies the extent to which efforts are meeting intended objectives - to help plan and prioritise activities - to have proof for advocacy purposes at the regional, state and perhaps national level - to enhance the capacity of farmers and unions for autonomous planning and implementation of sustainable agriculture activities - to strengthen interaction between recently formed farmer experimentation groups

2. Negotiate and agree on "how much" participation for whom. Assessing how much participation for which groups depends largely on the purpose of participatory M&E.

If the purpose is setting up locally sustainable processes of monitoring and evaluating (for example, soil fertility), then local farmers and extension staff will need to be involved in the entire process: methodology design, information collection, information collation/calculation, analysis of findings, dissemination of findings. If the emphasis is internal project learning about local soil fertility management, participants can be limited initially to project staff but farmers' assessments of local indicators will be essential. If it is about improving project accountability, then perhaps it is a case of conventional M&E, using participatory methods to find the information and analyse it with primary stakeholders.

Assessing the need for participation of the possible stakeholder groups (community members, farmer groups, community leaders, government agency or NGO staff, etc.) can be guided by asking the following questions.