A Guide for Project M&E - Managing for Impact in Rural Development - Section 2 - Using M&E to Manage Impact
HOME
Introduction
M&E and Impact
Design, Planning and M&E
System Set-up
What to Monitor
Information Management
Capacities and Conditions
Critical Reflection
Glossary
Logframe Sample
M&E Matrix Sample
M&E Methods
Sample TORs


 

 

 

 

2.1 An Overview of Using M&E to Manage Impact

Rural development projects aim to improve the lives of the rural poor. As manager of a project or part of it, do you always know what impact you are having and why? Learning about successes and failures through regular monitoring and critical reflection is fundamental for guiding your intervention towards achieving maximum impact. Monitoring and Evaluation (M&E) is the heart of "managing for impact" (see Box 2-1).

"Managing for impact" means you need to respond to changing circumstances and increased understanding by adapting the project so that it will be more likely to achieve its intended impacts. Such adaptation may entail small changes to activities or larger strategic revisions. Each project is being managed for impact under its own set of constraints. Keeping updated on the internal and external constraints will help you have realistic expectations of what can be adjusted and achieved.

To manage adaptively, project implementers and managers need to:

  • understand the project design;
  • gather and analyse relevant information to make good decisions;
  • facilitate learning with all key stakeholders;
  • negotiate required changes.

Managing for impact is only possible if you have reliable information about the progress of activities and their outcomes, the reasons for success and failure, and the context in which activities are taking place. This information is the output of your M&E procedures. Analysing this information with key stakeholders can support good decisions that improve the project. Only when information helps the project reduce poverty more effectively will an M&E system be worthwhile. Taking poverty seriously has implications for M&E, as RDRS (Rangpur Dinajpur Rural Service) in Bangladesh is aware. With the need to recruit more selectively from severely impoverished groups, it is moving to monitor individual households rather than credit groups. This will allow the organisation to improve its social accountability but also its services to the poor of northern Bangladesh.

 

Box 2-1. Defining monitoring and evaluation

Evaluation, in its broadest sense, simply means "to assess or judge the value or worth of something". In practice, this means that implementers need a questioning attitude for continual assessment. Evaluation events are often more periodic, and ask more fundamental questions about the overall progress and direction of a project. Self-evaluation processes combine well with external evaluations.

Monitoring helps continual self-evaluation by providing data to generate insights through formal and informal processes. Formal monitoring involves gathering data about chosen indicators and performance questions. Informal monitoring is about valuing and sharing impressions from chats with stakeholders and from observations in the field. Monitoring focuses on regular information-gathering and the frequent checking of short-term progress, with analysis about implications for the project.

There is no consensus about terminology in planning and M&E. This Guide does not make an absolute distinction between "monitoring" and "evaluation" because, in practice, the two processes overlap and are part of a systematic participatory learning process. For example, if regular monitoring reveals that things are not going as expected, you might find it necessary to undertake a more thorough, thematic evaluation to understand why and know what changes can be made.

 

2.1.1 The Four Elements of Managing for Impact

To know if you are managing for impact, check Figure 2-1 to see if you and other implementing partners are putting in place the four basic elements that will give you the information and insights you need.

1. Guiding the Project Strategy for Poverty Impact - understanding the goals and objectives of the project and then allocating the available resources and guiding relationships between stakeholders to maximise impacts.

2. Creating a Learning Environment - inspiring and helping those involved with the project to reflect critically on progress, to learn from mistakes and to generate ideas for making improvements.

3. Ensuring Effective Operations - planning, organising and checking staff inputs, equipment, partner contracts, financial resources, (bi-) annual work plans, and communications to implement activities effectively and efficiently.

4. Developing and Using the M&E System - designing and implementing information gathering and reflective learning processes to generate insights that help you to improve operations and strategic directions.

Figure 2-1. Key elements in managing for impact

Figure 2-1. Key elements in managing for impact

Table 2-1 illustrates how an organisation may or may not put these four ideas into practice. It can be used as a checklist by project managers to assess how well the project is doing in terms of managing for impact.

Table 2-1. Example of projects that do and that do not manage for impact

Elements

Project That Manages for Impact

Project That Does Not Manage for Impact

A focus on poverty

The implementing partners, including primary stakeholders, collect data on impact and meet regularly to consider if impacts can be seen and whether they meet expectations

The project leaves impact assessment to outside organisations and only focuses on monitoring the implementation of activities.

A learning environment

Project and partner staff and primary stakeholders regularly and openly discuss progress and problems. Problems and mistakes are the basis for learning how to work better. People feel safe enough to share their ideas and observations with higher authorities and peers.

Staff focus only on implementing activities, without asking if they are relevant to local poverty needs. Project/Partner organisation staff feel that their ideas and observations are not valued and do not speak up when they see things going wrong. Primary stakeholders are rarely asked their opinions.

Effective operations

Clear systems exist for tracking staff performance, equipment use, etc. People are clear about their responsibilities and deadlines, and the resources they need to do their work are available. The managers allocate sufficient budget and are building capacity where needed.

People are often unclear about what they should be doing, when and where. Vehicles and other resources are poorly coordinated and often not available on time. Much time is wasted in inactivity. No efforts are made to fill capacity gaps.

A supportive M&E system

The project manager has quick and easy access to information about project progress. Staff and partners generate information about outcomes and impact achievements. Information about project progress is shared regularly with all stakeholders in a visually appropriate way. The walls of the project office show up-to-date information, graphs, time charts and pictures about project progress. Studies are undertaken to explain any emerging problems. Project reports make interesting reading, and tell both the good and bad and how improvements will be made.

The project manager has quick and easy access to information about project progress. Staff and partners generate information about outcomes and impact achievements. Information about project progress is shared regularly with all stakeholders in a visually appropriate way. The walls of the project office show up-to-date information, graphs, time charts and pictures about project progress. Studies are undertaken to explain any emerging problems. Project reports make interesting reading, and tell both the good and bad and how improvements will be made. Few people know what the project has achieved to date. There is little evidence about whether all the activities of the project are leading anywhere. M&E is seen largely as an external reporting function. Project reports are uninteresting, are not analytical, and exaggerate successes while not mentioning problems. Little information is shared with project stakeholders.

2.1.2 Guiding the Strategy for Poverty Impact

A typical IFAD-supported project has about seven years during which to have an impact on poverty. At start-up, the project strategy is first revised. This is a critical opportunity for strategic guidance. After this, the project's energy tends to focus on putting capacities and procedures in place; while towards the end, efforts often focus on consolidating impact and phasing out the project presence. This leaves the project stakeholders with three or four years of prime time to take corrective action.

To know how best to mobilise project resources and partnerships for reducing poverty, the implementers need to understand the project strategy and redirect it when problems arise. Guiding the strategy is largely about asking the right questions - and getting answers - at the right moments (see Box 2-2). M&E processes are critical for making collaborative decisions about adjusting the project's direction.

 

Box 2-2. Five strategic M&E questions to manage for impact

  • Relevance - Is what we are doing now a good idea in terms of improving the situation at hand? Is it dealing with the priorities of the target groups? Why or why not?
  • Effectiveness - Have the plans (purposes, outputs and activities) been achieved? Is the intervention logic correct? Why or why not? Is what we are doing now the best way to maximise impact?
  • Efficiency - Are resources used in the best possible way? Why or why not? What could we do differently to improve implementation, thereby maximising impact, at an acceptable and sustainable cost?
  • Impact - To what extent has the project contributed towards poverty reduction (or other long-term goals)? Why or why not? What unanticipated positive or negative consequences did the project have? Why did they arise?
  • Sustainability - Will there be continued positive impacts as a result of the project after the project funds run out in four or five years? Why or why not?
 

Every project has an overall logic (see Figure 2-2 and Box 2-3) that describes what is to be achieved, why and how. The project logic starts by describing the situation that a group of stakeholders wishes to improve. The vision about how this situation can be changed should be based on the problems and aspirations of stakeholders, and particularly of the rural poor. This vision justifies the project's existence. Inevitably there will be different perceptions of the key problems and what constitutes an improvement. This is why participatory approaches to planning are so important (see Sections 2.7 and 3.2 for more details on participatory planning).


Figure 2-2. The project strategy and its local context

Figure 2-2. The project strategy and its local context

 

Box 2-3. From activities - via outcomes - to impacts

Impacts are changes in the lives of the rural poor or, more specifically, improvements in their wellbeing. To reduce poverty, a project team plans concrete activities. But these do not lead directly to impacts. There are many steps en route and many other players involved who influence final impact, so activities need continual steering.

Most planning approaches put two steps between activities and impacts: outputs and outcomes. Outputs are the direct products or services delivered by the project. The outcomes are what happen after outputs are delivered. They are the first signs of impact. Outcomes usually involve behavioural change in people or organisations as a result of the project. For IFAD-supported projects, impacts are both the outcomes and what happens with the rural poor after the outcomes have taken place.

For example, a project output might be "research into and development of an improved farming system" in the project area. The outcome for this is "adoption of the improved farming system by intended primary stakeholders". Then the impact will be that this adoption has improved the financial and food security situation of those who have made the change.

 

After making sure you have a clear "objective hierarchy" and the project is underway, guiding the project strategy for poverty impact becomes a matter of steering - continually checking, questioning and correcting. Proof that you have been guiding the project strategy well is when, from year to year, all stakeholders can see an improvement in how activities are implemented, in relationships between stakeholders and in the quality of project outputs.

Focusing only on poverty reduction is a long-term goal. So it is not very useful for everyday guidance. Reflections on the project's direction in, for example, monthly or quarterly meetings will focus on activities, outputs and processes, but will also require adjusting assumptions that underpin the project. Each year, a close look at the overall poverty reduction goal is needed to manage for impact. This is when impact monitoring plays a central role.

2.1.3 Encouraging Learning through Critical Reflection and Participation

It is the people involved in a development intervention who will make it succeed or fail. Their participation in learning how to improve a project throughout its existence is fundamental. For project and partner staff, this means listening carefully and regularly to the views of different groups - including each other - about what is working and what is not, and hearing reasons for why problems exist and what needs to improve. Learning certainly requires more than only "listening". Opportunities need to be created for staff from the project and implementing partners and primary stakeholders to meet and analyse their experiences with the project. Section 8 offers ideas to encourage reflection and critical analysis.

A good M&E system provides and communicates data to help stakeholder groups analyse progress. M&E is nothing more - or less - than an open and critically reflective communication process to strengthen project partnership. Putting participatory learning at the centre of good project management requires both data on project activities and personal accounts of people's experiences. It requires regular reviews by project staff with primary stakeholders, supported by occasional inputs from outside specialists.

In many development interventions, people lose motivation if they are either not invited to participate or the conditions are not created for their meaningful participation. Participation in M&E is meant to provide opportunities for people with relevant views about the project to learn how to improve it.

Behind this simple statement lie two questions. First, who has a relevant view? As it is neither practicable nor necessary to include everyone in developing the M&E system, choices need to be made. Second, how can different people best be involved? Different stakeholders' involvement in key M&E tasks needs to be negotiated. Do they want to participate, and under what conditions, in:

  • designing the M&E system (questions to answer, indicators for data collection, choice of methods and frequency, etc.);
  • data gathering and synthesis;
  • analysis for decisions to improve actions;
  • communication and feedback of results?

A participatory learning approach to M&E needs to be supported by a participatory style of management. If you are going to involve many stakeholders successfully in M&E, then they need to feel that the managers encourage and reward critical reflection. Only then will everyone be able to learn from problems in order to make the project work better. If management has a primarily controlling and punishing function, then those involved will not want to risk innovation. This will greatly limit a project's ability to manage for impact.

2.1.4 Ensuring Effective Operations

Ensuring effective operations requires putting in place the practical and operational conditions for carrying out project activities efficiently. Operations are guided by the annual work plan and budget (AWPB) and by regular meetings with implementers and primary stakeholders.
A project needs detailed annual and half-yearly work plans and budgets for six areas:

1. Staffing: numbers, salaries, capacities, quality of performance;
2. Equipment, goods, office buildings: amounts and qualities of which materials per location, plus processes in place and resources allocated for maintenance;
3. Managing contracts: tracking and guiding the quality of subcontracted parts of implementation, including contracts with communities about their rights and responsibilities;
4. Finances: tracking expenditure in order to (re-) allocate resources as necessary, as well as producing financial audits required by law;
5. Work planning: producing and monitoring the monthly, half-yearly and annual work plans for individual staff members, implementing teams and the project as a whole;
6. Communications: production and dissemination schedule and responsibilities for all communiqués, presentations and publications for shared decision-making and accountability.

2.1.5 Steps in Developing the M&E System

Figure 2-1 shows that managing for impact is based on the M&E system. Developing an M&E system involves six steps (see Section 2.5):

1. establishing the purpose and scope;
2. identifying performance questions, information needs and indicators;
3. planning for information gathering and organising;
4. planning critical reflection processes and events;
5. planning for quality communication and reporting;
6. planning for the necessary conditions and capacities.

Each of these design steps needs to be dealt with twice. First, as part of the appraisal report, in which the design team describes a general M&E framework for each of these six elements. Second, during project start-up, the project staff need to transform the general M&E framework into a detailed operational M&E plan. The six steps are discussed in detail in Sections 4 to 8.

The outputs of the project M&E system will provide answers to the five questions that guide the project strategy. Looking at the questions in Box 2-2, operational management is more frequent and must focus on the questions of "effectiveness" and "efficiency". More strategic reflections, for example during annual reviews and supervision missions, will look at the questions of "relevance", "impact" and "sustainability".

Back to Top

2.2 Management and M&E

2.2.1 Key Opportunities to Manage for Impact

The four elements of managing for impact are the broad processes that need to happen in any project. But on a daily basis, project and partner staff and primary stakeholders carry out more specific management functions, each of which presents an opportunity to refocus on impacts.

  • Leading - Providing vision, strategic direction and inspiration
  • Planning - Setting and adjusting goals and objectives, and then deciding when to achieve them and what needs to be done, how and by whom, including resource allocation
  • Organising (internal and external) - Setting up the internal structures and processes for the project to operate, plus dealing with the political system in which the project is set and coordinating stakeholders' roles
  • Staffing - Employing, supervising, training and motivating those involved
  • Checking - Ensuring that planned actions have been carried out and resources have been allocated and used appropriately

The idea of managing for impact can be implemented quite simply in regular events, such as annual project reviews, quarterly and mid-year partner/staff meetings, and during supervision missions. These ideas are easy for existing projects to implement as part of their current processes and for new projects to plan into their operating procedures.

For example, projects increasingly hold annual reviews with primary stakeholders as part of their ongoing self-evaluation process. During such a review, staff, partners and local people will discuss the monitoring data on activities, outputs and outcomes. They will analyse them with respect to project goals to see how activities are - or are not - contributing to poverty reduction. They will also discuss the quality of the project implementation process and of relationships amongst stakeholders. This leads to formulating the next annual work plan and budget (AWPB) and adjusting M&E plans. This self-assessment and development of the AWPB form the basis of the annual progress report, but more strategic issues can also emerge from community-level discussions (see Box 2-4). So an annual review process links all four elements of managing for impact: impact, strategy, operations and M&E.

 

Box 2-4. Changing strategic direction in the Northwest Agricultural Services Project, Armenia

In the initial project design, a community development component was included with the intention of funding community activities and infrastructure. However, once the project began and discussions were held with communities, the project team realised that a more sustainable approach to the project would be to use this funding to stimulate small-scale community enterprise development. If these enterprises turned out to be successful, then the community would be able to afford to invest in their own infrastructure needs, such as water supply and schools. The project team felt that this approach of building greater economic self-reliance would have a more lasting impact. With the agreement of the primary stakeholders, project staff, government and IFAD, the project strategy was changed to make this possible. Now the project needs to check that their assumption that people will invest in their own infrastructure needs is, indeed, a correct one. If this is not happening, then they might need to rethink the project strategy again.

 

Quarterly and mid-year review and planning meetings could operate similarly to a participatory annual review, perhaps with fewer stakeholders and more discussion on the quality of implementation and relationships. Using monitoring data in discussions can challenge people's impressions and the assumptions and so trigger analysis of what is really happening. Quarterly and mid-year reports could then focus on the achievement of activities, analysis of key achievements and problems, and - most importantly - on agreeing how to improve implementation. Such regular and improvement-oriented self-assessments are proof of a healthy learning environment that focuses on achieving impact by organising and implementing operations effectively.

Supervision missions and mid-term reviews are also occasions when all four aspects of managing for impact come together. But a project cannot rely on these alone, as MTRs come too late in a project's life (see Box 2-5) and supervision missions are not always in enough depth or timed appropriately to influence impact achievement. When project implementers take responsibility for their own learning process, they can then take corrective action when it is needed and not when it is too late. Such action involves redressing mistakes, expanding good practices, responding to changes in the context by rethinking activities and processes, and taking up new opportunities.

 

Box 2-5. Managing for impact - don't wait for the mid-term review!

Many IFAD-supported projects only start paying attention to M&E at the mid-term review stage. Such a late start with M&E is obviously not desirable. For example, according to the 1997 supervision mission report of one project, monitoring at the government level was not very effective, with the qualitative impacts not being monitored. According to the loan agreement, the government should have established institutional arrangements to monitor and evaluate progress with project impact on primary stakeholders. However, although the project was initiated in 1995, these arrangements were only established by September 1997. Until that time, there was an apparent absence of a proper M&E system in the supervision reports and the memoranda on the project. The only source of impact evaluation was the mid-term status review undertaken by a local research centre.

 

2.2.2 Knowing Your Information Needs and Planning Learning Opportunities

Clarity about your information needs helps in structuring the M&E system and, in particular, knowing how to make the most of events such as half-yearly revisions, MTRs and annual participatory reviews. To check if your M&E system is providing the information you need, refer to the five questions in Box 2-2. If part of the picture is missing for you, then you need to adjust existing M&E processes.

Not only managers have information needs. Everyone involved in the project has specific tasks in the project and therefore specific information needs. A project M&E system must consider as many needs as possible. Only then will managing for impact become a participatory learning process. Section 5 discusses some ways to deal with different information needs. For this reason, all projects must have a range of learning events. For instance, the Tropisec project in Nicaragua seeks inputs from 16 different events and information sources (see Section 2.7.2).

To use information most effectively for managing for impact, think about the key moments during the project life when strategic decisions are made that enable you to move closer to a poverty reduction impact (see Figure 2-3). Information from M&E will be most useful if it feeds into these moments.

Thinking about these key moments as learning opportunities to manage for impact can reveal their value as strategic steering exercises, rather than as obligatory. Keep the level of discussion and type of decision appropriate to the event. For example, an annual review process is usually not the best moment to discuss how to organise the delivery of stationery supplies to village groups. Nor is a weekly staff meeting the appropriate place to agree on the new terms of contracts with partners.

Figure 2-3. The sequence of learning opportunities during the project cycle

2.2.3 Recognising and Tracking Operating Constraints in the Project Context

Understanding context is critical when it comes to assessing relevance of strategy and activities, anticipating operational problems and judging a project's contribution, and also when designing the M&E aspects of a project.

You can start to disentangle the project's contribution by analysing the evolution of project interventions alongside other concurrent external influences that affect primary stakeholders. Continual updating on context also allows you to adapt the project strategy and operations while en route. A systematic and regular assessment of operating conditions helps with anticipating some of the operational issues that could arise. For example, in Nepal, one project manager receives quarterly "political situation monitoring" reports to understand how the project may be affected. These reports include a brief description of that period's critical events - such as safety and insecurity, government actions, demonstrations and dialogues - and their implications. The information is collected via key informants, newspapers, radio and television. It is essential for deciding whether or not to suspend activities in an insecure area.

Because contexts vary across projects and during the lifetime of a project, constraints and coping strategies will also vary (see Box 2-6). It is important to adjust project strategy and implementation to the extent that circumstances allow. The first element of managing for impact - guiding the project strategy (see Section 2.3) - may be particularly helpful for dealing with changing contexts. Section 3, on adapting the initial project design, also provides useful ideas.

 

Box 2-6. Enabling and constraining contexts

  • Guatemala: At project start-up, political and socio-economic changes had led to a resizing of the state and to decentralisation of its functions. The project responded with a new strategy. It involved local actors (especially grassroots organisations) in a continual process of planning and evaluation, which facilitated the gradual decentralisation of project services. So an operating "constraint" was turned into an advantage and led to more participatory forms of monitoring and self-evaluation.
  • Ghana: In 1996, clients complained that they would only rarely see the intermediaries - the banks that provide credit. After this complaint, the project created a "cycle of training" so that there would be constant follow-up and counselling. This created a new challenge - ensuring sufficient staff at the banks. Although they stipulated three staff members per bank to follow up with the clients, in practice there was often only one because of the government's no-hire policy.
 


Those involved in designing a project also need to understand contextual issues. Design teams may unknowingly specify certain modes of operations and organisational relationships that cause significant problems for negotiating and implementing good M&E (see Section 4 for more details on designing M&E). Tables 2-2 to 2-4 list project and context features that affect project management and M&E. As a project manager or the person responsible for M&E, you might wish to keep these in mind when deciding what direction to take in developing an M&E system.

Table 2-2. Project features with methodological implications

How many implementing partners are anticipated?   The more there are, the more time is needed to build capacity and negotiate for an M&E system that meets all information needs and is genuinely shared.
What aspects of the local culture and government system may help or constrain participatory processes?   The existing local and organisational cultures may help or hinder more participatory M&E efforts.
Who is the cooperating institution - and what does it say about or require from M&E?   Make sure that its and IFAD's understanding of M&E procedures and requirements do not contradict each other. Ask both to clarify what they want/need.
How long is the timeframe for the project?   The longer, the more time you have for developing effective M&E with all stakeholders but also the greater the pressure to produce far-reaching impacts and the greater the need to adjust the project en route in reaction to changing contexts.
How much of the project design is fixed?   Knowing the boundaries for adapting the project enables you to focus your critical-reflections effort on those areas where strategic change is possible.
Have learning opportunities been designed into the project?   The less this is the case, the more effort is needed to put in place incentives to use M&E as a genuine learning tool to adjust the project strategy and operations


Table 2-3. Project features with communication implications

How large and decentralised are project operations?   The larger and more decentralised, the more decentralised analysis of M&E data can be and also the more attention needs to be paid to setting up communications flows so that the managers stay informed of what is happening locally.
How decentralised is the government in the project area?   If more decentralised, then theoretically there is more potential for stakeholder involvement in project M&E, with more options for downward accountability as a form of beneficiary-oriented transparency.
How hierarchical is the governance tradition in the project country/area?   The form of governance may require or prohibit the involvement of various levels of bureaucracy in M&E and may bring with it more or less bureaucracy.
How receptive are the local cultures to sharing problems and learning from mistakes?   Cultural-historical contexts will determine the extent to which certain types of information can be criticised and to which people learn from error and share problems.
How many funding agencies and partners does project management have to report to - and what do they need?   The greater the diversity and number of funding agencies and implementing partners, the more care is needed to keep different reporting/accountability demands manageable.

Table 2-4. Project features with implications for the quality of information

Is the project in an area of civil unrest?   In areas of conflict, the "start-stop" nature of projects will compromise continuity of implementation and thus of progress assessment, besides making the attribution of impact more difficult due to the large impact of external factors.
What kind of operational style does the project have?   Different types of projects (more or less decentralised or open-ended) will need to consider different types of indicators to assess the quality of the processes.
How decentralised and "good-governance" driven is the government in the project area?   The less transparency in a government sector, the more difficult it is to unearth and use highly critical data that emerge from the M&E system.
What type of development goals does the project strive to achieve?   Depending on the project focus, the impact on poverty alleviation will be more or less direct and will therefore affect how to undertake impact assessment and demonstrate attribution clearly.
Does the project focus on physical changes or on human capacities - or both?   More quantitative and physical change-oriented project goals versus capacity development and empowerment-focused goals will make other demands on assessing long-term impact in terms of improved lives.
How expansive is the project?   The larger the geographic coverage of the project, the more conscious effort needs to be made to understand local people's views and to assess above the activity level.

Back to Top

2.3 Guiding the Project Strategy for Poverty Impact

2.3.1 About the Project Strategy

Let's turn now to the first of the four elements of managing for impact in more detail - the project strategy and how to guide it.

The main output of the formulation phase is a draft strategy for the project. The strategy consists of an objective hierarchy and a description of the necessary implementation arrangements and resources needed. The strategy is the basis for the project appraisal report. For IFAD-supported projects, the project strategy is based on the logical framework approach (LFA) and project objectives are summarised as a hierarchy in a logframe matrix (see Section 3). But irrespective of how the project strategy and objectives are formulated, the ideas in this section remain relevant.

By necessity, any project strategy simplifies reality. It cannot possibly describe all details of a context and of the intended plan. This means that the documented strategy is a management tool that requires continual adjusting to reflect current contexts and changing needs.
It is fundamental that the project strategy be made as clear as possible at the onset for - and with - all those involved. In one IFAD-supported project, the staff did not have a clear understanding of what the project was about. The initial project design was poorly formulated and had been based on very limited stakeholder consultation. Since the document was so unclear, people were not motivated to investigate and respond to the problems they encountered. Staff were passively implementing whatever the document said, so were driven by activities rather than impact.

2.3.2 The Objective Hierarchy and Assumptions

The objective hierarchy is the spine of the project strategy (see Figure 2-4 and Box 2-7). The hierarchy describes how lower-level activities contribute to higher-level objectives, and how these, in turn, help achieve the overall project purpose(s) and goal.

Figure 2-4 shows an objective hierarchy with four levels and assumptions between all levels. Development organisations use many different names for the levels in the objective hierarchy and even different numbers of levels (see Section 3). But all levels can be considered objectives, as they are something that the project stakeholders want to achieve. This is why the term, objective hierarchy, is used (although the terms, intervention logic and narrative summary, are also used). The hierarchy is the first column of the logframe matrix and shows how the "means" lead to the "ends" of an intervention.

Figure 2-4. A project's objective hierarchy

Figure 2-4. A project's objective hierarchy

 

Box 2-7. Definitions for four levels of an objective hierarchy

  • Goal: The long-term objective, change of state or improved situation to which a development intervention, such as a project or project component, is intended to contribute. For IFAD-supported projects, the goal is some form of poverty reduction. The extent to which the project contributes towards the goal is the impact of the project.
  • Purpose: The overall objective of the project (or project component), in terms of overall observable changes in performance, behaviour or status of resources that the project (or project component) is responsible for achieving. Standard logframes use one project purpose while IFAD recognises that a complex project may have several purposes.
  • Outputs: The products, services or results that must be delivered by the project implementers for the project purpose(s) or project component purposes to be achieved.
  • Activities: The actions taken by project implementers, which are required to deliver the outputs by using inputs such as funds, technical assistance and other types of resources.
 

Getting the logic of the hierarchy clear with the implementing partners, including primary stakeholders, is essential - although this does not necessarily mean using the logframe terminology with them. Simply ensure that there is consensus about what is to be implemented in the short-, mid- and long-term and for what overall purpose and goal. If a hierarchy is not logical, then you may end up implementing many different fragmented activities that do not lead to a clear output. Poor logic can most certainly lead to project failure (see Box 2-8). Or you may be promising to deliver an impact for the rural poor that is totally impossible given your timeframe and budget. Finally, if you are not clear about what you intend to achieve by when, then you will find it very difficult to know precisely what you should be monitoring and evaluating.

 

Box 2-8. How project logic of the strategy affects project success1

A Successful Project

Good theoretical [logical] model + Good implementation = Leads to project success

Theoretical Failure of a Project

Incorrect theoretical [logical] model + Good implementation = Leads to project failure

Failure in Implementation

Good theoretical [logical] model + Failure in implementation = Leads to project failure

Absolute Failure

Incorrect/Illogical theoretical model + Failure in implementation = Leads to project failure

 

Figure 2-4 also shows that a project strategy contains many assumptions. An assumption is any condition outside the direct control of the project that is important for the project to succeed. There are two types of assumptions: external factors and those relating to the internal cause-effect logic (see Box 2-9). You will need to make more assumptions as you move higher up the objective hierarchy, as the project is only one of often many stakeholders contributing to rural poverty reduction and so there are increasing issues outside the direct control of the project.

Assumptions need to be identified in initial project design, but this is often poorly done. The assumptions are an important tool for guiding the project strategy. Identifying them helps you know if the project strategy has a reasonable chance of success or is based on unlikely assumptions. Checking them regularly to see which ones are risky for the project, updating them based on better understanding from experiences, and identifying new ones is critical for guiding the project strategy. See Section 3 for ideas on how to work with assumptions as part of your M&E system.

 

Box 2-9. Working with assumptions

In Nepal, the Western Upland Development Programme has the following assumptions related to one of its desired outputs, "implementing a micro-finance programme, which includes training, and credit and savings":

  • market trends and fluctuations do not adversely affect the economic viability of on- and off-farm activities;
  • project staff are properly trained and motivated;
  • target groups are willing and able to participate effectively.

Most project designs include assumptions, such as these, that describe external factors needed for a project to succeed. For example, improving irrigation might be based on an assumption of continued water supply from a dam. But the dam could silt up due to poor environmental management or the government could decide to divert the water to other users for political reasons. By identifying those assumptions, project implementers can accept which ones are outside their direct control or see what they can do to reduce external risks.

But equally important - and more neglected - are assumptions about the internal cause-effect logic of the project. For example, an IFAD-supported project in Mali invests in building covered markets in isolated zones as a strategy for poverty reduction. This means that it assumes that the physical presence of these markets will increase household incomes. But who has access to these markets, will these markets be used, and is there produce to exchange in these markets? Will incomes actually increase for those who use the markets to trade? If there is increased household income, who controls how it is spent? And would it be spent on essentials? These are examples of internal logic assumptions that lie behind the simple statement of "marketplaces will reduce poverty in isolated areas".

Making assumptions explicit helps you to check where the objective hierarchy has weak spots and so tells you what to adjust. It is also an important focus for analysing your monitoring data. For example, in an agroforestry initiative in southeast Brazil, the participating farmers were monitoring their investment in on-farm experiments. They were shocked to see how much time they were still spending after several years and how little productivity had increased. This forced everyone to reject the original assumption that poor farmers would get sufficient returns on their labour after two years. The project component on agroforestry was revised with an improved assumption about a slower rate of return for farmers and thus with more realistic levels of anticipated impact.

 

2.3.3 Accountability and Adjusting the Project Strategy

Knowing how to adjust the project strategy requires knowing how flexible you can be. There are two ways in which projects are adapted.

1. If projects are designed as open-ended strategies, with general directions indicated but with freedom for project partners to define the details of operations and activities… you will refine the project strategy as you proceed with implementation. The more flexible the design, the more you will need a good M&E system to provide information that can help.

2. If the project has been rigidly designed… M&E findings may lead the implementing partners to conclude that certain activities, processes or relationships are no longer relevant or that others are missing. You adjust the project strategy based on a better understanding of what is needed to reduce poverty in your area.

Knowing how to adjust the project strategy requires clarity about what project management is accountable for achieving. You should adjust whatever is necessary to be convinced that you can deliver what you are accountable for - and no more or less.

A project's control over factors in the project environment that influence the achievement of objectives decreases with each level of the objective hierarchy (see Figure 2-5). At the level of "activities", project implementers have much control. External factors are unlikely to threaten the implementation of activities seriously. But at the level of the "goal", many factors beyond the direct control of implementers will influence impact. At this level, the project is one of many stakeholders contributing to the reduction of rural poverty. So a project's accountability at higher levels also decreases but never disappears entirely.

Figure 2-5. The limits of control and accountability in a project


Figure 2-5. The limits of control and accountability in a project

For example, a project might include training activities for farmers. The project can directly control the hiring of a training venue, the preparation of materials, the provision of a qualified trainer and the invitation of suitable participants. It has less control over whether potential participants will attend and considerably less control, if any, over whether the skills the participants learn will actually be used back in the workplace. While the training unit of a project can be held accountable for making the training relevant and accessible, it cannot be held accountable for whether the farmers have all the necessary conditions on-farm to implement the new skills they have learned.

Therefore, if you are concerned that the project purpose can only be achieved if certain adaptations are made - and you know that you are accountable for delivering the purpose-level outcomes - then you will need to revise the strategy accordingly (see Box 2-10). You may need to negotiate with funding agencies and supervising organisations for approval of suggested changes. The limits of accountability may need to be negotiated with funding agencies - critical if you are to keep the expectations of impact realistic.

 

Box 2-10. Revising a project strategy

An M&E study in Challa village, China, on the welfare impact of an IFAD-supported credit project highlighted the need for a more holistic approach to poverty reduction. Although credit had helped to increase farm-level productivity because of higher adoption of technological packages, this did not automatically lead to increased income and welfare. The study showed that other factors such as post-harvest losses, marketing and agro-processing hamper incomes. Some factors, such as off-farm income-generating activities, had been incorrectly omitted from the initial project design. So the project strategy needed to change to consider farmers' problems in a more integrated way. Local leaders, other primary stakeholders and experts were brought together so that farmers' problems, needs and priorities could be identified in their complexity, thus improving the strategy.

 

2.3.4 Guiding the Strategy

To guide the project strategy, you have several management tools.

1. The idea of negotiation and shared decision-making with implementing partners and primary stakeholders may seem so obvious as to be ignored. Building agreement of the need for and types of changes is the basis of strategic guidance, as this increases the chance that changes will be carried out. Negotiations about change will also be needed vis-à-vis the funding agencies and the government structures in which the project works. Not all changes will be possible but with good M&E data to explain why changes are desirable, your manoeuvring space might increase.

2. The objective hierarchy is a valuable focus for regular (semi-annual and annual) review and planning events during which you compare achievements with targets and try to understand why differences occur - which they inevitably will. Many projects focus on activities during progress reviews, but this is not enough to manage for impact.

What a project aims to achieve is the purpose and goal level, while outputs and activities describe how it thinks it can do this. A progress review needs to look both at the "how" and the "what" question. Looking only at the activities and outputs, you could conclude these are all going as planned. But you also need to ask, "Where is this leading?" to know if you are on track with the planned outcomes and impacts. This will help avoid wasting time and resources on pointless outputs and activities.

3. Assumptions can be reviewed regularly to check if they are still valid. Identify new assumptions that have emerged and delete those that are no longer relevant.

4. Based on your assessment of problems, successes and revised assumptions, check each level of the hierarchy for relevance and completeness. Add new activities or outputs and delete irrelevant ones in line with your assessment. Adjust targets as necessary. This may require negotiation with funding agencies, particularly at higher levels.

5. An M&E system can provide data for and organise reflection processes around points 1 to 4 above.

Box 2-11 shows how one project in Venezuela is assessing achievement of project component outcomes and linking them to an improved strategy.

 

Box 2-11. Self-evaluation for strategic assessment in Venezuela

PRODECOP in Venezuela analyses project component results in five steps.

1. Anticipated outcomes (as planned): assessing for each objective on a scale of 0 (not available) to 5 (excellent) if the objectives were achieved.

2. Unanticipated outcomes: describing the outcomes and assessing whether they made a high, medium or low contribution to the project component.

3. Factors affecting the outcomes: listing the factors and assessing if they were positive or hindering and why.

4. Possible impacts realised: for four levels (primary stakeholders and communities, PRODECOP staff, PRODECOP, other programmes/organisations) listing the possible social, economic, educational and other impacts that might be occurring now.

5. Actions needed to improve success with component objectives: listing all actions that the project will take on itself and all actions that are the suggested responsibility of others (indicating whose responsibility they should be).

The M&E data for this analysis comes from surveys and key informants. The five steps are followed for the three project components: (1) capacity-building for development and civil participation, (2) the financial services and rural finance system, and (3) monitoring and evaluation.

 

Back to Top

2.4 Creating a Learning Environment

2.4.1 What Is a Learning Environment?

The second element of managing for impact concerns the learning environment that needs to be created if people are to provide strategic and operational guidance by reflecting critically on what is happening.

How can you know if your project is actively learning? If you can clearly say "yes, this is happening here" to the following items, then you know that in terms of managing for impact your project is well on its way to having a culture of learning through critical reflection:

  • Individuals feel that their ideas and suggestions are valued.
  • Mistakes and failures are considered important by everyone for learning and not shameful.
  • All the key groups involved in project implementation communicate openly and regularly.
  • Project implementers, including primary stakeholders, regularly and informally discuss project progress, relationships and how to improve actions.
  • Managers listen carefully to others and consciously seek solutions together.
  • During regular meetings and workshops, time is set aside for discussing mistakes and learning lessons.
  • The question, "Why is this happening?" appears often in discussions.

A learning environment can be created through many small changes as well as more far-reaching events and changes. One project in Tanzania integrates more than 20 different ways of working in order to stimulate learning - from the very way in which the project is designed to how fieldwork happens as well as annual reviews with villagers in the project area. Critical to this is the attitude of and example set by senior management and also a dialogue between implementing partners (see Box 2-12).

 

Box 2-12. Importance of feedback to develop a learning environment

The management team for the rural micro-enterprise development project PADEMER in Colombia is very committed to M&E. A success of management has been its ability to act on M&E-based information. Monitoring visits to the field have led to the cancellation of contracts with some implementing agencies, and changing the implementation processes and feedback has been used to encourage implementing partners to adapt their strategies. Four times a year, the partners submit financial and technical reports to the M&E unit, which analyses them and provides feedback. This feedback has been important for the partners to learn about what information is relevant for reporting. They are reporting results and processes, rather than simply listing activities and goal achievement.

 

2.4.2 How Management Styles Affect Learning

In one project in Latin America, the project manager asks colleagues to evaluate her performance. This type of management attitude is rare. Yet it sends a clear message of being open to feedback and prepared to learn from colleagues. Few managers are selected on their management skills and attitudes, but all can work on improving the needed skills. Staff in one project identified two essential qualities required by project managers to support monitoring and learning:

  • Attitude - The person chosen for project manager must have a basic understanding that M&E is essential to the project and that M&E staff are colleagues, not competitors.
  • Willingness - There must be a sense of commitment to learning and to creating a positive team spirit. This is demonstrated through staff-manager relationship building, open and transparent feedback and dialogue, and allocating sufficient resources for M&E.

While the wider cultural context will affect how a project creates its learning environment, the internal project culture lies within the influence of a project manager. When two projects in the same country were compared, one was designed better but managed the M&E system poorly, while the other had a poorly designed but well managed system and appeared to be doing better. Good management can deal with many problems, even those of poor design.

Encouraging learning does not have to be complex (see Box 2-13). In Ghana, the World Bank director has an "open door" policy. If project staff have a quick question, they can call and ask the question and frequently get verbal clearance and advice immediately. This streamlines project decision-making because team members do not have to try and frame the question in a letter, mail it, then wait for a reply. This has been particularly helpful in procurement and financial management. An even simpler idea comes from India, where the project director has a habit of extensive touring to the local development groups. This enables more qualitative monitoring and direct impressions. The project director provides a feedback letter to whomever he has visited. Section 8 gives more ideas on encouraging critical reflection.

 

Box 2-13. Sharing ideas can avoid staff frustration

Although exchanges between projects can greatly stimulate learning, IFAD-supported projects have tended to act as isolated islands of activity. An outward-looking style of management can avoid this. Two IFAD-supported projects in the same country share similar goals and many of the same management staff within the Department of Agriculture, including the same provincial project coordinator and the same provincial agricultural monitoring person. Each project has separate managers. Yet they rarely meet together or organise any kind of forum for consolidation of project experiences or simple coordination of activities. This has frustrated other project staff due to the ensuing lack of direction, prioritisation and linkages both in and between projects, leaving them to balance their own work schedules between the two projects and with their other non-project related work duties.

By contrast, one of the most important sources of inspiration for organising the M&E of operational aspects in a project in Benin came from contact with another nearby IFAD-supported project that had begun a year earlier. Most important was the production of an M&E manual specifically focusing on, for example, formats for recording M&E findings and making reports, as the first project had done. Documentation was borrowed from the first project, and the managers from both projects have visited each other and exchanged thoughts on M&E.

 

2.4.3 Valuing Problems to Avoid Failure

You may have had the experience of reading a report about a project that you know well and realised that what you are reading is a very different story than what you know to be true. It is likely that what has not worked well is not being reported.

Problems occur on a daily basis in any project. They are not the same as failure. In fact, mistakes can help in avoiding failure - but only if they are used for learning. It is common wisdom that we learn more from failure than success. So is it not strange that everyone tends to overemphasise, and even exaggerate, success while downplaying problems and failure?

All organisations and individuals generally want to portray themselves as being successful. The desire for good news is present at all levels of the system, from ministries and funding agencies, down to field reports. It stimulates the ongoing reporting of the "myth of success" (see Box 2-14), which means missing a key learning opportunity.

 

Box 2-14. Under-reporting non-success

In one project, indicators were developed for classifying cooperatives in terms of their performance. The cooperatives were classified but with superficial analysis. A key staff member involved in cooperative development keeps much detailed performance analysis in his head. Without this more extensive analysis shared and documented, he says that the analysis tool can often give the impression that a cooperative is more advanced than it is; for example, that the cooperative could operate independently even though, in reality,