Published Version of "A Guide for Project M&E - Managing for Impact in Rural Development"

A. Background and Rationale

1. While formulating its New Approach to Evaluation (NAE) in 1999, the Office of Evaluation and Studies (OE) decided, as part of that new approach, to develop appropriate monitoring and evaluation (M&E) methods and tools for IFAD-supported projects. The consensus reached in-house on the new strategic direction was duly reflected in OE's Programmes of Work for 2000 and 2001.

2. The first step in the process was to review the M&E section of most IFAD evaluation reports prepared since 1990. The overall picture was not favourable as many projects saw M&E as a policing exercise, which led to a lack of commitment, mistrust or even resistance to learning from experience. Seven key problems were identified as a result of the review:

(i) Project staff lack commitment to monitoring, leading to delays in the implementation of such systems and little use of the information gathered on the part of project management.

(ii) Monitoring is seen as an obligation imposed from the exterior, with project staff mechanically completing forms and project managers seeing the task merely as the collection of data for writing up reports for donors.

(iii) Irrelevant and poor-quality information is produced through monitoring as it focuses only on physical and financial aspects and ignores factors such as project outreach, effect and impact.

(iv) Scarce attention is paid to the monitoring needs of other stakeholders, such as beneficiaries, community-based organizations and other local partners.

(v) There are very few internal project reviews or ongoing self-evaluations, adjustments being triggered mainly by external evaluations or supervisions.

(vi) There is a widespread lack of integration and cooperation between the M&E function and project management.

(vii) Poor use is made of participatory and qualitative M&E methods due to limited capacity and little recognition of the need for such methods.

3. Over the past year, OE has also made a comparative review of several major development agencies' strategies and approaches to project-level M&E systems. The review confirmed that poorly performing M&E systems are not unique to IFAD, similar difficulties having been encountered by other multilateral and bilateral organizations and non-governmental organizations (NGOs)¹. It is clear that, in most project settings, much improvement and clarity is required.

4. Both exercises revealed that while a considerable amount of material is available on M&E concepts and theories, there is a lack of practical resource kits on methodologies and processes at the operational level. The only publication on the subject currently available at IFAD is the booklet Monitoring and Evaluation Guiding Principles for the Design and Use in Rural Development Projects and Programmes in Developing Countries, produced by IFAD in 1985 in cooperation with the Food and Agriculture Organization of the United Nations (FAO) and World Bank. These rather general 'Guiding Principles' were not specifically intended to promote the establishment and implementation of effective project-level M&E systems. They are not relevant to current policy or the institutional environment or to the new generation of IFAD-funded projects, which are now of a considerably more flexible and participatory nature.

5. In May 2000, an in-house workshop reviewed a report entitled Strengthening IFAD's Support to the Development of Effective and Efficient Monitoring and Evaluation Systems at the Project Level. In stressing the need for clearer and more operational definitions the workshop strongly recommended that IFAD's existing M&E guidelines should be updated and rendered as 'implementable' as possible in order that they could be used by all stakeholders, particularly project field staff. It was agreed that focus should be placed on identifying steps to be followed in designing M&E systems and processes for full integration into project implementation.

6. Under Objective B (iii) - Improve Impact Assessment - of the IFAD V: Plan of Action (2000-2002), it was specified that IFAD should, inter alia, "identify best practices and develop tools and guidelines for an effective monitoring and evaluation (M&E) system at the project level". This objective was also reflected in the scorecard for 2000 and included as a priority activity in OE's work programme for 2001. Furthermore, the in-house working group on Impact Achievement through the Project Cycle (IAPC) held in July 2000 recommended the establishment of more participatory M&E systems, using a logical framework (logframe) approach and focusing on impact.

7. A methodological study on strengthening IFAD's support to the development of effective and efficient M&E systems was presented to the 25th Session of the Evaluation Committee in September 2000. In November of the same year and within the framework of the IAPC working group, further discussions on the subject were held with IFAD staff, project managers, consultants and field agents.

8. All these initiatives and discussions have led to the conclusion that there is need to develop new and more practical guidelines or a resource kit to ensure that M&E has a greater impact at the project level.

Back to Top

B. Objective and Main Target Audience

9. The objective of the proposed practical guide is to facilitate the development, installation and use of effective project-level M&E systems as tools for impact-oriented management. As such, its overriding goal is to contribute to improving the impact of IFAD-funded projects, as reflected in the proposed title of the guide: Managing for Impact in Rural Development - A Practical Guide for Project M&E. This initiative is being undertaken within the framework of the IFAD V: Plan of Action (2000-2002) and complements other IFAD initiatives currently under implementation or scrutiny, such as the installation of a dynamic logical framework approach and a revised project cycle, as proposed by the IAPC working group.

10. The guide is to be mainly targeted at the directors and M&E officers of IFAD-supported projects, although some sections will be particularly relevant to project implementation partners. It will also constitute an important reference resource for consultants contracted to provide technical support/facilitation services to projects and for IFAD's cooperating institutions. As M&E is heavily influenced by project design and budgeting decisions, a module on integrating M&E into project design will be targeted to IFAD staff and consultants involved in this phase of the project cycle.

Back to Top

C. Progress to Date

11. In December 2000, a core group of IFAD staff and representatives from cooperating institutions, known as the Core Learning Partnership (CLP)² , was set up to steer the process of developing the guide. The services of two consultants with considerable experience in participatory and learning-oriented M&E systems were contracted for this purpose.

12. At the first meeting of the CLP in December 2000, participants reviewed the draft terms of reference for the consultants and agreed on the form the guide would take as well as the road map and the steps involved. The second meeting of the CLP, held in January 2001, reviewed the initial conceptual framework and draft outline of the guide, which were subsequently presented to PD and OE for review and comment.

13. Between March and July 2001, necessary tools and processes for the design and use of M&E systems during the project cycle were included in the draft outline together with a number of case studies (good and bad practices). In addition to the main authors of the guide, a number of other consultants have been recruited to prepare case studies on selected projects in consultation with the Fund's regional divisions. From May to September 2001, visits were made to 30 projects in 16 countries (Armenia, Bangladesh, Benin, Colombia, El Salvador, Ghana, Guatemala, India, Indonesia, Mali, Morocco, Peru, United Republic of Tanzania, Uganda, Venezuela and Yemen). Meetings were held with other project managers during two regional meetings in Asia and Latin America.

14. A first draft of the guide was reviewed by the CLP in late July 2001, followed by a workshop to discuss the preparation of a second draft for review in October. The second draft was completed in early October and distributed both within IFAD and to a number of external reviewers who were invited to attend a workshop scheduled for 30-31 October. Apart from IFAD staff, the workshop was attended by 11 managers and M&E officers of IFAD-funded projects and programmes, ten consultants and resource persons, five project supervisors from cooperating institutions and staff of FAO, the World Food Programme and the Swiss Agency for Development and Cooperation. The objectives of the workshop were to:

(i) contribute directly to the production of a satisfactory 'prototype' of the guide by December 2001, by exchanging comments and suggestions with a wide range of potential users; and

(ii) agree on a process for testing the guide in IFAD-funded projects.

15. The 'prototype' is expected to be finalized during November 2001. However, a draft version (in English only) will be tabled at the Twenty-Ninth Session of the Evaluation Committee, to be held in December 2001.

Back to Top

D. What is New about This Guide?

16. While numerous guidelines and handbooks on M&E have been produced, almost none of them have achieved the desired result. IFAD believes that the new guide will be more effective than others and that it will be of real use in IFAD-supported projects for the following reasons:

• The guide is being developed in consultation with potential users. It will be used as a tool for addressing users' problems, starting with their current M&E practices, and, to the extent possible, will include case study examples of good practices from IFAD-funded and other rural development projects. Once a satisfactory 'prototype' has been completed, the guide will be tested in the field and customized according to different countries and regions.
• The guide will not provide a blueprint M&E plan or standardized list of indicators. However, it will be tailored to the specific context, procedures and partnerships of IFAD-supported operations. It will focus on participatory learning processes and propose options that can be adapted to requirements imposed by project managements in different national and regional contexts.
• Above all, the practical guide will not be a stand-alone initiative. It will be part of a range of other efforts, including, in particular, the installation of a logical framework approach and development of regional support networks based on the experience of PREVAL³ in the Latin America region and other regional programmes concerned with training, technical assistance, communication and knowledge management.

  17. Seven basic principles will guide the selection of material for inclusion in the guide:
  
  (i) thinking in terms of a learning approach to project M&E and management;
  
  (ii) emphasizing the value of stakeholder participation in M&E;

  (iii) focusing on the concrete needs of IFAD-supported projects;
  
  (iv) structuring the guide around stakeholders' routine functions and tasks and integrating M&E into the project cycle;
  
  (v) ensuring that a common planning and M&E language is used;

  (vi) giving value to a diversity of working methods and styles; and

  (vii) acknowledging the importance of institutional and human capacities for implementing M&E.

Back to Top

E. Proposed Contents of the Guide

18. M&E in rural development means learning about achievements and problems in order to improve implementation and thus achieve a better impact. The guide will offer ideas on how to set up and improve such M&E systems and processes, and it will discuss key aspects that a project management team must bear in mind in developing a sound learning system that will help to improve management and ensure accountability. It will include many practical examples of how project staff have successfully strengthened learning systems in order to achieve an impact.

19. In particular, the guide will respond to a growing number of requests from project staff, funding agencies and project beneficiaries for practical ideas on: assessing impact, rather than measuring only outputs or activities; and being more learning/lesson-oriented, and not merely driven by accountability, to ensure greater involvement of beneficiaries in M&E.

20. The ideas put forward in the guide will not constitute a mandatory M&E system that all projects must comply with. Rather, it will describe what are considered - and have proved to be - good practices in project M&E.

21. Although it is essential that there should be flexibility in taking up options, good M&E must meet a minimum set of requirements and standards. The guide will discuss these requirements and standards, and indicate where options are possible.

22. The guide is being developed as a series of stand-alone sections of two different types: (i) 'how to' sections, e.g. how to design an M&E system and how to conduct an annual project review; and (ii) reference sections, e.g. M&E methods and tools, detailed examples of M&E plans and logframe matrixes. The sections will be assembled in separate binders. (An Internet-based electronic version may also be developed to facilitate the tailoring of the guide to the needs of the various countries and regions.)

23. At the present stage, i.e. before the findings of the October 2001 review workshop have been finalized, the contents of the draft guide may be summarized as follows.

Part I - How to Make M&E the Basis of a Learning Project

24. The first part comprises nine, 15-25 page, sections.

25. Section 1 - Introducing the Practical Guide. The key messages in this introductory section are as follows:

• The guide has been written to help project managers and M&E staff improve the quality of M&E in IFAD-supported projects. It responds to a study that unearthed a generally low standard of M&E in most projects.
• The focus of the guide is on how M&E can be used to support project management. It engages all project stakeholders in learning how to improve project implementation, the ultimate objective being to both maximize and demonstrate the impact of rural development projects.
• The guide provides examples of good practices and comprehensive details of how to set up and implement a project M&E system.

26. Section 2 - Using M&E to Manage for Impact: An Overview. This section shows how M&E can help a project to learn about its work, and how it can guide a project to achieve maximum impact on poverty reduction. It is pertinent to all potential users - project managers and M&E officers, IFAD staff and consultants and supervisors from cooperating institutions - and explains the four key elements of impact-oriented management:

(i) guiding the overall project strategy;
(ii) creating a learning environment;
(iii) ensuring effective project operations; and
(iv) developing and using the M&E system.

27. Other key messages are as follows:

• M&E is most effective when it involves stakeholders in a creative process of learning how to improve projects on a continual basis.
• Setting up M&E to support 'managing-for-impact' requires an understanding of key management functions and information needs.
• Making M&E participatory calls for careful attention to how different stakeholders can be involved in the most constructive way.

28. Section 3 - Linking Project Design, Annual Planning and M&E. The focus of this section is on project design and its relationship with annual planning and M&E, demonstrating how to make the best use of the logical framework approach. The main audience is intended to comprise project managers, M&E officers, consultants and facilitators and project designers. Since the guide acknowledges that project (re)design is an ongoing process throughout the life of the project, managers and implementing partners must understand the principles of good design to be able to adapt project strategy and operations in response to changing circumstances and lessons of implementation experience. The key messages here are:

• Good practices for project design (and redesign/adoption) include:

  - involving stakeholders;
  - completing a detailed situation analysis;
  - ensuring a logical intervention strategy;
  - identifying cross-cutting objectives;
  - planning for capacity development and sustainability; and
  - planning for learning and adaptation.

• The logical framework approach is a valuable tool for project design provided it is used in a flexible manner and its common problems are understood and addressed. The logical framework matrix summarizes a project's intervention logic, its underlying assumptions and how M&E will be undertaken. The matrix constitutes a useful management tool and forms the basis for operational and annual work plans, but consistent and regular use and attention to detail are needed.
• Developing a good M&E system calls for adequate attention to M&E during the initial design phase. The M&E system should be outlined in the project appraisal report.

29. Section 4 - Setting Up the M&E System. This section shows what needs to be done in order to create and initiate an operational plan for a comprehensive M&E system. The key messages here are:

• M&E must be understood as a complete system within a given project and should be planned, managed and funded. There are four key parts to a M&E system:

  (i) setting up (or designing) the system;
  (ii) gathering and managing information;
  (iii) using experience and information more effectively to improve action; and
  (iv) communicating and reporting results.

• A good M&E system can be used to guide project strategy and ensure effective operations; it is a part of the larger project management system.
• At each stage of the project cycle, consideration should be given to a set of key M&E tasks.
• A detailed M&E plan should be drawn up during the project start-up and mobilization phases and clearly documented. The M&E system will need to be monitored and updated throughout the life of the project.

30. Section 5 - Deciding What to Monitor and Evaluate. This section, which will be relevant to all users, shows how to choose what to track, document and analyse and with whom. Identifying the information needed, for what purpose and by whom is critical to the success of any M&E system. The key messages here are:

• Five broad project issues guide M&E: relevance, effectiveness, efficiency, impact and sustainability of actions.
• In supporting 'managing-for-impact', M&E systems should highlight successes and failures rather than merely reporting on progress in meeting targets. It is also important to identify unintentional positive or negative effects.
• Less is more! A common mistake of M&E is trying to gather too much data without then analysing it all.

31. Section 6 - Gathering and Managing Information. This section focuses on how best to gather information from the field and use it for good project management. The main audience of this section comprises M&E practitioners, project managers and implementation partners. The key messages here are:

• Methods are needed for collecting data and subsequently checking, sampling, recording, collating and analysing them. Any method used must be technically and financially feasible, accurate (enough), consistent and not prone to distortions. Any number of methods can be used to collect indicators. Therefore, before making a final choice, the options should be listed and their advantages and disadvantages assessed.
• Consider the practicalities of adopting a more-or-less participatory approach to data collection and processing to avoid the primary stakeholders being only superficially involved.

32. Section 7 - Reflecting Critically to Improve Action. This section explains what can be done to ensure sufficient and appropriate reflective learning during the lifetime of a project. M&E data will only help the project if it is reviewed at meetings with the stakeholders involved.

33. Section 8 - Communicating and Reporting M&E Results. This section describes what needs to be communicated as part of the M&E process, how and to whom. The main audience here are project managers, M&E officers, responsible ministries and IFAD and cooperating institution staff. (This section, which was suggested by the CLP, has not yet been written. However, it will contain information of fundamental importance to the ministries concerned, IFAD and its cooperating institutions.)

34. Section 9 - Putting the Necessary Capacities and Conditions in Place. This section explains what is needed to build up human and technical capacity so that effective M&E can be carried out. The main audience here are project managers, M&E unit coordinators and project designers. The key messages are:

• Capacity is more than just 'ability' and involves many people other than project staff.
• People need to be motivated if they are to do a good job. The right incentives are crucial, and can come in many shapes and forms. Equally important, everyone should be clear about his/her responsibilities.
• Careful planning of an M&E unit's organizational structure within a project (including hierarchies, areas of responsibility and authority among stakeholders) ensures that everyone works together for a smoothly run project.
• People's ability for good M&E is dramatically enhanced if the technical capacity of information systems is adequate. People need appropriate conditions, including management information systems, to register and make important information available.

Part II - Detailed Reference Material

35. Part II provides examples and background material to support the practical application of the ideas and principles presented in Sections 2-9 of Part I. It contains eight sections.

36. Section 10 - Glossary of M&E Concepts and Terms. This section, which explains what M&E words mean, is useful to all users.

37. Section 11 - Checklists for Project Managers, Cooperating Institutions and Country Portfolio Managers (CPMs). This section summarizes the core tasks of key actors in project M&E.

38. Section 12 - Methods for Monitoring and Evaluation. The main targeted audience here are the staff of M&E units and consultants. This section presents 50 methods for undertaking specific M&E tasks, and summarizes the purpose, the 'how to' and the strengths/limitations of each task.

39. Section 13 - Annotated Example of a Project Logframe Matrix. This section gives practical examples of M&E experience in IFAD-supported projects. The examples are accompanied by comments on the improvements that were made to projects. Illustrations are provided of how to improve a project logframe and avoid the most common mistakes.

40. Section 14 - Annotated Example of an M&E Matrix. This section, which applies good logic for tailoring the M&E mechanism to a practical example from an IFAD-supported project (as in Section 13), gives examples of evaluation questions, indicators and monitoring mechanisms that can inspire project actors to develop their own. The main audience here are participants in the design and start-up stages of the projects.

41. Section 15 - Indicative Table of Contents of M&E Strategies and Plans in Project Reports. This section lists what it would be useful to include in the strategic and operational M&E plan. The main audience here are project managers, M&E officers, supervisors and technical assistants/consultants.

42. Section 16 - Sample Job Descriptions and Terms of Reference for Key M&E Tasks and Actors. This section describes what should be remembered when writing up the terms of reference for critical M&E tasks/events. The main audience here are project directors and M&E coordinators.

43. Section 17 - Bibliography and Further Reading lists other reading materials available on the issues discussed in the guide.

Back to Top

F. Next Steps - The Guide and Beyond

44. The October 2001 review workshop is expected to result in a substantial improvement in the contents of the guide based on users' needs and expectations. In 2002, the prototype will be tested, in both ongoing and new projects in the different regions. It will be translated and customized according to different users and contexts. This customization phase is essential: (i) to further improve the 'practical' nature of the tool and ensure ownership by local practitioners; and (ii) to facilitate the development of new partnerships and networks of resource persons, facilitators, trainers and other service providers who are needed and requested by many projects.

45. No practical guide or handbook can in itself make a difference, however. To increase the ability of projects and programmes to achieve greater impact, a range of other complementary efforts will need to be undertaken at the field level, with and between IFAD, cooperating institutions and governments.