Enabling poor rural people
to overcome poverty
I. Review of Achievements in 2001
1. The Office of Evaluation and Studies (OE) priority areas for 2001 were: (i) improving IFAD’s ability to assess impact; and (ii) promoting innovations and policy dialogue.
A. Improving the Ability to Assess Impact
2. Assessing the impact and performance of IFAD’s operations has been a priority in IFAD, and in OE in particular, for quite some time. This priority has been further boosted by the Plan of Action, which emphasizes two interrelated areas that need improvement: (i) IFAD’s own methodology for impact assessment; and (ii) the capability of IFAD-supported projects to undertake systematic monitoring and evaluation (M&E).
New methodology for impact evaluation
"…reexamine current practices, develop an improved methodological framework for impact assessment and use it consistently in evaluating IFAD’s projects and programmes." – Plan of Action (Objective Biii, Action 2)
3. The ultimate objectives for introducing this methodology are to: (i) achieve a better evaluation and analysis of impact; (ii) produce a consolidated picture of achievements and performance for a cohort of completed projects; and (iii) provide an indication of IFAD’s contribution to the International Development Goals. The methodology consists of a set of common evaluation criteria, developed in 2001 and includes agreed categories of impact indicators for rural poverty reduction. The common evaluation criteria are: Rural Poverty Impact; Performance of the Intervention; and Performance of the Partners (see box, paragraph 23). The use of common criteria across evaluations will increase consistency in approaches between evaluations, ensuring that the impact of projects is systematically assessed and that the results are comparable across projects. Consequently, an overview of impact and performance of a group of evaluated projects per year can be provided, and generic lessons identified and analysed. This will greatly facilitate periodic reporting to senior management and subsequently to the Board.
4. In September 2001, OE, supported by the Technical Advisory Division (with funding contribution from the Belgian Survival Fund), conducted an international workshop in Fiuggi on malnutrition and related anthropometric indicators (stunting, wasting, underweight). Participants from donor agencies, NGOs, research institutions and governments attended the workshop, which reached a consensus on the importance of including nutrition in IFAD’s definition of poverty. This outcome was taken into consideration in developing the new impact evaluation methodology.
Improving M&E systems at the project level
"…identify best practices and develop tools and guidelines for an effective monitoring and evaluation (M&E) system at the project level." – Plan of Action (Objective Biii, Action 3)
5. To analyse the requirements for better practical tools and guidelines to support M&E processes at the project level, OE undertook a review of the M&E sections contained in evaluation reports prepared by OE since 1990. The review confirmed that, in general, the M&E systems in IFAD-supported projects are performing poorly. The review also revealed that while a considerable amount of material is available on M&E theories, there is a lack of a practical guide on methodologies and processes at the operational level. The only publication on the subject currently available at IFAD is a now outdated booklet ("Monitoring and Evaluation Guiding Principles for the Design and Use in Rural Development Projects and Programmes in Developing Countries") produced by IFAD in 1985. In consultation with the Project Management Department, it was therefore decided that OE would develop a new comprehensive guide entitled, "Managing for Impact in Rural Development – A Guide for Project M&E". The new guide has been developed in consultation with more than 30 ongoing projects in 16 countries and with the active involvement of IFAD and UNOPS staff. It is targeted primarily at the managers, M&E officers and implementation partners of IFAD-supported projects.
B. Promoting Innovation and Policy Dialogue
6. One way for IFAD to play a catalytic role is to multiply its impact by promoting the replication and upscaling of innovative approaches. Another way is to promote effective advocacy and policy dialogue. In 2001, OE work contributed to both aspects of IFAD’s catalytic role.
Promoting innovation
"…develop methodology and evaluate IFAD’s capacity as a promoter of replicable innovations in rural poverty reduction, in co-operation with other partners." – Plan of Action (Objective Biv, Action 1)
7. In 2001, IFAD developed and finalized the "Evaluation of IFAD’s Capacity as a Promoter of Replicable Innovation", with co-financing by the Swiss Development Cooperation (SDC) and the Finnish International Development Agency. This exercise was one the first evaluations at corporate level that OE has undertaken. The evaluation provided the following recommendations and detailed steps to enhance IFAD’s institutional capacity as a promoter of replicable innovation: (i) promote a common understanding of innovation and its processes in IFAD; (ii) ensure strategic commitment to innovation; (iii) create a pro-innovation human resource policy and incentive framework; (iv) improve the management of the "innovation chain", from testing/adapting to replication/upscaling to marketing, and including the development of the capability to perform in this chain; and (v) enhance the culture and leadership role within IFAD to strengthen innovation capacity.
8. As one of the main objectives of IFAD technical assistance grants is to promote innovation, OE initiated an Evaluation of the Technical Assistance Grant Programme for Agricultural Research. This evaluation is expected to have an important bearing on the formulation of IFAD’s policy on the Technical Assistance Grant programme, expected to be submitted to the Board at the end of 2002.
Promoting policy dialogue
9. One of OE’s current strategic directions is to shift from a past emphasis on project evaluations to a "higher plane", in order to generate, through Corporate Level Evaluations, Country Programme Evaluations and Thematic Evaluations, the knowledge and insights required for IFAD to formulate new or revise country, regional and sectoral strategies. In so doing, these higher-plane evaluations will produce the building blocks for, and in several cases actually initiate, the advocacy work and policy dialogue that IFAD is expected to perform at country and regional levels in the future. Throughout 2001, OE continued to emphasize Country Programme Evaluations, Thematic Evaluations and other activities that lend themselves to a broader analysis of policy and strategic issues.1
10. For example, in 2001 OE conducted a Thematic Evaluation entitled "Agricultural Extension and Support for Farmer Innovation in Western and Central Africa: Assessment and Outlook for IFAD." The purpose of the study was to draw useful lessons in order for IFAD to revisit its existing regional approach to agricultural extension and to support farmer innovation that could provide guidance for designing future operations and orient policy dialogue. The preliminary results of the study were presented at the September 2001 session of the Evaluation Committee as well as at the annual meeting of the Neuchâtel Initiative Group2 in London in November. The evaluation will be used as a prelude to a regional workshop and policy dialogue initiative that PA and OE will be undertaking in Western and Central Africa in 2002.
11. In 2001, OE completed three Country Programme Evaluations (Papua New Guinea, Syria and Vietnam) and began two new Country Programme Evaluation in Sri Lanka and Tanzania. As part of the Country Programme Evaluation process, the OE approach to evaluation requires that a national workshop, with the participation of a broad range of partners, be held at the end of the evaluation in order to finalize the Agreement at Completion Point. The Country Programme Evaluation processes triggered a dialogue over a wide spectrum of policy issues, and the outcomes of the evaluations were debated by IFAD staff, government line agencies, civil society and NGOs, universities and donor agencies. In short, the Country Programme Evaluations, and more specifically their Agreements at Completion Point, have provided timely and substantial strategic building blocks, for formulating or revising existing COSOPs.
| Country Programme Evaluation processes contributing to policy dialogue3 The Vietnam Country Programme Evaluation workshop was held in March 2001 in Hanoi. Three main policy issues were discussed with a broad partnership: (i) eliminating subsidies to the rural financial sector and strengthening rural financial institutions; (ii) empowering women; and (iii) decentralization and bottom-up development. Intensive dialogue produced the following outcomes:
|
C. Other Achievements
Communication
12. OE completed the development of its new approach to communicating evaluation results. The key elements to effective communication were articulated in OE’s "Towards A New Approach to Communicating Learning Generated through Evaluation". One of the changes introduced by OE is the shift from lessons learned to the production of "Insights" . Experience has demonstrated that lessons from evaluation were often generated without an appropriate consultation and validation process and were prescriptive in nature. As a consequence, the learning generated was not always used broadly or was not always considered very useful. The shift in process and content proposed, with the emphasis on "Insights" reflects a more incisive and inclusive process of knowledge generation that aims to spark debate on learning hypotheses rather than produce prescriptive lessons.
13. Among the new initiatives launched in 2001 to improve the dissemination of evaluation results are the introduction of a communication approach in each evaluation and the development of a standard package of core communication products for each evaluation.4 OE also continued to develop a new home page on evaluation (which may be directly accessed through the IFAD web site) that contains a repository of evaluation-based knowledge, insights and reports. In addition, a feasibility study was undertaken with the World Bank on the creation of a "help desk" function for OE in order to improve dissemination and sharing of evaluation findings, recommendations and insights to internal and external clients, and help identify gaps in knowledge
Partnership with evaluation units of other development agencies
14. In 2001, OE continued to hold partnership meetings with the evaluation divisions of other development agencies. The overall objectives of these meetings were to understand the role of evaluation in other organizations, exchange experiences on methodologies and assess opportunities for future partnerships in areas of mutual interest. A concrete outcome was the agreement to exchange consultant lists as well as work programmes on a regular basis. In 2001, IFAD also signed a partnership agreement with the SDC on "Partnership on Development Effectiveness through Evaluation." During the year, this partnership included cooperation in the Tanzania Country Programme Evaluation, the production of the M&E practical guide, the thematic evaluation on agricultural extension and the evaluation of the evaluation of IFAD’s capacity to promote replicable innovations. The partnership has also resulted in OE extending assistance to help SDC adopt IFAD’s new approach to evaluation.
D. Taking Stock of 2001
15. Demand for OE’s evaluations is high. In addition, OE’s approach to evaluation has the objective of producing concrete learning effects together with the evaluation partners and users. As such, each evaluation requires an intense debate of its findings and recommendations, as well as systematic communication efforts. Ultimately, this results in increased demand for resources, in particular staff time. The achievements of the Work Programme for 2001 were due in great part to the extraordinary efforts of all OE staff, as well as to the mobilisation of supplementary funds. Despite these efforts, a number of activities scheduled to take place could not be undertaken with the desired depth, owing to time constraints and limited staff resources. The level and intensity of effort made in 2001 will not be sustainable in 2002, especially since two of the three Senior Evaluations Officers left IFAD at the end of 2001. Moreover, the additional position of senior evaluator envisaged for the implementation of the new impact methodology (see paragraphs 3 and 23) did not materialize due to the decision to adhere to the zero-growth policy in the administrative budget.
16. In June 2001, OE held a Mid-term Review Retreat to discuss the three most prominent issues affecting the implementation of the work programme: time constraints, consultant management and teamwork. Staff were unanimous in their concern over the extraordinary amount of work that was expected to be achieved. In addition to an already demanding work programme OE was requested to contribute to a number of unforeseen activities such as the Re-engineering, presentation of and follow-up to the Rural Poverty Report and the Strategic Framework. OE staff were eager to find a solution on how to best manage their time in order to prioritize and successfully accomplish their work. Staff agreed that an analysis of use of time within OE should be undertaken in order to reduce time constraints in the future. Linked to the issue of time constraints is the process of managing a large number of consultants each year, which ultimately has a significant impact on the quality of the evaluation work undertaken by the division. With respect to teamwork, staff members emphasized the need to improve peer reviews and the exchange of evaluation knowledge and experience between evaluators. This was perceived as a powerful factor towards enhancing the quality of OE’s work.
| Building upon the lessons for 2002
|
II. Priorities and Objectives for 2002
17. While formulating its priorities for 2002, OE was influenced by the following considerations:
- The New Strategic Framework is a key driving force for OE, particularly its three strategic objectives6 as well as its emphasis on the need to increase IFAD’s impact in the field and enhance its catalytic role in the international community. In 2002, OE will concentrate its evaluation work on contributing to the implementation of the Strategic Framework. Project evaluations will use the new impact methodology developed in 2001. Impact will also be enhanced by promoting more effective M&E systems at the project level through the regionalization of the Practical Guide on M&E (see paragraph 23). Thematic and Country Programme Evaluations will continue to foster the development of IFAD’s country, regional and sectoral strategies as well as generate the knowledge required for policy dialogue, advocacy work and partnership building.
- The Sixth Replenishment of IFAD’s Resources is expected to request a report from IFAD on the progress made in implementing the Action Plan. In 2001, OE made a significant contribution to the Action Plan on three fronts7, and these efforts will continue in 2002. This will include the communication of the results achieved to date.
18. Other considerations in formulating the 2002 priorities were the criteria identified during the June Mid-Year Review and elaborated during a number of OE staff meetings, and ongoing activities that need to be carried over into 2002 for completion.
19. Based on these considerations and criteria, OE has identified the following priority areas for 2002:
(i) assessing, communicating and learning from impact;
(ii) contributing to IFAD’s catalytic role; and
(iii) contributing to ongoing debate about IFAD’s field presence.
20. OE’s work programme is primarily demand-driven; that is, it is based in great part on requests from divisions for evaluations of various types. In setting priorities for the number and types of evaluations to be conducted, the following points had to be considered:
- Interim Evaluations are mandatory before beginning the design of a second phase of the project;
- In order to produce the Annual Report on IFAD’s impact evaluation and development effectiveness scheduled for 2002 (see paragraph 23), a critical mass of project evaluations is needed;
- Country Programme Evaluations are required to contribute to the formulation of new or revised existing COSOPs in countries of priority to IFAD;
- Thematic Evaluations are required to support regional divisions in the further development of their regional strategies; and
- Corporate-level evaluations are important for IFAD as a whole, and the two corporate-level evaluations to be conducted in 2002 (IFAD’s mode of supervision and the agricultural research component of the technical assistance grant programme) are particularly timely.
21. The number of evaluators participating in Project Development Teams (PDTs) is also an important factor. At any given point in the year, each evaluator is participating in two PDTs. By the end of each year, each evaluator will have participated in up to four PDTs. In 2002, OE will participate in some 20 PDTs. In addition to PDTs, OE will contribute to the knowledge sharing and feedback process through its Core Learning Partnership (which can be considered a pre-phase of PDTs) and by participating in the Operational Strategy and Policy Guidance Committee, which meets to discuss new projects, policies and country strategies before they are presented to the Board. Taken together, these represent a considerable input to the learning process that should be invested in the design and preparation of projects and strategies.
22. A schedule of OE’s work programme for 2002 is provided in Annex II. The following sections describe how OE intends to implement these mutually reinforcing priorities.
III. Main Features of the Work Programme for 2002
Assessing, Communicating and Learning from Impact
23. Under this priority, OE intends to implement the methodological work undertaken in 2001, which aims at improving IFAD’s capacity to assess the impact of its operations. Activities under this priority area can be divided as follows:
- Implementing the new methodology for impact evaluation at corporate level.The new methodology will be developed and applied by OE in phases and builds on IFAD’s efforts to strengthen M&E systems at the project level. It embodies a unified definition of rural poverty impact based on the following six domains of the livelihoods of the rural poor (I) physical and financial assets; (ii) human assets; (iii) social capital and people’s empowerment; (iv) food security (production, income and consumption); (v) environment and common resource base; and (vi) institutions, policies and regulatory framework. The choice of these domains and their definition are based on the Rural Poverty Report 2001 and IFAD’s Strategic Framework 2002-2005, as well as OE’s experience in the field evaluating a large number of IFAD-supported projects. In 2002, OE will ensure the comprehensive application of the common evaluation criteria (see box below) in all evaluations. The results of these evaluations will provide the basis for the production in 2003 of an Annual Report on IFAD’s impact evaluation and development effectiveness, to be published thereafter on a yearly basis. The report will analyse the results of a number of project evaluations undertaken in a given year, as well as Thematic and Country Programme Evaluations, in order to provide senior management and the Executive Board with a consolidated picture of impact achievement and effectiveness derived from these evaluations, as well as a synthesis of lessons learned during the year.
| The main evaluation criteria to be used in all evaluations undertaken in 2002 are:
|
- Regionalizing the Practical Guide for Project M&E. In December 2001, the final draft of "Managing for Impact in Rural Development – A Guide for Project M&E" was completed. In 2002, the guide will be tested in the field and customized to reflect regional diversity. Workshops will be conducted in Western and Central Africa (PA) and Asia and the Pacific (PI) to launch the guide and fine-tune it according to regional specificities (including translations), as well as to train project managers, consultants, government counterparts and other potential users. The process of regionalizing the guide will also aim at establishing networks of consultants and technical advisors with expertise in the field of M&E and impact-oriented management. Above all, OE, while supporting the regionalization of the guide, will seek to transfer the ownership of this process to IFAD’s regional divisions and ultimately the projects supported by IFAD.
B. Contributing to IFAD’s Catalytic Role
24. In 2002, OE will contribute to IFAD’s catalytic role by continuing to emphasise Thematic and Country Programme Evaluations. These evaluations are expected to generate pro-poor policy dialogue with other relevant development actors. Activities under this priority area include:
- Conducting Thematic Evaluations. Thematic Evaluations contribute to taking stock of IFAD’s experiences and approaches in specific themes and regions, and provide building blocks towards the formulation of sectoral and regional policies and strategies. In conducting Thematic Evaluations, OE will make a particular effort to engage other like-minded development agencies and actors to participate in the undertaking of these evaluations. In addition to opening greater opportunities for learning, such partnerships will bolster IFAD’s leverage in policy dialogue as well as strengthen its catalytic role. In 2002, OE will conduct six Thematic Evaluations.8
- Conducting Country Programme Evaluations. Country Programme Evaluations have proven effective in promoting pro-poor policy dialogue with recipient governments, civil society, donor institutions and other development partners, and thus contributing to IFAD’s catalytic role. In 2002, OE will conduct Country Programme Evaluations of Indonesia, Senegal and Tunisia, in addition to completing its CPE of Tanzania and Sri Lanka, which were initiated in 2001. The Country Programme Evaluations will contribute to generating knowledge required for revising the respective Country Strategic Opportunities Paper (COSOPs).
25. Three of the above mentioned Thematic Evaluations will advance the findings of the evaluation of IFAD’s Capacity as a Promoter of Replicable Innovation conducted in 2001. This evaluation illustrated through several case studies that, quite often, successful innovations and adaptation are based on indigenous knowledge and social organization models at the local level. The Thematic Evaluations addressing this theme are:
- "Agricultural Extension and Support for Farmer Innovation in Western and Central Africa: Assessment and Outlook for IFAD".This Thematic Evaluation, which was initiated in 2001 with support from SDC, will be discussed at a regional workshop in the Western and Central Africa region during the first half of 2002. Participants will include farmers’ organizations, managers of IFAD-supported projects, public and private extension service providers, regional government representatives and development institution members of the Neuchâtel Initiative Group. Policy shifts in agricultural extension that emerged in the evaluation will be discussed, this entails involving the beneficiaries in decisions on financing for agricultural support, providing research and support for farmers’ innovations, and promoting capacity-building for stronger autonomous management by farmers’ organizations.
- "Promotion of Local Knowledge Systems and Innovations in the Asia and Pacific Region". The rural poor whom IFAD supports in this region have developed critical coping strategies based on their local knowledge systems and innovations. This evaluation will assess and document the use of local knowledge systems, good practices and innovations in IFAD-supported activities in the region. The output of the evaluation is expected to include a series of insights and recommendations that will contribute to the development of approaches that foster better use of local knowledge in future activities, with the ultimate objective of achieving greater developmental impact.
- "Review of Innovative Approaches in Peru." This evaluation will analyse the innovative and successful project approaches that have been undertaken and replicated in Peru during the past ten years as well as determine potential for replication elsewhere. Innovations include: supporting the development of private extension services for smallholders, creating service centers and hubs for poor farmers living in the surrounding areas etc.
C. Contributing to the Ongoing Debate about IFAD’s Field Presence
26. Supervisory missions, which are conducted primarily by cooperating institutions, are among the main tools for IFAD to follow up the implementation of the projects it finances. This arrangement is not perceived as being an effective provider of the implementation support that would be required to promote impact-oriented project management. Furthermore, the current arrangement of indirect supervision separates IFAD from the field and much of the experience generated during the implementation process -- experience that would be invaluable for IFAD, especially in terms of the knowledge it manages and the level of impact it seeks to achieve. IFAD’s very limited and impermanent presence in the field has also been repeatedly cited as a major drawback to its capacity to "participate" in the various projects and policy dialogue/coordination partnerships at country level. The issue is how IFAD could consider increasing its field presence (not necessarily by establishing field offices manned by international staff) in order to have a more active role in in-country partnership, and to acquire the knowledge required for achieving impact. As a contribution towards addressing this question, OE will conduct an evaluation of IFAD’s supervision arrangements. The evaluation will assess several dimensions related to the ways (i.e. through cooperating institutions or directly) in which IFAD undertakes project and programme supervision. The evaluation will also include an analysis of the quality of and approaches to supervision and their contribution to impact achievement. The evaluation will also examine the cost factors that affect the supervision process.
D. Other Activities
27. Strengthening communication. In 2002, every evaluation will be required to consider from the outset of the process how learning and knowledge from evaluation can best be communicated to all the partners concerned. Reports will strive to be reader-friendly and not exceed 30 pages, and each will have a four-to five-page Executive Summary as well as Working Papers of a more generous length. Reports will be systematically distributed in hard copy as well as published on the Internet. In addition, the two core products developed in 2001 will be produced and distributed on a systematic basis: Profiles for all evaluations and Insights for Country Programme and Thematic Evaluations.
|
This one- to two-page document captures the key conclusions from each evaluation in a reader-friendly format. The objective is to provide a "taste" of the evaluation and an incentive to readers to deepen their understanding by reading the executive summary or the main report. Profiles may also provide early warning signals about major issues emerging from an evaluation that may require immediate attention. Profiles are produced in the original language of the evaluation and in English. |
|
This one- to two-page document focuses on key learning effects and conclusions emerging from Thematic or Country Programme Evaluations. It serves to direct attention to critical learning hypotheses and forms the basis for further discussion among professionals and policy makers at IFAD and outside the institution. Insights is prepared through discussions and interactions among the Core Learning Partnership members and other partners. Insights will be a mandatory output from Corporate-level, Thematic and Country Programme Evaluations. |
28. OE will be organizing special workshops and seminars at IFAD and in the field to communicate evaluation results to a broad spectrum of stakeholders. The focus of these fora will be on those evaluations that go beyond the project level (i.e. Corporate-level Evaluations, Country Programme Evaluations and Thematic Evaluations). For example: in the first quarter of 2002, a workshop on the Tanzania Country Programme Evaluation will take place in Dar es Salaam; during the Governing Council, an informal session on evaluation issues will be organized. On this occasion, the evaluation of IFAD’s Capacity as a Promoter of Replicable Innovation will be discussed. OE will also continue to experiment with customized communication products to reach a variety of partners. In addition, the "Evaluation Help Desk" function, for which a feasibility study was undertaken with the World Bank, will be established in 2002.
29. The Evaluation Committee and the Executive Board. IFAD’s Evaluation Committee will hold sessions in February, September and December 2002. At these sessions the Committee will review OE’s work programme for 2002 and will consider a number of evaluations, including the Tanzania Country Programme Evaluation, the Thematic Evaluation on Organic Agriculture in Latin America, the evaluation of the Technical Assistance Grant Programme for Agricultural Research, and the evaluation of IFAD’s Capacity as a Promoter of Replicable Innovation. A written report of the Committee’s activities in 2001 will be presented to the Executive Board during the latter’s Session in April 2002, as part of OE’s Annual Report on Evaluation.
30. In 2002 OE, as requested by the Evaluation Committee,
will explore possible instruments and mechanisms through which to monitor
the implementation of evaluation recommendations that have been reached
in previous Agreements
at Completion Point. This will also require OE to make additional
efforts to articulate recommendations that are time-bound and can be translated
into
action – the ultimate aim being to enhance the performance of IFAD’s
operations and policies.
1 For a complete list of evaluations conducted in 2002, see Annex I.
2 An international consortium of representatives of bilateral and international cooperation agencies active in the field of agricultural extension.
3 Positive policy developments were also achieved through the Country Programme Evalualtions for Syria, Papua New Guinea and Sri Lanka.
4 Two of these products, Profiles and Insights, are described in Section III.
5 See Annex III for a sample of Individual Work Programme and Time Budget.
6 The three objectives are: (i) strengthening the capacity of the rural poor and their organizations; (ii) improving equitable access to productive natural resources and technology; and (iii) increasing access to financial assets and markets.
8 For details, see Annex II.