IFAD’s performance with regard to gender equality and women’s empowerment

diciembre 2010

Corporate-level evaluation

Background and objectives

The Consultation on the Eight Replenishment of IFAD's Resources decided in 2008 that the IFAD Office of Evaluation (IOE) would undertake this corporate-level evaluation on IFAD's performance with regard to gender equality and women's empowerment. The objectives of the evaluation are to: (i) assess the relevance of IFAD's strategy in promoting gender equality and women's empowerment; (ii) learn from the experiences and good practices of other development organizations; (iii) assess the results of activities funded by IFAD related to gender equality and women's empowerment in its country programmes and corporate processes; and (iv) generate a series of findings and recommendations that will assist IFAD's Executive Board and Management in guiding the Fund's future activities in this area.

Process

Four building blocks form the basis of the evaluation: (i) an analysis of the evolution of gender-related concepts and development approaches, and a comprehensive documentary review of the policy and evaluation documents prepared by other development organizations; (ii) an assessment of key IFAD corporate policy and strategy documents; (iii) a meta-evaluation of past operations based on existing evaluative evidence, a review of recent country strategic opportunity programmes (COSOPs) and ongoing projects, and five country visits to gain insight into the perspectives of partners in these countries and collect evidence from the field about the evolving approaches and results of IFAD-funded projects; and (iv) a review of selected corporate business processes that have implications for IFAD's performance in promoting gender equality and women's empowerment in partner countries. Section C in chapter I of the main report gives a more detailed account of the objectives and processes related to the evaluation's four building blocks.

Main findings

There has been an evolution globally in approaches to building gender equality and women's empowerment. Pre-1975 efforts were mainly addressed to men as producers and women as homemakers, which ignored the important role of women as farmers and food producers. Subsequently, there was a shift to women-focused approaches and approaches focusing on changing the relations between women and men. A recent and promising gender equality approach promotes the complementarity of women and men in family production and farmers' organizations (see paragraphs 11-19 of the main report).

The review of literature on the topic revealed that there are two major constraints to the effective application of lessons from previous operations: a reluctance to address gender as a major organizing principle of society and a failure to invest sufficiently. All development agencies with recent evaluations have revealed, at best, mixed success in implementing gender mainstreaming. There was a broad consensus among development partners that this was mainly due to the lack of: (i) results orientation; (ii) consistent leadership and follow-up by senior management and executive boards; (iii) staff incentives and accountability through performance management systems; (iv) a clear understanding of how best to address gender inequality; (v) adequate investment in gender equality expertise in operations; (vi) attention to gender balance in staffing; and (vii) an inclusive organizational culture. Some of the findings from these evaluations are also applicable to IFAD (see paragraphs 20-29).

Because it was established at a time of significant global attention to the need for development to include women, IFAD has always paid attention to gender equality and women's empowerment. Between its inception and 1992, it did so mainly through women-specific project components; since then, it has included women as beneficiaries and as actors more systematically in its projects (see sections A and B in chapter III).

The new millennium marked an increase in efforts to "mainstream" gender equality and women's empowerment in the design, implementation, supervision and evaluation of IFAD-funded operations. This was done mainly in connection with IFAD's Gender Plan of Action 2003-2006, and, subsequently, with the 2008 Framework for Gender Mainstreaming in IFAD Operations. These are two of the key documents that capture IFAD's corporate strategic approach to promoting gender equality and women's empowerment (see paragraphs 40-48).

IFAD's corporate strategic approach to gender is largely relevant and consistent, but fragmented across numerous documents, including the gender action plan and gender framework. Moreover, synergies with other thematic and corporate policies and strategies are not clearly articulated. Interpretations of the terminology and understandings of the topic also differ widely, which has led to alternative approaches in COSOPs and in project design and implementation (see paragraphs 63-76).

The evaluation found no evidence of systematic monitoring or reporting on progress related to gender equality and women's empowerment by either Senior Management or the Executive Board. Reporting is in fact largely confined to project-level activities. It is also fragmented across numerous documents, preventing a consolidated picture of the main results, opportunities and challenges associated with the topic. However, compared with comparator agencies, IFAD has done better in instilling a results orientation in its gender work (see paragraphs 73-74 and 78).

The evaluation reviewed older and recently designed IFAD-funded projects, five country programme evaluations, and five COSOPs developed last year. Overall, the performance of past IFAD-financed projects is moderately satisfactory, but this average rating masks significant variance among projects and countries in attention to gender-related dimensions of design and implementation. Performance in efficiency and promotion of innovations and scaling up is inadequate. The evaluation found a relationship between gender achievement and a project's overall achievements, a finding confirmed by evaluations in other organizations. However, the analysis does not reveal a causal relationship in either direction. Compared with earlier ones, newer COSOPs and project designs are paying increasingly more attention across the board to gender issues. On the whole, the evaluation considers that IFAD's performance is moderately satisfactory in achieving its first two corporate objectives related to gender, but moderately unsatisfactory in achieving the third objective1 (see sections B-E in chapter IV).

The review of corporate business processes found that factors that support IFAD's work on gender equality and women's empowerment include a recognition of the importance of the issue at the most senior levels (in Management and the Executive Board); operational systems and processes that have a reinforced quality enhancement and quality assurance system; direct supervision and implementation support; and wider country presence. IFAD has established an elaborate results measurement framework including gender indicators for design and implementation, but this framework consists of multiple layers and systems and requires streamlining. Independent evaluations assess gender as part of the various evaluation criteria applied (e.g. relevance, effectiveness, etc.), but do not have dedicated gender indicators and/or questions that are applied in each evaluation (see paragraphs 48 (i), 166 and 181-183).

In terms of knowledge management, few efforts have been made to aggregate results coherently at the regional or corporate levels, and lesson-learning and cross-fertilization of experiences on gender issues is limited and ad hoc. Like some of the comparator agencies, IFAD does not invest sufficiently in learning from its experience and building on its successes. Some initiatives to ensure learning and knowledge management are taking place, but they are not systematic or adequately resourced. Their analysis of what has contributed to or prevented progress on gender equality and women's empowerment needs to be strengthened. The role of communication in highlighting IFAD's work related to gender equality and women's empowerment is generally positive (see paragraphs 176, 184-185 and 189-190).

There are some good examples of policy dialogue by IFAD on gender issues in partner countries. However, in spite of this, on the whole policy dialogue performance at the country level is variable, but generally unsystematic, with little analytic underpinning and not backed by the required human and financial resources. On the other hand, IFAD has played a useful role in selected global policy and advocacy platforms (e.g. farmers' associations) by drawing attention to the plight of rural women and their central role in smallholder agriculture and rural development processes (see paragraphs 186-187).

Partnerships with civil society organizations and NGOs working on gender issues are generally positive. Some good examples of relations with borrowing-government agencies that deal with gender equality and women's empowerment were also found; this varies considerably from country to country. Partnerships with donor governments have been good in terms of the supplementary funds mobilized at the corporate level, but generally limited in terms of discussion of content issues with bilateral aid agencies involved in operations. However, IFAD representatives actively participate in the gender networks of the United Nations and those of the Organisation for Economic Co-operation and Development/Development Assistance Committee. No significant partnerships devoted to gender were apparent with the private sector, apart from a handful of initiatives at the project level and with the Farmers' Forum (see paragraph 188).

In terms of gender architecture, the gender desk within the Policy and Technical Advisory Division has spearheaded the process of ensuring attention to gender in project design and implementation, and supports other corporate processes such as communications and fundraising. In the five geographic regions covered by IFAD operations, the gender desk has provided support to leaning and knowledge management, but resources for this function are variable and inadequate. The contributions of the thematic group on gender were influential in the early years of the gender action plan, but more recently, there are indications that the group needs revitalization through the development of clearer results-oriented objectives and workplans. The role of gender focal points is unclear, and their accountability and working relationships also need review (see paragraphs 168-170 and 177-178).

Arguably, the greatest challenge in IFAD's gender equality and women's empowerment work relates to its human resources management and corporate culture. Historical data reveal that the ratio of women to men in the organization is rather traditional, with many women in support staff positions and few in leadership functions, even though in 2010 IFAD hired its first woman Vice-President and first woman Director of the Office of President and Vice-President. There are an encouragingly high proportion of women in the more junior Professional category. Even though there is room for further improvement, IFAD nevertheless compares well in terms of its gender balance in staffing with selected United Nations organizations and international financial institutions. Although IFAD's Human Resources Policy explicitly makes reference to ensuring gender balance in IFAD's workforce, the guidelines for consultants' recruitment makes no such provision. In fact, the evaluation found that few women and gender experts were recruited as consultants in the teams responsible for COSOP development and key phases of the project life cycle (e.g. design and supervision missions, including evaluation) (see paragraphs 193-200).

IFAD's human resources policy includes anti-harassment provisions and a variety of work-life balance policies. The evaluation found poor uptake of the options provided in formal policies (e.g. special leave without pay). IFAD's informal culture has not traditionally encouraged the inclusion of perspectives and ways of working that are women-friendly or family-friendly, although these are important for building an organization capable of delivering on gender equality and women's empowerment (see paragraphs 202-204).

Similarly, there are no specific incentives for attention to gender equality in staff, consultant or divisional performance; and no accountability or negative consequences for lack of attention. Performance in this area does not figure in individual workplans, which tend to be largely activity-based. In other words, the type of results orientation that is strongly advocated in IFAD-funded operations is not applied to an equal degree in individual work planning and human resources management (see paragraphs 176 and 179).

Another shortcoming is that IFAD is unable to track in its loan investments the amounts allocated to advancing gender equality and women's empowerment, and there is no evidence that it has taken concrete steps to address this problem. Similarly, there is no indication of the funds allocated ex ante for this purpose in IFAD's annual results-based programme of work and budget. The evaluation also found that there has been a disproportionately high reliance on supplementary funds and grants for core gender activities. Access to such sources of funding is not secure – and, for supplementary funds, specific efforts are required for additional periodic reporting to the concerned donors. The Board, on its part, has not asked IFAD to provide information on the amount of resources the organization is investing in promoting gender equality and women's empowerment (see paragraphs 165, 172 and 191-192).

Conclusion

IFAD's strategic approach to gender is relevant, but guidance is fragmented in several corporate documents. The Fund's effectiveness in meeting one of its three strategic objectives (improve women's well-being and ease their workloads by facilitating access to basic rural services and infrastructure) is moderately unsatisfactory. Moreover, results of IFAD-financed operations are moderately satisfactory on the whole, even though there is significant variability across projects and countries. A number of key corporate business processes that are essential for supporting IFAD's gender work remain weak. In sum, as far as the promotion of gender equality and women's empowerment is concerned, there seems to be a gap between rhetoric and practice, which raises the question of whether IFAD is indeed "walking the talk".

Recommendations.

The evaluation makes the following recommendations:

  • Develop an evidence- and results-based corporate policy on gender equality and women's empowerment. IFAD should develop its first overarching corporate policy on gender equality and women's empowerment, for submission to the Executive Board in 2011. The policy would be IFAD's principal reference document on gender equality and women's empowerment, bringing under one umbrella the Fund's main strategic objectives and priorities in this area. The recommended policy should indicate how key corporate business processes will be adjusted for better results on the ground.
  • The gender policy should include a section on who within IFAD Management will be responsible for implementation, oversight and reporting. IFAD should produce a consolidated annual progress report, covering the results achieved in the implementation of the new policy, lessons learned and adjustments made to key corporate business processes that affect performance in gender-related activities. The policy should also include an overarching results measurement framework for IFAD's gender work, and specify how the Executive Board will fulfil its role in providing guidance and support, as well as oversight on results.
  • Knowledge management, learning and analytic work. IFAD needs to invest in building a common evidence-based understanding among staff of the theory of gender equality and women's empowerment, and its related terminology. Among other issues, this should include attention to the systematic documentation and cross-fertilization of lessons learned and good practices across projects, countries and regions, and at headquarters as well as in the field.
  • Innovation and scaling up as key principles of engagement. The corporate-level evaluation on innovation recommended that IFAD define an IFAD-wide innovation agenda at the corporate level that consisted of a few selected themes or domains. The themes or domains selected, "big bets", should be in areas of the agriculture and rural sector where there is a proven need for innovative solutions and where IFAD has developed (or can develop) a comparative advantage in successfully promoting pro-poor innovations that can be scaled up. In this regard, this evaluation recommends that gender equality and women's empowerment be included as one of the "big bets" in IFAD's corporate innovation agenda until 2015. IFAD should, however, remain open to promoting gender-related innovations at the country or project level that respond to context-specific challenges. COSOPs and project designs should outline the specific efforts needed to ensure that successful innovations can actually be scaled up for wider impact on gender equality and women's empowerment.
  • Policy dialogue. Policy dialogue and advocacy work should focus on the selected "big bets", but also on specific thematic areas that might require attention in a given country context. Furthermore, staff competencies and skills will need to be enhanced for effective engagement in policy processes, which also requires continued attention to partnerships with multiple stakeholders for advocacy at global and country levels.
  • IFAD's gender architecture. The evaluation recommends that Management conduct a dedicated, comprehensive review of IFAD's overall gender architecture to ensure that the organization has the required human resources and funds to achieve the desired results on the ground in borrowing countries. The review should include not just the Programme Management Department (PMD) but all other departments in the organization. The evaluation makes specific recommendations for the gender architecture, which may be seen in chapter VI of the report.
  • Tracking investments and budgets. It is recommended that Management undertake an analysis of spending on gender equality and women's empowerment in a regionally based sample of projects that have good gender equality results. This would allow it to determine the level of costs incurred in the past, which can serve as a guideline for future project designers. In addition, efforts should be made to indicate the amount of administrative budget being devoted annually to gender-related activities.
  • Training. While gender-specific training and awareness-raising is needed on key concepts, it is also recommended that a gender perspective be incorporated in training events organized by PMD on operational aspects and by the Human Resources Division on core competencies and in staff induction programmes. A gender training programme should be developed as soon as possible and implemented from the beginning of 2011.
  • Assessment of gender equality and women's empowerment in evaluations. It is recommended that IOE develop specific indicators and key questions for assessing gender equality and women's empowerment in country programme and project evaluations. 2 In addition, it should include a dedicated section in all evaluation reports in order to provide an overall account of performance on this indicator and to highlight the proximate causes of good or less good performance. The same recommendation is also applicable to the various components of IFAD's self-evaluation system.

1/ Box 5 in the main report contains the three corporate objectives for gender equality and women's empowerment.

2/ That is, in the context of project completion report validations and project performance assessments.

 

 

 

LANGUAGES: English

IFAD’s capacity to promote innovation and scaling up

junio 2010

Background. At its Ninety-fifth Session in December 2008, the Executive Board requested the Office of Evaluation (IEO) to undertake a corporate-level evaluation of IFAD's capacity to promote pro-poor innovation. The main objectives of the evaluation were to: (i) assess the performance of the Fund in promoting innovations that can be scaled up; and (ii) generate findings and recommendations for enhancing future IFAD activities in this area. The evaluation and IFAD Management's response thereto were discussed by the Evaluation Committee and the Executive Board at their respective sessions in April 2010.

Main findings. The evaluation found that, since the mid-1990s, concerted efforts had been made to incorporate innovation into the Fund's key policy and strategy documents. This is demonstrated by the inclusion of innovation, learning and scaling up as one of the six principles of engagement in the IFAD Strategic Framework for 2007-2010, and by the fact that, of the five organizations covered by the evaluation's benchmarking study, IFAD was the only one to have a definition and stand-alone strategy for innovation. Nonetheless, the evaluation found that insufficient resources and attention had been allocated for the purpose of translating policy and strategy pronouncements into concrete action. For instance, some recommendations generated by the first corporate-level evaluation on innovation, undertaken in 2000/2001, were never implemented, others only partially.

As far as results on the ground are concerned, the performance of IFAD-funded projects in promoting innovation has improved over time. This should not, however, give rise to complacency as almost 50 per cent of all projects evaluated in 2008 revealed only moderately satisfactory results in terms of innovation; very few were satisfactory; and even fewer were highly satisfactory.Two qualifications should be borne in mind when interpreting positive results: (i) evaluations have paid more attention to assessing the innovations introduced and piloted during project/programme execution, and much less to verifying whether successful innovations had been scaled up – which would have been essential to ensure they had a greater impact on rural poverty; and (ii) IEO's selection of projects for evaluation in any one year is largely done on a non-random basis, which may lead to a bias towards evaluating better-performing IFAD-funded interventions.

The evaluation revealed that the Fund has paid relatively more attention to (and found more success in) innovative solutions in social engineering and institutional arrangements (e.g. promoting participatory approaches to planning and resource allocation) rather than in agriculture. Although IFAD has provided a fair amount of grant resources for agricultural research to develop innovative, low-cost agricultural technologies that can lead to better productivity and incomes, the results of such research have not easily found their way into its investment projects. The relatively higher proportion of social engineering and institutional innovations may be attributed to the fact that, in the 1990s and the first part of the new millennium, IFAD generally focused more on social capital formation and empowerment than on agricultural activities or identifying economic opportunities for poor rural people. This gives rise to concern, first of all because of the number of poor people who are also food-insecure, and because of the Fund's mandate to enhance agricultural productivity and incomes through on-farm activities.

Scaling up is particularly weak in IFAD-funded operations. With IFAD's relatively limited resources, scaling up is of paramount importance. While the evaluation found examples of successfully scaled up innovations, these were largely the result of individual initiatives and commitment rather than of a systematic approach. Indeed, it was found that far too much is left to the initiative and entrepreneurial skills of individual IFAD country programme managers, who frequently act without clear incentives and/or accountability.

There are two other reasons why IFAD's performance in scaling up has been inadequate. First, as a general rule, little attention has been given to knowledge management, partnership-building, policy dialogue, etc., partly because IFAD has concentrated on designing investment projects/programmes (and, more recently, on direct supervision and implementation support) instead of allocating time, space and resources to non-lending activities. Second, in the past, the Fund's operating model (which did not allow it to perform supervision directly or to provide implementation support) and lack of a country presence constrained its ability to promote, replicate and scale up innovations. In any event, it is fair to state that, having recognized the importance of scaling up, IFAD is now making due efforts in that regard, including a scaling-up initiative in collaboration with the Brookings Institution. Moreover, the strengthening of the Fund's country presence, better quality assurance and quality enhancement systems, direct supervision and implementation support, and greater focus on non-lending activities are expected to collectively contribute to better results, not only in scaling up but also in identifying and piloting innovations.

As mentioned above, lesson learning and knowledge management are essential for documenting and sharing successful innovations with a broader audience. While it is recognized that IFAD introduced a dedicated knowledge management strategy in 2007 and that some useful initiatives have been made to share experiences over the last two years, knowledge management may be further strengthened to support innovations within IFAD-supported country programmes/projects. The recent decision to integrate stand-alone knowledge management and innovation strategies is a step in the right direction, given that the two processes are mutually reinforcing and essential for innovation management.

In tandem with loans, grants can play a useful role in selected phases of the innovation journey. IFAD has invested a fair amount of grant resources in developing pro-poor innovative solutions in agriculture and related areas. Nevertheless, although IFAD's grant programme could play a strategic role in supporting the innovation agenda, evaluation experience confirms that the links between grants and investment projects have not been adequately defined in country strategic opportunities programmes (COSOPs) and thus have been mostly weak in operations. It is to be noted, however, that IFAD's revised Grant Policy, approved in December 2009, emphasizes the strategic role of grants in innovation and, for the first time, also provides an opportunity to involve the private sector in research and pilot innovations for replication and scaling up through investment projects.

Perhaps the evaluation's most important finding is that IFAD's past efforts to promote innovation have been too broad-based. That is, rather than pursuing innovation in a focused manner, building on its comparative advantage, track record and specialization, the Fund has followed a "let a thousand flowers bloom" approach. One reason is that the Fund's innovation strategy does not require it to channel resources to selected strategic areas, or to chart the way to become an innovative organization.

The evaluation found that while IFAD's organizational capacity and culture to promote innovation have both improved since 2000, results have been rather poor and in any event start from a very low base. Among other things, there is a need for further development of human resource skills and competencies, for strengthening knowledge management systems, promoting a more open environment to foster creativity and for setting clear, focused directions for promoting innovation and scaling up. In this regard, IFAD's recent appointment of a Chief Development Strategist as the focal point for knowledge and innovation is a move in the right direction.

In continuation to the above, the evaluation notes that a number of organizational capabilities are required to support the innovation journey: systematic learning, structure and processes, culture, competencies (including staff skills and incentives), decision-making, and leadership and direction – capabilities that the 2000/2001 evaluation considered to be weak. The present evaluation found that the third, and probably the most important objective of the Initiative for Mainstreaming Innovation (IMI) – changing the organizational culture and practices to support innovation – has not, by and large, been met. The evaluation pointed out that, overall, IFAD's organizational capacity for innovation remains weak and has changed only marginally since the beginning of the decade. In other words, the Fund's strong strategic commitment to, and pronouncements on, innovation have not been adequately converted into action to become part of IFAD's corporate culture.

According to a staff survey conducted in 2009, despite having moved up five places IFAD still falls within the lowest quartile of the 43 organizations surveyed. More generally, the survey also showed that while staff considers IFAD to be relatively good at searching out or scouting for innovations, it is somewhat weak in prioritizing their promotion and scaling up. The survey also noted that while a number of key operational processes (e.g. quality enhancement and assurance) have been strengthened, the required human resource skills/incentives have not been put in place to promote innovation. Training opportunities are limited and it is not easy to obtain additional resources for advancing promising innovations. It was found that managers do not deal promptly with blockages that may hamper change (e.g. identifying additional resources for scaling up), and that IFAD's knowledge and information systems do not perform well with regard to deciding on innovations for scaling up. The Fund is also slow in taking new ideas through the system and, importantly, is not sufficiently open to ideas from a wide diversity of sources, including poor rural people. All these and other factors constrain the development of IFAD into a more effective, innovative organization.

Generally speaking, there is a disconnect between IFAD's strategic pronouncements and its (still) weak institutional capacity to promote pro-poor innovation on the ground. Undeniably, however, progress has been achieved and a number of initiatives have been taken (such as that on scaling up). But if IFAD is to become a more effective, agile and innovation-driven development organization in the twenty-first century and, even more importantly, if it aspires to becoming a leader in promoting pro-poor innovation, it will need to make a quantum leap, particularly in terms of organizational culture change and capabilities, and follow its "let a thousand flowers bloom" approach within a few strategic areas. However, the evaluation recognized that IFAD should also remain open to promoting country-/project-level innovations that respond to perceived challenges related to the agriculture and rural development of specific country circumstances. It should also focus more attention on the process of scaling up than it has in the past. Clearly, this will not be possible without sufficient allocations of resources.

Recommendations. The following recommendations aim to improve IFAD's capacity to move from its strong strategic commitments and pronouncements towards a more systematic approach, thereby enabling it to achieve better results on the ground in promoting pro-poor innovations for subsequent scaling up.

Define an innovation agenda for IFAD. As the Fund has followed a "let a thousand flowers bloom" approach to promoting innovation in the past, the evaluation recommended that an IFAD-wide innovation agenda, consisting of a few selected themes or domains, be developed at the corporate level. The themes or domains selected – "big bets" – should be in areas of the agriculture and rural sector where there is a proven need for innovative solutions and where the organization has (or can develop) a comparative advantage in promoting pro-poor innovations that can be scaled up. The selected "big bets" would be part of the Fund's innovation agenda, conceived as a corporate rolling programme over a period of, say, three years, and including specific objectives, activities, timelines, budgets, management and oversight arrangements, and monitoring and reporting requirements. The agenda would be approved by the President of the Fund, who would communicate it to IFAD staff and the Executive Board, with a commitment to provide annual reports on its implementation. However, the evaluation recognizes that the Fund also needs to remain open to promoting innovations at the country/project level that respond to perceived challenges related to the agriculture and rural development of specific country circumstances.

The evaluation recommended that more attention be paid to developing innovative solutions in agriculture technologies and related areas aimed at the economic empowerment of poor rural people. Examples of domains that IFAD might consider as "big bets" include health and weather insurance for poor rural people; rural finance products for dispersed populations; research on high-yielding crop varieties both in rainfed areas and for poor small-scale farmers; carbon projects; market access and value chain development (e.g. risk mitigation for the transition from subsistence to commercial farming); land titling (to include rural women); valorization of out-migration, and so on.

Treat scaling up as mission-critical. It is essential that concrete approaches and strategies for scaling up be articulated by the time of COSOP formulation and project design. The role and contribution in this regard of IFAD's direct supervision and implementation support, and of its country presence, should be clearly defined. IFAD should set corporate targets for scaling up and monitor and report on them annually. In this regard, it is also important to underline the accountability framework for scaling up, which would ensure that this critical phase in IFAD's innovation journey is given due attention and resources. Adequate resources and space need to be allocated to non-lending activities, which are essential for scaling up. Staff competencies should be further developed to ensure success in this area. Greater effort is needed in terms of exchanging experiences and lessons on innovation and scaling up within and across the five geographical regions in which IFAD works, both in the regions and among operational staff at headquarters.  IFAD's policy dialogue and partnership-building agenda at the country level should be also driven by the objective of scaling up, and thus should focus on a few topics that are part of the Fund's innovation agenda in the country concerned.

The evaluation found that the concepts of innovation and scaling up were lumped together as a unique block in IFAD and that the measurement and reporting systems, including IEO evaluations, do not always distinguish between them. It is recommended that, in future, innovation be assessed and reported on as a separate process from scaling up. However, given the intrinsic relationship and dynamic between the two concepts, assessing the achievements of IFAD's efforts in the entire innovation journey from scouting, to piloting, documenting and scaling up will be also essential.

Strengthen organizational capabilities and culture. The Fund will need to develop practical innovation management skills. Management of innovation is different from implementing proven approaches inasmuch as it requires entrepreneurship, a capacity to cope with greater uncertainties, adaptation, a range of skills and the ability to make difficult choices on emerging evidence. Thus IFAD should develop an innovation-specific competency model for individuals and teams, drawing on current best practice. Such a model would provide the basis for a comprehensive skills enhancement programme and development of relevant tools, processes and monitoring systems. Innovation management skills should be developed as personal, team and networked competencies and adopted both by the staff of IFAD and by its partners.

Staff recruitment should explicitly include innovation as a necessary characteristic. Incentive systems should be introduced that reward staff for promoting innovation and fostering the learning/sharing of good practices and experiences in innovation, i.e. the annual staff performance evaluation system should consider innovation in the assessment process. The evaluation found that there had been some improvement in operational processes in past years, but recommended that a study be undertaken to establish whether further adjustments were required in areas such as policy formulation, COSOP development, project design, supervision, evaluation systems (including monitoring and evaluation), non-lending activities, etc., to ensure that innovation is fully built into key phases of the country strategy and project life cycle.

All the recommendations generated by the present evaluation, including that of improving organizational capabilities and culture, will have a bearing on the Fund's administrative budget. If IFAD's overall innovation and scaling up efforts are to give the desired results in future, a detailed analysis will be first required to determine the financial implications involved and to ensure that adequate resources are allocated in a timely manner.

The Initiative for Mainstreaming Innovation. The evaluation recommended that funds left unused from the IMI should be used for initiating implementation of the three main recommendations contained in the evaluation, in particular, for changing the organizational culture and practices to support innovation, i.e. the IMI objective that has not been satisfactorily achieved. The evaluation also recommended that the IMI be extended and a future work programme developed, which could be funded either through IFAD's administrative budget or from supplementary funds mobilized for that purpose.

LANGUAGES: English

AfDB-IFAD joint evaluation on agriculture and rural development in Africa

abril 2010

The African Development Bank (AfDB) and the International Fund for Agricultural Development (IFAD) completed a joint evaluation on agriculture and rural development (ARD) in Africa, in December 2009.

On 16 July 2007 in Tunis, the two organizations signed a Memorandum of Understanding outlining the overall framework of cooperation within this evaluation, as well as governance and management arrangements, financial matters, procurement of services and disclosure of documents.
The main strategic goals of the joint evaluation were to:

  • contribute to enhancing the relevance of AfDB and IFAD policies and operations in ARD in Africa
  • improve their operational performance

This forward-looking joint evaluation did accordingly adopt the following objectives:

  • determine the relevance of IFAD and AfDB policies and operations in ARD in Africa, in the light of current and emerging ARD issues affecting the continent
  • assess the performance and impact of AfDB and IFAD policies and operations in ARD in Africa
  • evaluate the strategic partnerships between IFAD and AfDB and with other prominent ARD actors in the continent
  • understand the proximate causes of AfDB and IFAD relevance and performance in ARD, and develop recommendations to enhance their effectiveness and partnerships network

The evaluation benefited from the cooperation of three senior independent advisers who provided strategic and technical support to AfDB and IFAD, and assessed and confirmed the independence and quality of the joint evaluation.

Furthermore, specific communication activities provided information about key events and documents that were produced and shared with different stakeholders. The evaluation was completed by the end of 2009 upon delivery of the final joint evaluation report.

Download the full document "Towards purposeful partnerships in African agriculture" in English and French.

 

LANGUAGES: English

IFAD’s rural Finance Policy - Corporate Level Evaluation

julio 2007

Corporate-level evaluation

Rural finance. The rural poor – over 800 million people – require and use a variety of financial services. However, in most cases these services are inappropriate and provided on usury terms and not on conditions that are conducive to rural poverty reduction. Microfinance has evolved as an efficient and effective means to provide financial services to the poor. Some of the good practice standards of microfinance have been applied to rural areas. However, low population densities, remoteness of areas, and risks associated to lending to agricultural production increase the risks and impede the extent to which financial service providers extend outreach to rural areas.

IFAD's lending, grants and policy for rural finance. Over the last ten years (1996-2005) IFAD approved 194 projects with rural finance components for a total amount of US$3 billion. Of these total loan amounts US$822 million were earmarked for rural finance with an additional US$912 million raised from co-financiers. In addition, the International Fund for Agricultural Development (IFAD) approved US$21.5 million in grants for global, regional, and project level rural finance activities. In 2000, IFAD adopted its Rural Finance Policy (RFP) to underpin its commitment to the sector and provide policy guidance to its operations. The RFP was complemented by Decision Tools, a donor peer review organized by the Consultative Group to Assist the Poor (CGAP), and an action plan for rural finance.

Corporate level evaluation of rural finance. The rationale for this evaluation was the size of the rural finance portfolio, its performance observed by previous evaluations, and the fact that the RFP had been in place for five years. The objective of the evaluation was to assess the quality and effectiveness of the RFP, and in particular to determine whether it meets best practice standards, has been implemented and to which effect, and whether IFAD has the right resources, instruments and processes to meet the RFP objectives. The evaluation analyzed corporate policy and strategy documents, country strategies and projects of twenty countries, plus selected evaluative information for an additional eight countries. The coverage was global and included visits to ten countries in all regions assisted by IFAD.

Meeting best practice standards. The RFP sets out four challenges: sustainability and outreach, stakeholder participation, differentiated (or diversified) financial systems, and conducive policy and regulatory frameworks. On the positive side, the RFP sets standards for sustainability that exceed best practices for microfinance, and includes sector diversification and conducive regulatory frameworks, which are a sine qua non for sector development. However, the RFP does not meet best practice standards in a number of areas: for outreach it does not set clear targets (rapid expansion), for sector diversification it falls short of the meso-level intermediary institutions necessary to make the sector function, and for the regulatory framework it could have included a condition, namely to work at this level only when necessary, rather than making it an equal part of the whole. The challenge that stands out the most is that of stakeholder participation, which fits more with IFAD's modus operandi rather than being specific to rural finance. On the contrary, making it one of the four challenges diverted attention away from the more important concept of demand-driven services, which is fundamental to providing the rural poor with appropriate financial services that meet their needs.

Consistency with other IFAD policies. The RFP is consistent with other corporate policies of IFAD, which generally are supportive of the rural finance objectives set in the RFP. However, in a couple of areas improved consistency between policies would enhance IFAD's work in rural finance, in particular: (i) the rural enterprise development policy speaks of guarantee mechanisms, the soundness of which would have to be assessed in context of the financial sector; (ii) the gender dimension of rural finance is insufficiently represented in the RFP; and (iii) environmental sustainability and the link to rural finance is not recognized as an area relevant to IFAD.

Providing strategic guidance. The RFP combined information on rural finance and rural development together with IFAD's policy in such a way that description and prescription were mixed together. While informative, this limited the extent to which policy requirements stood out clearly, priorities were set, or norms were established. Some of these shortcomings were addressed through subsequent initiatives, such as the Decision Tools, the action plan for rural finance with followed the donor peer review, or the learning notes on rural finance. The regional strategies, generally, did not translate the corporate RFP into region-specific strategies, with the exception of one division where such strategy was articulated and resulted in better policy compliance and higher project design quality. Systematic portfolio reviews in this and another division contributed to strategic orientation and knowledge generation in those regions.

Project performance in meeting RFP standards. Projects are increasingly meeting the RFP challenges, although overall performance falls short of being satisfactory. The average performance rating for the five years before adopting the RFP was 2.7, which improved to 3.2 for the five years after the RFP was approved. This rating indicates that positive performance characteristics only marginally outweighed the negative. Some projects embodied RFP principles even five years before it was written, although the number of better performing projects increased after its adoption. In terms of each of the RFP challenges, the evaluation observed (i) on sustainability: 60 per cent of partner finance institutions achieved operational self-sufficiency, but only 24 per cent met the RFP's requirement of meeting financial self-sufficiency. This is comparable to the performance of financial institutions working with the United Nations Development Programme; (ii) greater diversification has been introduced in terms of types of financial intermediaries – gradually moving away from state-owned service providers – and of types of financial products – gradually introducing more off-farm credit services, but still falling short of a gamut of new services like insurance and remittance transfers; (iii) there was limited change in the frequency of working on policy or regulatory frameworks, but a slight improvement in the quality of interactions was noted; and (iv) there was no change in stakeholder participation. In terms of outreach, IFAD-supported partner finance institutions continued to reach the rural poor, although the percentage of women among clients of rural finance services was lower than of comparators working with the poor.

Explaining performance: process and experience. The continuous, but limited improvement in project performance can be explained by a number of factors. Some of them are systemic – related to the project cycle – others were generated from the project examples that the evaluation reviewed. The project cycle shows weaknesses that are not conducive to designing rural finance projects. The design process is longer than that of comparators and a technical appraisal of the project concept (for rural finance) often comes too late in the process. Project implementation is managed by units and cooperating institutions that do not have the technical expertise to manage the rural finance component with the level of competence required for this sector. Reporting lines mean that rural finance components can be subjected to political interests rather than professional considerations, introducing interest rate caps or preferential treatment for select clients, none of which ensures the rural poor receive professional and reliable rural financial services. In the earlier projects in the sample, rural finance projects lacked sector analysis and, instead provided supply-led directed credit: clients and products, including terms and conditions, were so over-defined that they were impossible to implement and of limited interest to the rural poor. More recent projects work, more and more, through financial service providers that are professional and provide an increasing range of financial products to their clientele. However, whatever gradual improvements in design have been achieved also mean projects get technically more sophisticated, which goes hand-in-hand with increasing challenges during implementation.

Corporate arrangements – matching RFP requirements? Best practice experience shows that governments are best at providing a conducive policy and regulatory environment and ensuring supervision of the sector and its actors. It is less efficient and effective in providing financial services. IFAD's lending to governments thus is inherently not in line with best practice and will always affect the way in its assistance to rural finance development is delivered. For instance, financial service providers that do not perform according to IFAD required standards (in terms of sustainability and/or outreach to the rural poor) cannot be called to correct their action, unless this is specified in the loan agreement (and subsidiary loan agreement) and government accepts to act together with IFAD. Related to this issue are limitations that arise from IFAD's instruments, typically a loan to government, which makes it difficult to meet more diverse financing needs of partner finance institutions, such as venture capital, private equity, or subordinate debt. Finally, the limits of working through cooperating institutions and without field presence apply to rural finance components as well, although technically specialized expertise would be needed in the field to have a significant impact on the performance and impact of IFAD-assisted partner finance institutions.

Human and financial resources. The RFP did not estimate whether human or financial resources would be sufficient to attain its objectives. Instead, it assumed the status quo of resource allocations. This meant that a considerable amount of IFAD lending (27 per cent of the total over the last 10 years) was handled by one full-time staff and country programme managers, who however, in general, would not be able to dedicate a commensurate amount of time to rural finance. This is reflected in IFAD's administrative budget (dedicated fully to rural finance), which is small compared to that of other institutions, such as the International Finance Corporation, the United States Agency for International Development, or the United Nations Development Programme (UNDP). In terms of qualifications, country programme managers need to cover a range of sectors, of which rural finance is but one, even if an increasingly demanding area from the technical point of view. IFAD has tried to address this issue through staff training, establishing a network among technical regional centers that will provide expertise during design and implementation, and through an internal thematic working group that serves to exchange views and distribute latest knowledge products. IFAD also supports an international database for performance data from financial service providers by aiming to have its partner finance institutions report to the database, which will not replace the need for technical in-house expertise to assess the information, but make it more accessible and comparable with international, regional and country standards. On this basis, country programme managers will know whether their partner finance institutions are performing according to expectations.

Conclusions. IFAD has led the work in rural finance in terms of approved loan allocations. It has made steps in the right direction with the RFP, which meets in some areas international best practice, while it can be improved (easily) in others to catch up with evolving best practice. The prescriptions of the RFP are increasingly met in project designs, although unevenly so, and challenges remain in project implementation. Many of the shortfalls can be explained by systemic weaknesses, in particular in the project cycle and in the founding agreement of IFAD that limits through whom and how IFAD provides loans and how it manages its assistance programme. Nonetheless, given the amount of resources and some of the key ingredients that IFAD has, it has the potential to become a much needed leader in assisting the development of rural finance, provided a commitment exists to making fundamental changes.

Recommendations. The evaluation provided two options to IFAD's Senior Management and Executive Board. The first option would entail gradual and less resource intensive change, but also mean only a gradual improvement in the programme. It would require clarifying the norms set out in the RFP, ensuring compliance with policy requirements through an integrate quality check, provision of technical appraisals of project proposals earlier in the process, and building greater in-house capacities. The second option would require more fundamental changes, but would promise IFAD could assume rightfully a leadership role in the sector. This option would entail deciding to become a leader and developing strategy for doing so, with region-specific strategies, allocating commensurate resources to the rural finance portfolio; developing and testing new instruments; and shortening the project cycle to become more efficient and relevant to the development of rural finance services.

 

LANGUAGES: English

IFAD’s Field Presence Pilot Programme

julio 2007

Corporate-level evaluation

Introduction

Background. In December 2003, at its eightieth session, the International Fund for Agricultural Development (IFAD's) Executive Board approved the three-year Field Presence Pilot Programme (FPPP). The main aim of the pilot programme was to enhance the effectiveness of IFAD operations by focusing on four interrelated dimensions: implementation support, policy dialogue, partnership-building and knowledge management. While approving the pilot, the Board requested the Office of Evaluation (OE) to evaluate the FPPP during its third year of implementation and present its results to the Board.

Evaluation objectives. Accordingly, the main objective of this evaluation was to: (i) assess the performance and impact of the FPPP in achieving IFAD's overall objectives; and (ii) generate findings and recommendations to help IFAD's management and Board decide on the pilot's future and lay the basis for the formulation of an IFAD country presence policy.

Evaluation methodology. The evaluation: (i) established a comparator group of countries without any form of IFAD field presence to gain a better appreciation of the results in countries with and without field presence; (ii) gave emphasis to the assessment of results achieved before and after the establishment of field presence by mainly obtaining the views of stakeholders; (iii) secured systematic feedback about the benefits of an actual or potential field presence within IFAD and at the country level; and (iv) carried out a comprehensive benchmarking study to identify good practice in country presence and learn from the experience of other development organizations.

While the focus was on the FPPP (including satellite countries1), the evaluation also examined the experience gained with: (i) two outposted Country Programme Managers (CPMs) in Panama and Peru; and (ii) proxy field presence2 arrangements. This facilitated the assessment of alternative field presence arrangements pursued by IFAD.

The evaluation faced three major challenges: (i) limited availability of self-evaluation data, incomplete accounting of FPPP costs and lack of any baseline data on the performance and outcome indicators adopted by the Board; (ii) the short and incomplete implementation period for most of FPPP initiatives3; and (iii) unrealistic expectations regarding the assessment and attribution of the pilot programme's impact on the rural poor4.

Country visits took place in 25 of the 35 countries included in the evaluation sample, some with and others without any form of field presence. In line with the IFAD Evaluation Policy, OE set up a Core Learning Partnership (CLP) and benefited from the inputs of two Senior Advisers (Dr Nafis Sadik and Professor Robert Picciotto), who provided advice on the design of the evaluation and reviewed all major evaluation deliverables.

Assessment of IFAD'S Field Presence experience

Design and management of the FPPP. The evaluation found that the focus of the FPPP on the four interrelated dimensions (implementation support, policy dialogue, partnership building and knowledge management) was appropriate for furthering the objectives of IFAD country programmes. However, the FPPP was critically under-funded, and the human resources allocated to the pilot were inadequate. The Consultation on the Sixth Replenishment of IFAD's Resources and the Executive Board were quite involved in the design of the FPPP, including establishing the pilot's objectives and resource allocation. For example, a group of IFAD member states prepared and circulated a "non-paper" outlining the objectives and design of the FPPP in 2002, and a specific Ad-hoc Working Group of the Executive Board on Field Presence5 was established in December 2002 to oversee its development and implementation. This involvement delineated a clear framework that the management followed in preparing the final proposal on the FPPP.

The FPPP design devoted inadequate attention to the interface between CPMs, IFAD's cooperating institutions and the field presence officers in general, but in particular with regard to their contribution to implementation support activities. This led to confusion on the ground among key partners about their respective roles and responsibilities. The self-assessment report on the FPPP issued by Programme Management Department (PMD) echoes these sentiments.

The Fund did not take advantage of the FPPP by systematically experimenting with alternative models to field presence except for the inclusion of satellite countries – a subregional approach to field presence connected to the establishment of field presence in a particular country. Most pilot countries followed the same model. It involved appointing a local staff member and arranging for office space. Nor did the Fund outpost any CPMs from Headquarters under the FPPP, although such experimentation was specifically envisaged by the pilot.

The implementation of the FPPP was also characterised by the inability to capture reliable cost data and the absence of a platform for systematic knowledge sharing among FPPP officers and CPMs, as well as inadequate reporting on performance indicators. Furthermore, no human or financial resources were specifically dedicated by IFAD for the management of the FPPP, so the pilot had to be implemented within existing management and staff capacities. As such, it can be said that pilot has not provided IFAD's management and Board with appropriate guidance for the formulation of an authoritative country presence policy at this stage.

Organizational aspects. Field presence officers draw on the administrative services of host organizations. This inevitably leads to some loss of IFAD identity and visibility. The majority of field presence officers are recruited and hosted by United Nations Development Programme (UNDP). The others are recruited and hosted by the Food and Agriculture Organization of the United Nations (FAO) and the World Food Programme (WFP), except for four field presence officers directly recruited as consultants by IFAD. No pilot is housed in the offices of international financial institutions6 – a lost opportunity for enhanced partnership with organizations that are especially well placed to help ‘upscale' IFAD-funded activities.

The Peru outposted CPM works from a privately rented office, whereas the Panama CPM is hosted in the UNDP office. Most proxy field presence officers work from private offices (or their homes). They are all recruited directly as IFAD consultants.

The effectiveness of the FPPP and proxy field presence officers has been constrained by limited delegation of authority. Nor has systematic coaching been provided.  Field presence officers are not authorized to represent IFAD formally or to take decisions on operational or financial matters. Partners at the country level are aware of this and tend to contact the headquarters directly. This contrasts with the orderly country relations pattern experienced in the two countries where IFAD has outposted CPMs. They enjoy the same status as their colleagues in Rome and this is recognized by country partners.

IFAD has attracted highly qualified field presence officers, although and understandably they are not equally competent to implement all four dimensions of the FPPP. Proxies have focused on one or two main areas of work (such as policy dialogue and donor coordination). On the other hand, outposted CPMs with delegated authority have been able to mobilise national expertise to pursue all four dimensions. In general, no systematic induction and training was provided at the outset of the FPPP or for proxy field presence officers. On-the-job training has been ad hoc. No special training provisions were made for outposted CPMs either. While field presence officers have recently been given access to the IFAD Intranet and provided with IFAD email accounts, they still do not have access to other key information systems such as the Project and Portfolio Management System (PPMS) and the Loans and Grants System (LGS).

Financial issues.  It is difficult to draw an accurate picture of the pilot's costs - and for the other models of field presence. Managers and staff did not use available accounting systems in a way that would have enabled proper tracking of FPPP costs. It appears that several country pilots have spent more than the anticipated amounts, largely as a result of the escalation in staff costs. According to a recent internal audit, the actual costs of the FPPP will be around US$4 million (on the basis of a full three year implementation period for all FPPP countries), rather than the US$3 million approved by the Board for the 15 country pilots7. The evaluation notes that individual pilots with an average of US$67 000 per year (with a maximum of US$80 000 per country each year) are severely under-resourced to handle the variety of tasks implied by the four FPPP dimensions.

A cost analysis conducted by the evaluation on the outposting of CPMs found that this is likely to involve substantial costs. Outposting a P4-level staff member could involve either a savings of around US$12 000 or additional costs to the Fund of around US$34 000 per year, depending on the duty station and the related post adjustment entitlement. For a P5-level staff member, savings could be around US$17 000 or additional costs around US$35 000. These estimates make no provision for a hazard allowance (an entitlement in some cases), for costs related to rental subsidies or for one-time costs of more than US$50 000 per staff for duty travel, family travel and household goods removal related to the outposting of headquarters staff. The investment costs in infrastructure required to make outposted staff operational also need to be factored in. On the other hand, savings can be generated by recruiting local administrative and secretarial staff to support the outposted CPMs. All in all, it would seem that a budget neutral outcome (and in some cases savings) might be achieved only if much of the operational work arising from the planned expansion in the programme of work8 is transferred to field offices in countries where professional salary scales are lower than at headquarters.

Assessment of Country with and without Field Presence

Key dimensions

Implementation Support
Policy Dialogue
Partnership Development
Knowledge Management Overall
Field Presence
 
5.2
4.5
4.5
4.1
4.6

Comparator countries

4.6
3.4
4.4
3.6
4.0

(1= lowest score, 6 = highest score)

Performance and results. As a group, the results related to the FPPP, proxy field presence and two outposted CPMs are better across the four inter-related dimensions, in relation to the cohort of countries in the comparator group without any form of IFAD field presence (see Table 1). Performance is even better in countries where field presence was established two or more years ago. However, these results must be interpreted with caution since the FPPP was directed to countries where borrowers' attitudes and capacities were relatively favourable.

While some examples of innovations were found in comparator group countries, the results in field presence countries are better in terms of replication and scaling up of innovations. In this regard, the role of the outposted CPM in Peru stands out in terms of the promotion of innovations, as confirmed by other OE evaluations. While such innovations would not have taken place without the incumbent's special skills, the delegation of responsibility to the field was a necessary condition of success.

Ratings of the four inter-related dimensions across the different types of Field Presence

As to the before and after scenarios, apart from knowledge management, the overall effectiveness of field presence countries is rated   between moderately high and high in the other three dimensions to which field presence is expected to contribute. All three field presence models appear to yield overall moderately high to high results in terms of improving IFAD activities in the four inter-related dimensions.

Within the FPPP countries, the best results are reportedly achieved in implementation support activities. Overall, the results achieved in knowledge management were not as good but this may be due to the lack (until recently) of an overall IFAD knowledge management strategy. On policy dialogue and donor coordination, it is revealing that the results are above the FPPP averages in countries such as Mozambique, Nicaragua, Tanzania and Uganda with emphasis on sector wide approach programmes in agriculture or rural development. The same can be said for partnership strengthening.

Although the outposting of CPMs emerges as the most successful model of IFAD field presence, it must be stressed that the results are based on a sample of only two countries where IFAD presently has outposted CPMs. The largest difference in performance between the outposted CPMs and the FPPP and proxy field presence officers is in the area of knowledge management.

The group of satellite countries covered by FPPP showed broadly the same results in implementation support, but lower overall effectiveness in policy dialogue, partnership strengthening and knowledge management. This is largely because the FPPP and therefore the satellites gave more priority to implementation support than to the remaining three dimensions. It also attests to the difficulties faced when engaging in policy dialogue activities outside the duty station country.

Most proxies cover only one or at most two of the four FPPP dimensions. The area of focus is driven largely by the most pressing operational needs. Proxy field presence has been effective especially in supporting policy dialogue, donor coordination activities, and less so implementation support. One problem is that some proxy field presence officers are hired on contracts of limited duration (e.g., on a retainer basis). This may lead to conflicts of interest, when the proxy field presence officers explore employment opportunities with institutions involved in IFAD operations.

Key findings from the benchmarking study

The benchmarking study generated a number of revealing findings. All comparator organisations considered field presence to be essential to their development effectiveness, especially in the framework of the evolving development architecture with emphasis on harmonisation, donor co-ordination and aid effectiveness. It has allowed them to pursue more systematically partnerships with like-minded development organisations at the country level. They emphasized that appropriate delegation of authority to country offices was crucial and that special training, induction and oversight arrangements are needed to secure full benefits from the field presence. Costs were merely one of the criteria considered by the comparator organisations in deciding to embark on decentralisation. Each organization has pursued alternative options for country presence in order to respond to different contexts, including the setting up of regional and subregional offices to complement the work of country offices. Finally, the study noted that decentralisation has consequences for the work of headquarters, and that ongoing institutional reform processes must take full account of decentralization of operations.

Conclusions

Overall, the evaluation concludes that the field presence model tested by the FPPP has had positive results. The same can be said of proxies, and of the CPM outposting model, although the size of the sample is small. The benchmarking study confirms that a permanent field presence is widely viewed by other development organizations as central to their effectiveness. But significant resources need to be invested for field presence to be effective. In sum, the central question for IFAD is not about the rationale of a field presence, but rather about the form of country presence most appropriate for the Fund and the countries it serves.

The overall effectiveness of IFAD measured along the four dimensions of implementation support, policy dialogue, partnership development and knowledge management has been greater in countries with field presence than in countries without. The FPPP has made IFAD more visible and effective and has allowed better and more consistent follow-up. This has had wholesome effects on the quality of country programmes and projects. The results would have been better and more solidly documented had the shortcomings in the design and implementation of the pilot been recognised and acted upon on a timely basis – particularly with respect to funding, delegation of authority, legal, logistical and training arrangements.

The FPPP had an ambitious design and was under-funding. This can be seen as a reflection of the compromise that had to be reached in order to garner the acceptability of Board members, several of whom strongly favoured IFAD field presence, whereas others did not.

Based on a small sample, the outposting of CPMs with full delegation of authority to advance IFAD's objectives at the country level emerges as a highly effective option. The evaluation made an initial attempt to determine the cost of outposting CPMs, which reveals that establishing this type (but also any other less effective type) of country presence for IFAD is not likely to be cost neutral and involves significant rethinking of the role, organizational structure and functioning of the institution as a whole, comprising of both outposted and headquarters staff.

The experimentation with the satellite country approach has also proven positive, particularly as far as implementation support activities are concerned. It is an interesting option from a cost perspective. Finally, the proxy field presence approach has been effective, when focused on one or two areas such as policy dialogue and/or aid coordination. 

In sum, in spite of the limitations of pilot design and implementation and the challenges involved in assessing FPPP results (see paragraph 5), the evaluation is able to conclude that an enhanced field presence would make a significant contribution to IFAD's development effectiveness in all four dimensions. However, the most promising approach to decentralization based, admittedly, on a very small sample (CPM outposting) was not tested under the pilot. Nor were the other options tested systematically in diverse country contexts and in conjunction with appropriate delegation of authority and suitable training and induction support. Moreover, it is not possible to conclude without access to better cost data that a budget neutral outcome can be guaranteed. In fact, available evidence (amply confirmed by the benchmarking survey) suggests that the full benefits of decentralization may require substantial incremental budget outlays.

Due to all these considerations, the pilot must be considered a missed opportunity even though enough reliable evidence now exists to confirm the need for an expanded field presence program in order to allow IFAD to play its distinctive role in a relevant, effective and efficient manner within a development environment in rapid transformation.

Recommendations

The evaluation has two specific recommendations: (i) embark on an expanded Country Presence Programme; and (ii) work towards the formulation of a Country Presence Policy by 2010.

Embark on an expanded Country Presence Programme

Given that the FPPP did not succeed in providing a conclusive indication of the most effective form of field presence for IFAD, the evaluation concludes that it is premature to propose a mainstreaming of the initiative. Instead, the FPPP should be transformed in a new programme - the IFAD Country Presence Programme (CPP)9 - that would aim at consolidating the evidence around the positive results as well as determining the most cost effective form of country presence that IFAD should adopt in diverse country contexts. The CPP would consist of two distinct tracks:

    • Continue implementation of existing FPPP country initiatives; and
    • Expand the programme to allow systematic experimentation with alternative country presence models.

First, the evaluation recommends the continued implementation, under the CPP of all FPPP country initiatives, whether they were due to complete their three year implementation by the end of 2007 or not.

In parallel, the evaluation recommends an expansion of the CPP to allow for experiments that were not undertaken in the first phase of the FPPP, namely with outposting of CPMs and establishing subregional offices. Specifically, it is suggested that the FPPP be expanded to cover an adequate number of countries in all IFAD regions, including 2 to 3 subregional offices located in different regions. The expansion should entail the outposting of around 10 CPMs in FPPP and other countries, with preference given to large and active country programmes.

For all countries in the CPP, a reassessment of budget allocations should be made, to ensure that each country pilot has access to the funding needed to achieve the intended objectives. Proper use of IFAD's accounting system should be ensured so that accurate monitoring of costs related to the CPP is carried out. Finally, a platform for sharing experiences across the concerned CPMs and field presence officers should be set up with full access to the PPMS and LGS provided for all country offices

Furthermore, it is imperative that all monitoring, evaluation and reporting measures be put in place to ensure the evaluability of the extension phase in order to avoid the shortcoming of the first pilot phase. The need to collect baseline data in all countries under the CPP is especially critical.

The Management should be comprehensively engaged in country presence issues, for example in ensuring that adequate delegation of authority is provided to field presence officers and that appropriate systems are in place for training, induction, coaching and oversight of outposted personnel. The delegation of authority to field presence officers from headquarters should be articulated with clarity and realignment of responsibilities between field and headquarters staff should be specified to minimize duplication and enhance accountability. Where field presence officers have consultancy contracts that have performed competently, IFAD should devise specific legal instruments that allow their contracting as local staff.

A cross-departmental committee should be established to facilitate organizational learning and discussion of cross-cutting issues emerging from the CPP. Furthermore, IFAD should consider taking the lead in forming a committee of Rome-based UN agencies on country presence issues, as a forum for exchanging experience and good practices.

Development of IFAD's Country Presence policy after 2010

The evaluation concludes that it is premature for IFAD to formulate its country presence policy, given the limited experience both in terms of the implementation duration and diversity of country presence models experimented under the FPPP. Therefore, it is recommended that a self-assessment of the CPP (including the FPPP) be undertaken by IFAD Management in 2010. This would serve as the basis for the development of IFAD's comprehensive country presence policy to be submitted for approval to the Executive Board following the final assessment in 2010.


1/Satellite countries are those neighbouring countries covered by the field presence officer, in addition to his/her country of residence in one of the 15 FPPP countries.

2/ As with the two outposted CPMs, proxy field presence countries are outside the FPPP. Under proxy field presence, IFAD normally recruits a consultant locally who can undertake a range of activities in support of the IFAD country programme, such as attending donor co-ordination meetings.

3/Ten of the FPPPs only started in the last quarter of 2005 or in 2006.

4/The limitations were addressed in a number of ways, including the adoption in the evaluation of a comparator group of countries and assessing the before and after field presence situations, undertaking a benchmarking study, and visiting 25 countries to collect information from concerned partners.

5/ This group is still operational and is composed of 9 members of the Executive Board.

6/One reason might be that generally the rental costs of space in such institutions was found to be higher than under the current FPPP host organisations.

7/In their initial proposal to the Board in September 2003, the management estimated the costs of the 3 year pilot to be US$3.6 million. However, based on discussions with the Board, the budget submission of the management was reduced to US$3 million when the final FPPP proposal was presented for approval by the Board in December 2003.

8/See section on "Programme of Work 2007-2009" (paragraphs 57-59) in the document IFAD's Contribution to Reaching the Millennium Development Goals: Report of the Consultation on the Seventh Replenishment of IFAD's Resources (2007-2009), which articulates the background and magnitude of the annual increases in the Fund's programme of work. The target is to achieve a US$2 billion work programme for the Seventh Replenishment period.

9/ It is proposed to replace the term ‘field' with ‘country', given that the word field is normally associated with geographic areas where IFAD-funded projects are implemented. This should not however preclude the possibility for IFAD to establish country presence outside the capital city, should this be considered appropriate in any particular case.

 

 

LANGUAGES: English

Evaluation of IFAD’s regional strategy in Asia and the Pacific (EVEREST)

julio 2006

The International Fund for Agricultural Development (IFAD) adopted its Regional Strategy in Asia and the Pacific in March 2002, which was the first time that the Fund coherently developed and formally adopted a Regional Strategy. During its session in December 2004, IFAD's Executive Board decided that the Office of Evaluation (OE) should undertake, in 2005/06, the Evaluation of IFAD's Regional Strategy in Asia and the Pacific (EVEREST), which would represent the first such type of evaluation conducted by the Fund.

EVEREST, which was conducted under the overall provisions of the IFAD evaluation policy, is expected to provide the key building blocks for the preparation of the new Regional Strategy in Asia and the Pacific that IFAD plans to present to the Board in 2007.

Evaluation objectives and methodology

The main objectives of the evaluation were to: (a) assess IFAD's performance and impact in the region during 1996-2005, with particular reference to the 2002 Regional Strategy in Asia and the Pacific; and (b) develop a series of findings and recommendations that would serve as building blocks for formulating the next IFAD Regional Strategy in Asia and the Pacific. In order to achieve its main objectives, the EVEREST defined the following three key questions that the evaluation would address: (a) What has been the performance and impact of IFAD in the region? (b)Has IFAD made the appropriate strategic choices in the region, including its operational emphases and the choice of niche, instruments, target group, partners, interventions and implementation modalities? and (c) How have IFAD business and management processes influenced performance and impact and the strategic objectives of IFAD in the region?

For each of these questions, more detailed questions were formulated and information sources identified by the evaluation team at the outset of the process. In order to facilitate the work of the evaluation team and to ensure that various stakeholders were able to gain an overview of and comment on the main issues being covered by the evaluation, OE prepared a detailed evaluation framework that captured the main questions the evaluation would cover and linked them to each of the three overarching questions listed above.

Evaluation approach and process

The evaluation approach paper was discussed in two subregional EVEREST launching workshops, one held in Islamabad, and the other in Bangkok, in July 2005. The evaluation benefited from the views and experience of staff in the Asia and the Pacific Division (PI) throughout the process. Discussions were also held with other IFAD staff, including the three Assistant Presidents, various directors in the Programme Management Department, the director and staff in the Office of Human Resources and staff in the Strategic Planning and Budget Division.

A two-person senior advisory panel was constituted at the outset of EVEREST, including Professor M. S. Swaminathan and Dr Robert Picciotto. Their role was to provide comments and guidance on the overall evaluation objectives, methodology and process, as well as to review key deliverables generated throughout EVEREST. Their final report is included in this document (see Appendix 4).

The evaluation noted that the 2002 Regional Strategy formalized a strategy that IFAD had already been following in the region for a number of years before 2002. Given this, it was decided that the evaluation period for the EVEREST analysis would run from 1996 to 2005.

It is important to note that, as a contribution to EVEREST, PI undertook a comprehensive "Self-Assessment of PI Performance in Implementing the Regional Strategy in Asia and the Pacific". The objective of this self-assessment for IFAD was to take stock of the Fund's overall experiences in implementing the Regional Strategy. The self-assessment served as a useful input in EVEREST, inter alia, in collecting and analysing data on the alignment of project design with the strategic directions of the Regional Strategy and on cofinancing and counterpart funding.

The five evaluation phases. The evaluation was organized in five phases. First, an inception phase served to fine-tune the methodology, finalize the selection of countries to be covered, define in detail the workplan and develop the instruments for data collection. Second, a desk review was undertaken in the 12 countries 1 included in EVEREST. More than 25 existing OE evaluation reports were reviewed, and extensive use was made also of the reports prepared during the External Review of the Results and Impact of IFAD Operations (2002) and the Independent External Evaluation of IFAD (2004/05). In addition to the aforementioned, all the IFAD country opportunities strategic papers produced during the evaluation period were reviewed, including the subregional strategies for Central Asia and the Pacific Islands, various country assistance strategies and evaluations of other international organizations, several government documents and so on.

Third, the evaluation team undertook country visits to China, India, Pakistan and the Philippines. In addition, the team visited the headquarters of the ADB twice and met with staff in the regional offices of the Food and Agriculture Organization of the United Nations (FAO) and the United Nations Office for Project Services. In all countries visited, the evaluation team also met with senior government officials, representatives of other international financial institutions (IFIs), selected UN organizations represented at the country level (FAO, the United Nations Development Programme (UNDP) and the World Food Programme) and bilateral aid organizations, as well as academic and research institutions, NGOs, IFAD consultants and project staff.

Fourth, a meeting was held on 10 May 2006 at the M. S. Swaminathan Foundation in Chennai to discuss the draft evaluation report with representatives of IFAD and selected countries in the region. Fifth, OE co-organized with ADB a regional EVEREST workshop in Manila on 28-29 June. This workshop focused on learning from EVEREST with a view to enhancing the effectiveness of IFAD's future strategy in Asia and the Pacific and to pave the way for the production of the Agreement at Completion Point for the evaluation.

It is important to highlight that the Board approved 68 loan projects in the region during the evaluation period, and the analysis conducted for the country working papersand cross-country analysisextends mainly to 22 of these, plus eight approved before 1996 for which OE evaluations have been completed since 2000. Of the 68 loan projects, 17 are not yet signed or effective, only one of which is included in EVEREST. This means that the evaluation includes 41% of the effective projects approved during the evaluation period. The evaluation also reviewed more than 25 (mainly large) technical assistance grants.

Rural poverty and agriculture in the region4

Poverty in Asia and the Pacific. The Asia and the Pacific Region's track record in reducing income poverty in the 1990s was impressive. Poverty incidence, using a dollar-a-day standard, declined by about 30% over the decade. In 1990, about 32% of people in the region were living below the poverty line. By 2000, this proportion had come down to 22%. Rural poverty in the region declined from 39% to 28%. The absolute number of the poor also declined, by around 180 million, from 900 million in 1990 to 720 million by the end of the decade. This was achieved in spite of a major financial crisis in 1997 that depressed regional growth and increased poverty incidence.

Given continued growth, the Asia and the Pacific Region appears to be on track to meet the Millennium Development Goal for income poverty. However, several countries are likely to be left behind, and others that are likely to meet the target will still have to deal with pockets of poverty. In addition, worsening income distribution patterns are accompanying the economic growth, and the non-income dimensions of poverty have seen much less progress in the Asia and the Pacific Region. In particular, South Asia's progress in reducing the proportion of undernourished children, expanding immunization coverage, increasing the number of births attended by skilled health staff, and combating HIV/AIDS has also been slow. These problems reflect a parallel lag in the progress made towards the gender equality and empowerment goal within this subregion.

Agriculture in Asia and the Pacific. Although the agriculture and rural sectors in the region continue to grow, the sectors are declining in relative importance in Asia, both in terms of their contribution to GDP and their share of the labour force. Urbanization is increasing, and farm households are diversifying their sources of income beyond agriculture. This relative decline of agriculture is inevitable in countries that experience economic growth, which has been widespread in the region. Nevertheless, more than half the economically active population is still involved in agriculture in the region, and agricultural employment is especially important among the livelihoods of the rural poor. In addition, agriculture remains a major sector in nearly all Asian economies even before adding the value generated through downstream processing. Agriculture also serves as a buffer and a safety net by providing employment in the face of large economic shocks, such as the financial crisis in 1997-98.

The importance of agriculture to the macroeconomy, the labour force and the rural poor suggests that investment in agriculture should continue. Indeed, studies in China and India have shown that, dollar for dollar, agricultural research has historically been one of the most effective means of government spending to reduce poverty. Other research has shown that agricultural growth in Asia is typically more pro-poor than growth in other sectors. Unfortunately, public funding for agriculture is declining in many countries, although there are notable exceptions, such as China. Furthermore, much of the funding for agriculture often goes to subsidies that are not targeted on the rural poor, instead of being allocated for activities that increase productivity.

In sum, the Asia and the Pacific Region is home to many of the world's most dynamic economies, but it is also home to a majority of the world's poor. Domestic agriculture provides the bulk of food for the poor in both rural and urban areas, and it is a key provider of jobs in rural areas, where the majority of the poor live. In order to promote agricultural development and growth, deeper attention and more resources will need to be invested in the agriculture sector by governments and the international community so as to ensure that rural poverty does indeed become history.

The role and experiences of other IFIs and UN organizations in the region. Both ADB and the World Bank have increased their attention on and their investments in agriculture and rural development in 2005 and 2006. ADB issued a new poverty reduction strategy at the end of 2004, while the World Bank developed a global rural development strategy in 2002, as well as two subregional strategies for South Asia and East Asia-Pacific shortly before that.

Numerous evaluations examined by the EVEREST team had been prepared by the respective independent evaluation offices of ADB and the World Bank. ADB notes that high performance and impact are difficult to achieve in the agriculture and rural sectors relative to other sectors for a variety of reasons, including complexity in project design, overoptimistic project objectives and inadequate beneficiary participation. The Annual Evaluation Review of ADB (2005) mentions that there is a low probability of success in agriculture and natural resource projects. In comparison, however, according to the 2004 Annual Review of Development Effectiveness of the World Bank, close to 85% of the World Bank's operations globally showed satisfactory outcomes in the agriculture and rural sectors from 2000 to 2004, up from around 75% between 1995 and 1999, with operations in all sectors in Asia and the Pacific performing better than operations in other regions. Interestingly, according to the World Bank, the general perception that rural projects are riskier is not supported by the Bank's quantitative data analysis.

FAO has a regional strategic framework for 2004-06 for Asia and the Pacific, which puts food security and small farmers at the centre of its regional priorities. FAO does not produce a report providing an overview of the organization's development effectiveness and has not undertaken any evaluation of its Regional Strategy in Asia and the Pacific. The UNDP has a Regional Cooperation Framework (2002-06) for Asia and the Pacific, in which agriculture or rural development is not a thematic priority. However, the UNDP does focus on promoting gender equality and tackling HIV/AIDS. Although it produces a development effectiveness report, given the nature of its mandate and operations, comparisons with the UNDP may not prove to be of much significance to IFAD.

IFAD 's strategy in Asia and the Pacific

Strategy preparation process, coverage and format. The process leading up to the development of the strategy was not clearly defined, nor did the preparation include the active participation of persons from all subregions and countries across Asia and the Pacific. The strategy document did not clearly articulate its purpose and intended audience. Moreover, in spite of the dissemination efforts, important partners in the region at the country level and other key development organizations, such as ADB, conveyed to the EVEREST team that they were not familiar with the IFAD Regional Strategy.

However, feedback from IFAD staff, the Executive Directors and others highlighted the usefulness of the document as a reference point for the development of country strategies and operations, as well as for general communication purposes. That said, the evaluative content of the document is weak, as it does not refer adequately to operational experiences or the extensive range of lessons learned generated through OE evaluations. Moreover, it does not include a results framework or a timeline and lacks consideration of the issue of retrofitting ongoing country strategies and operations so as to bring them into conformity with the Regional Strategy. A full costing of the strategy, including human resources and organizational issues, was left out entirely.

Nor does the strategy articulate how it would engage donors and non-borrowing country partners in the region, and it does not sufficiently consider the diversity of the subregions in Asia and the Pacific. In sum, the Regional Strategy does not serve fully as an effective management tool.

Relevance of the strategic directions of the regional strategy. Three of the five strategic directions covered in the document are considered relevant both at the time of the strategy's development and in the context of today's development challenges and opportunities for rural poverty reduction in the region. The development of indigenous peoples, enhancing the capabilities of women and building coalitions of the poor are appropriate as IFAD's strategic directions in Asia and the Pacific. However, in light of IFAD's overarching mandate to promote replicable innovations, its experience and knowledge, and the role of other development organizations, the EVEREST team believes that the development of less favoured areas and enhancing peace for poverty reduction should be reconsidered as strategic directions. The following sections underline some of the key reasons for the findings mentioned in this paragraph.

It is known that large numbers of indigenous peoples are living in the region. These peoples are among the most impoverished and disadvantaged parts of society, and they require special attention and assistance. IFAD has many achievements to highlight in supporting indigenous peoples in Asia and the Pacific. However, while focusing on indigenous peoples, IFAD must also find ways to support other rural poor people living in the same project and programme areas. According to the EVEREST team, this strategic direction, along with the above qualifications, was and remains pertinent for IFAD in the region. This is so because governments and donors alike expect IFAD, given its accumulated experiences, successes and comparative advantage, to take the lead in addressing rural poverty among indigenous peoples. This would also be consistent with and contribute to furthering IFAD's commitment in relation to the Paris Declaration on Aid Effectiveness, which states that "Donors commit to make full use of their respective comparative advantage at sector or country level by delegating, where appropriate, authority to lead donors for the execution of programme, activities and tasks".

EVEREST finds that the emphasis devoted in the Regional Strategy to enhancing the livelihoods of rural women was equally appropriate. For example, women in the region have suffered and, generally, continue to suffer from heavy workloads both at home and in support of family incomes, less nutritional intake than other family members, limited access to assets such as land and housing, domestic violence and lower levels of education. They continue also to have a limited role relative to men in key decision-making processes, especially, but not only in formal institutions and political bodies at all levels. That said, the EVEREST team notes that more attention should have been devoted to gender roles and relations, that is, the consequences produced by the attention on women's development in terms of the overall social and cultural fabric need greater analysis and consideration.

Building coalitions of the poor, including the mobilization and participation of poor people in projects and programmes, is at the centre of IFAD's strategy and operations in the region. It is an underlying condition for achieving results and sustainability. This is so within today's context as much as it was so in 2002 at the time when the Regional Strategy was adopted. The Fund has generally been quite successful in social mobilization, promoting participation and contributing to building grass-roots institutions throughout the region. Many good examples can be cited. At the same time, there are three specific points that need consideration. First, various evaluations have found that greater attention needs to be paid to understanding the types and capabilities of existing institutions at the local level. Second, while emphasis on social mobilization, people's participation and training and on grass-roots institution-building is important in empowering the rural poor, the EVEREST team found that, in many instances, this objective needs to be better integrated with agricultural and non-agricultural productive activities that can contribute to raising incomes. Third, various evaluations call attention to the fact that governments are not always the most appropriate channel for promoting participation, especially in the remote geographical areas covered by IFAD operations. In addition to the above, EVEREST notes that building coalitions of the rural poor has been difficult in some cases, partly because processes to link groups or form associations are not sufficiently well defined in the project designs.

Focusing on the development of less favoured areas, another strategic direction, is a topic that requires careful reflection. Selecting less favoured areas as a precondition to IFAD's assistance could compromise what EVEREST sees as IFAD's central mission: the promotion of innovative approaches to rural poverty reduction that can be replicated and scaled up by others. Less favoured and, especially, remote and marginalized areas are not always a priority for governments and donors, and finding partner institutions that have a track record in innovations and in working in such geographical locations is therefore challenging. Furthermore, less favoured areas are often characterized by poor soils, a harsh climate, weak institutions and limited marketing opportunities. As a result, they may have limited agricultural development potential. Thus, they may represent major risks and limited rewards for the IFAD operations focused on agricultural innovation. On the other hand, it is undeniable that some of the poorest people, including many indigenous people, are living in the less favoured areas, and one might argue that it is IFAD's mandate to contribute to livelihood development among such people not only as a solo agent, but also by demonstrating to other donors that there are ways to assist the poorest even if they are living in the harsh conditions of less favoured areas. Such a policy stance would help in the reduction of regional inequalities. In turn, more equitable distribution of the benefits of growth may yield a reduction in conflicts. On balance, according to EVEREST, investment in less favoured areas should have a place in IFAD's lending priorities, but an exclusive focus on such areas should not determine where IFAD should work within a given country. Rather, IFAD should channel its assistance to those parts of a country where the operating environment is conducive to promoting innovative approaches to agricultural development and rural poverty reduction with a view to their replication and scaling up by others.

The fifth strategic thrust, enhancing peace for poverty reduction, is innovative and bold, given that numerous countries were faced with serious problems of civil unrest and that these problems had not been factored into the strategies of other donors. However, it was not possible for the EVEREST team to assess IFAD's effectiveness in pursuing this policy direction since few projects funded by IFAD in the region focused on peace building. Thus, the self-assessment report prepared by PI notes that only 3% of lending following the adoption of the Regional Strategy was assigned to enhancing peace for poverty reduction. This limited involvement can be traced to a host of constraints on IFAD's capacity to operate in insecure environments. Recently, the development community has begun to give higher priority to peace building activities, including greater support for fragile states, so that IFAD may be called upon to play a pioneering role in peace building, e.g., in community-based projects geared to the reintegration of former combatants into the rural economy. Hence, the question arises whether IFAD should gear up with adequate capacity to work in conflict-ridden and conflict-prone areas, and whether, in this context, it should begin nurturing alliances with development and humanitarian organizations better equipped and specifically mandated to focus on this important theme. Until this policy shift takes place, a key message of the Chennai workshop ("IFAD cannot do everything and needs to be selective in addressing rural poverty") will remain relevant.

Other issues related to strategic directions. The importance attributed to microfinance as an overarching feature in the Regional Strategy is highly relevant, and the results have been positive even though there is room for improvement in targeting the poor and building the sustainability of grass-roots institutions involved in microfinance. However, the strategy did not consider adequately a number of other development issues. First, the centrality of promoting innovations and having demonstration effects for replication and for purposes of scaling up through IFAD-funded operations was not given due treatment. This issue is important throughout the region, but even more so in the larger economies, where IFAD financial resources are not the only attraction available to borrowing countries.

In addition, the strategy, surprisingly, paid limited attention to rural infrastructure and the livestock subsector, which are important subsectors in the region. The issue of how IFAD might complement the work of the main institutions combating HIV/AIDS was not covered. Likewise, in spite of the good experiences in various countries, the Regional Strategy paid limited attention to decentralization and failed to address corruption seriously both at the time of the development of the strategy and following the approval of IFAD's anticorruption policy. The role of the private sector, migration and remittances, their effects on the rural economy and ways in which the Fund could cost-effectively engage in the Pacific Islands were also not covered.

Alignment of the country strategic opportunities papers and operations with the Regional Strategy. By and large, the country strategies and projects formulated by IFAD are in line with the main elements of the Regional Strategy. Depending on the country strategy or project objectives, there is a stronger focus on some strategic directions than others. However, as mentioned elsewhere above, only one country strategy and very few operations have focused on enhancing peace for poverty reduction. Loan resources have been, by and large, allocated in line with the main priorities of the strategy, which are separate from the area of enhancing peace for poverty reduction.

Assessment of resource allocation and results

Resource allocation. Since its establishment, IFAD has provided around USD 2.8 billion to 21 countries in the region in the form of loans. During the evaluation period, IFAD loans in the region stood at around USD 1.2 billion through 68 projects and programmes. This represents 41% of the commitments and 37% of the projects approved for the region since IFAD was established in 1978. Total project and programme costs during the evaluation period were nearly USD 3 billion, and all lending since 1998 has been on highly concessional terms.

According to the regional lending allocations established by IFAD in 1994, the share of Asia and the Pacific Region was set at about 31% of the total for IFAD. Before the introduction of the performance-based allocation system (PBAS), PI worked out country-level allocations largely on the basis of precedent. Seen in conjunction with the current regional shares, these outcomes suggest that, prior to the introduction of the PBAS, IFAD's actual lending in the Asia and the Pacific Region, as also in other regions, was driven by legacy rather than strategy, except that the addition of new borrowers diluted the share of old ones. These are some of the reasons for the introduction of the PBAS, which is expected to establish a more systematic and transparent resource allocation process.

The addition of new borrowers (e.g., Cambodia, Laos and Vietnam) tilted IFAD lending towards East and South-East Asia and away from South Asia. Perhaps the biggest loss was experienced by the Pacific Islands, however; these went from a 2.1% share of the PI portfolio to zero. The introduction of the PBAS is more likely to safeguard the interests of the smallest states if the minimum lending stipulated in the PBAS (USD 1 million per country) is followed. Moreover, because of the weight given to (rural) population in the PBAS, country allocations will shift in favour of South Asia.

EVEREST also notes that, within many of the countries in the region, IFAD support and resources are dispersed in various geographical areas, which, inter alia, militates against promoting synergies across the programme at large and limits the Fund's ability to build deep knowledge on specific rural poverty issues. With regard to grants, the division's current portfolio includes 47 grants, worth USD 17.3 million, that are divided into large and small grants and also regional and country-specific grants. The reported grant portfolio in the region is inexplicably only about 1.5% of the loan portfolio (excluding the programme development financing facility), which seriously limits the division's ability to, inter alia, promote innovation, undertake knowledge management and conduct research through grant-funded initiatives. Finally, it is noted that there have been increases in grant allocations recently for partnership building, policy dialogue, and impact assessment.

Cofinancing and counterpart funding. There has been a drastic reduction in cofinancing in the past four or five years. Cofinancing as a percentage of IFAD lending varied between 65% and 68% in 2001-02 and only 3%-4% in 2003-04. There are various reasons for this, including the organizational decentralization of donor agencies, the increase in the use of other instruments to channel aid (such as sector-wide approaches and budget support) and limited knowledge among potential cofinanciers at the country level of IFAD's programmes and experiences. Furthermore, there is little management guidance, and the incentives for mobilizing cofinancing are few. The lack of continuity in cofinancing partnerships is also a concern: 35 different partners have cofinanced IFAD projects in Asia since 1978, but only 11 of these were actively cofinancing with IFAD in 2002-05. New cofinancing with ADB and the World Bank has been zero since 2002 (see Main Report, Tables 10 and 11). The level of counterpart funding has also fallen markedly, from 45% of total cost between 1978 and 1995 to 19% between 2002 and 2005.

Project portfolio performance (see Main Report, Table 3). The ratings of the 14 projects examined in EVEREST show more or less consistently better results than those reported for IFAD as a whole in the Annual Report on the Results and Impact of IFAD Operations (ARRI) during 2002-04. In 100% of the sample, project objectives are highly or substantially relevant in Asia and the Pacific. In terms of effectiveness, projects in the Asia and the Pacific Region achieved far better results than the projects considered in the ARRI. Some reasons for this include the following: (a) IFAD operations in the region benefit from sounder institutions, well-trained human resources and a relatively favourable policy environment prevalent generally in the region. This is reflected in the high levels of economic growth and relatively good development performance of the region. In fact, the outcomes for projects funded by the World Bank are better in Asia and the Pacific than they are in other regions, and they are above Bank-wide averages. (b) The extensive involvement and quality of NGOs and community-based organizations, as well as the enhanced participation of women in development activities, help explain the higher effectiveness. (c) The project implementation indicators are better relative to those in other IFAD regions, for example, in terms of disbursements and time taken from loan approval to effectiveness. As far as efficiency is concerned, the ratings of projects in the Asia and the Pacific Region are a little better than the ARRI figures. The measurement of efficiency is not facilitated by the unsystematic data collection and infrequent estimates of cost-benefit indicators in IFAD-funded projects.

Impact on rural poverty (see Main Report, Table 4). In terms of the impact of projects on rural poverty, it is notable that the EVEREST sample yields higher ratings than the ARRI sample in all but one of the impact indicators (environment and natural resources). A possible reason for this is that 34% of the ARRI sample consists of projects from middle-income countries, where the ARRI has found relatively poor impact in the past. The EVEREST sample does not include any representation of middle-income countries. As noted by the ARRI, policy and institutional environments have a bearing on the rural poverty impact of projects. In this regard, as also indicated above, the Asia and the Pacific Region provides a better-than-average operating environment. Nonetheless, in spite of the good project results and impact, important concerns remain about targeting, innovations, sustainability in general terms and impact on the environment and natural resources in particular. Few ratings are available for policy dialogue, donor coordination and partnership strengthening in the 14 projects covered by EVEREST. However, those available show a generally weak impact in these areas, for example, in Bangladesh and Indonesia, but also in other countries. This is a key constraint on IFAD's ability to promote innovative approaches to rural poverty alleviation that can be replicated and scaled up by others.

Assessment of key business processes

The EVEREST team assessed a number of business processes that are crucial to ensuring that the implementation of the Regional Strategy is smooth. These processes include overall PI organization, human resources, budgets, knowledge management, project and programme cycle management, policy dialogue, partnership building, and donor coordination and harmonization.

Following the adoption of the Regional Strategy, the evaluation found that no particular steps had been taken by the division to reposition its organizational structure and human and financial resources to facilitate the implementation of the strategy. In general, the evaluation notes the need for more systematic management of human and financial resources. For example, a number of vacancy announcements issued for country programme managers have been left unfilled for close to two years, and there is inadequate representation of women and staff from developing countries of the region in the regular professional positions. Due to limited resources, staff had few opportunities for training. The allocation of country programmes has not always been made based on experience, competences and staff skills. Attention is required to the management of consultants. For example, only 20% of consultants are women, and the ratio of PI professional staff to consultants (one to 3.5) is very high, compared to the rest of IFAD, which is around one to one. Finally, field presence staff, as part of the FPPP, have not been sufficiently integrated in the overall activities of the division, and insufficient attention and budgets have been allocated to activities like policy dialogue and partnership building during the overall evaluation period.

The division has recently set up Economics and Results teams. Among other useful tasks, the economics team is responsible for undertaking thematic studies on emerging issues, for regional planning and for the production of the new Regional Strategy. The results team is responsible, inter alia, for budget management, the divisional portfolio review and knowledge management. As these teams became fully operational only at the beginning of 2006, it is premature for EVEREST to make meaningful comments on their operations.

Country programme managers have responsibility for two or more countries spread across the region. EVEREST believes that this model may not be the most appropriate organizational set-up for country programme management in Asia and the Pacific Region for a variety of reasons. For example, although each country situation is unique, countries within subregions are more likely to face similar issues and opportunities, and, given the vastness and diversity of the region, there are considerable subregional differences. Moreover, the sorts of issues of interest and the cooperation among countries predominantly revolve around subregions. If country programme managers were to focus their work on one subregion, they would contribute to building more specialized knowledge and networks of subregional consultants and partners, and they would reduce administrative costs. Finally, many country programme managers feel that, professionally, they are working alone in their "country-programme-manager silos" with little horizontal communication and support, and, despite the introduction of the country programme teams, there is no concept in PI of subregional teams or managers that could, among other issues, be responsible for a subregion and for supporting, coaching and supervising less experienced country programme managers and other staff assigned to the subregional teams.

There have been some useful initiatives to promote knowledge sharing, such as the organization of workshops and exchange visits across countries, as well as the recent introduction of peer reviews. During the evaluation period, the knowledge networking for rural development in Asia/Pacific Region(ENRAP) was one of the key vehicles for knowledge management, in fact, devoting much emphasis on the promotion of information and communication technologies, rather than on learning content related to rural poverty reduction. Also, the division has not made proactive efforts to contribute to and benefit from interdivisional learning, and, broadly speaking, the division has not fully utilized OE evaluation results. Generally, few resources and little time have been allocated to the task, and staff do not appear to have the necessary incentives to engage in knowledge sharing activities. Between the country level and headquarters, the knowledge flows are poor, partly due to weak monitoring and evaluation systems in the projects and the lack of a permanent presence in the field (notwithstanding the recent arrangements set up under the FPPP).

Limited ownership and responsibility at the country level among key stakeholders, especially in project and programme design, are an underlying reason for the problems associated with sustainability and are a constraint on even better project performance. The mid-term reviews have proved to be an effective instrument for improving project implementation, but this leads to deferred project effectiveness because important decisions related to potential project redesign are left until the mid-term review exercise, which normally takes place several years after the start of project implementation. The divisional annual portfolio review process has been well implemented, and the attention to self-evaluation processes is increasing.

Notwithstanding recent efforts, limited consideration and resources have been devoted to building strategic partnerships and policy dialogue, as well as donor coordination and harmonization. Partnerships and coordination with key institutions, such as ADB, FAO and the World Bank, are weak even though efforts have been deployed recently to redress the situation. While there is evidence of successful policy dialogue in some countries, this was being pursued mainly through project-related processes, rather than the implementation of coherent policy agendas as a distinct non-lending activity. Limitations in staff time and competencies, as well as management guidance, have been constraints in achieving better results in these areas.

Conclusions

Overview. The storyline emerging from the evaluation of IFAD's Regional Strategy in Asia and the Pacific illustrates that portfolio performance in the region has been good, especially as compared to the IFAD-wide ratings reported in the ARRI. But concerns remain about the systematic promotion of innovations, replication, scaling up, targeting and sustainability, which the evaluation recognizes as areas in need of improvement.

EVEREST concludes that portfolio performance and overall results and impact could have been even greater if better business process management had been in place. For example, changes in key aspects of project and programme cycle management would be beneficial, for instance, by giving more responsibility to the country for project formulation and introducing a thorough annual review of projects/programmes to improve effectiveness, rather than waiting for the mid-term review exercise. Moreover, a more effective approach to human resources and budget management (e.g., by ensuring that country programmes are allocated according to staff experience, skills and competencies or by earmarking sufficient budgets to implement key aspects of the Regional Strategy, such as policy dialogue and impact assessment) and the creation of incentives for knowledge management with stronger linkages to and from the field, including a more systematic use of evaluation results, are likely to contribute to enhanced results.

Performance and efforts in building strategic partnerships with key institutions, the mobilizing of cofinancing and counterpart funds, the promotion of innovations, policy dialogue, and IFAD's participation in donor co-ordination and harmonisation have been overall weak. Low achievements in these critical areas reveal the ‘go it alone' characteristic of IFAD's operational posture.

In sum, results have been good in terms of portfolio performance and impact, which is IFAD's traditional area of business. But, there is no room for complacency given the size and complexity of rural poverty in the region, which IFAD wants to help reduce.

The regional strategy. The preparation process of the Regional Strategy was not well defined. It was not sufficiently inclusive of all key countries in the region, nor did the strategy document fully meet the requirements to allow it to be used as a management tool (see paragraphs 21 and 23). Even though the purpose and audience of the strategy were not articulated clearly, the document was considered by IFAD staff and others as a useful instrument and as an external communication vehicle.

According to EVEREST, the main directions covered in the strategy are mostly relevant. The focus on indigenous peoples is very important, given the poverty conditions faced by these people and the good results and experience accumulated by IFAD in supporting such communities. Likewise, the emphasis devoted to enhancing the capabilities of women and building coalitions of the poor is appropriate. The Fund has achieved positive results in empowering women, even though greater attention could have been given to analysing gender roles and relations, especially in a context where the role and status of women are constantly evolving. EVEREST found, however, that IFAD has devoted disproportionate attention to social capital formation and capacity-building and much less to enhancing productivity and incomes. Good results were achieved in using microfinance as an instrument for rural poverty reduction, even though there is scope for improving targeting and the sustainability of microfinance activities.

IFAD's main objective in the region is to play a catalytic role in addressing rural poverty issues and assisting those rural poor people and their communities that are marginalized and voiceless. This means that, in terms of geographical focus, the Regional Strategy should identify areas where there is a high incidence of marginalized poor people and where there is a realistic potential for IFAD to promote innovative development approaches, with the ultimate objective of allowing others to replicate them and scale them up, whether in less favoured areas or not. This would, inter alia, require: (a) the actual or potential presence in the selected geographical area of organizations committed to developing innovative solutions in agriculture and rural development; (b) plausible indications that the selected geographical area will not be shunned by the government and key donors that have the capability to replicate and scale up successful innovations tested on the ground; and (c) the availability of technical packages or institutional innovations that have good prospects of enhancing agricultural productivity and sustainable livelihoods and that take development experience and knowledge into account in similar geographical areas and with relevant target groups. Such operational criteria in defining IFAD's geographical focus in Asia and the Pacific would facilitate risk management and allow the Fund to make a difference by investing in those areas where others would not otherwise venture, while giving a voice to the many voiceless rural poor. Finally, while recognizing the importance of enhancing peace for poverty reduction, EVEREST concludes that this is not an area in which IFAD currently has a particular comparative advantage.

At the time of preparation, the strategy did not consider adequately several important development aspects. For instance, it did not address the issue of corruption. Likewise, in spite of the successful experiences of IFAD in this area in the region, the strategy did not underline the role IFAD would play in promoting decentralized development. Other important areas not given due treatment included the way IFAD would engage the private sector in its operations and the specific contribution IFAD could make in combating HIV/AIDS in collaboration with others (e.g., the Joint United Nations Programme on HIV/AIDS), for which this would represent a core area of mandate.

The strategy surprisingly underemphasized the rural infrastructure, livestock and fisheries subsectors, to which IFAD has allocated a fair amount of resources (and continues to do so), given their importance for food security and poverty reduction. The issue of migration out of rural areas and the impact of remittances on rural economies, as well as the way in which the Fund could engage cost effectively in the Pacific Islands, were similarly left out of the strategy.

Resource allocation. There is broad consistency between resource allocation and the five main strategic directions in the Regional Strategy, even though resources have been spread thinly across geographical areas within many countries in the region, and, as mentioned elsewhere above, very few resources had been allocated to enhancing peace for poverty reduction. The division has not made sufficient use of grants, thereby limiting, for example, its ability to promote innovations and to engage in policy dialogue processes. Cofinancing declined dramatically during the evaluation period, and counterpart funding has also been falling over the last decade. Declining cofinancing and the reduction in counterpart funds have not yet been matched by a corresponding increase in IFAD's participation in sector-wide approaches or the mobilization of cofinancing through alternative sources, such as from the private sector.

Performance and impact. The ratings on project performance and rural poverty impact in the region are, in general, consistently better than the IFAD-wide ratings for the same criteria. Part of the difference in results and impact may be explained by the operating environment and project implementation in the region, which are generally better than average. However, the ratings on impact on the environment and the communal resource base are less than half the ones in the ARRI: few projects in the region make environment a priority. Targeting and sustainability are causes for concern. IFAD has had some success in policy dialogue. However, the successes have largely been achieved by addressing policy issues in connection with project design and implementation processes, rather than by efforts at integrated policy dialogue at the country programme level. Partnerships with the IFIs and the UN have generally been weak. There are few examples illustrating that IFAD has engaged in donor coordination and harmonization activities, even though the field presence arrangements are contributing to a gradual improvement in these areas.

Programme and project cycle management. One main conclusion is that there is limited ownership and responsibility at the country level in various phases of the project life cycle. This is partly due to the way the project cycle is presently organized; IFAD takes the lead in most activities related to project design. The limited ownership exercised by borrowing countries constrains the achievement of even greater results and sustainability in the projects supported by IFAD. Moreover, while the mid-term reviews have proved to be an effective instrument for improving project implementation, this comes relatively late in implementation and leads to deferred project effectiveness because important decisions related to potential project redesign are often left until the mid-term review takes place.

Knowledge management and learning. Some interesting initiatives were undertaken in knowledge management, such as promoting South-South exchanges and, more recently, the introduction of divisional peer reviews. However, less attention was given to learning from the experiences of other international or bilateral organizations, and no comprehensive approach was defined until 2006 on how knowledge management would be addressed at headquarters, with the required linkages to the field and to others concerned. ENRAP did not focus sufficiently on building up and making relevant knowledge available to key partners. OE evaluation results have not been fully utilized, and there continue to be few incentives, limited time, and few funds to encourage staff to get involved in knowledge management.

Organization, resources and capacity of the Asia and the Pacific Division. Following the adoption of the Regional Strategy, no major steps were taken to reposition PI's organizational structure and human and financial resources in preparation for the implementation of the strategy. The timely filling of vacancies might have helped improve gender balance and enhance the representation of developing countries of the region among the staff. The recent creation of the economics and results teams has the potential to add value to the divisional and individual country programmes. However, a number of older problems persist, such as limited opportunities for staff training, the by and largely stand-alone mode of operation of country programme managers, the lack of a broader integration and utilization of the IFAD field presence arrangements, and inadequate budgetary allocations to further non-lending activities, even though added funds have been earmarked for this over the past couple of years.

Recommendations

There is a clear need for IFAD to develop a new Regional Strategy in Asia and the Pacific for a specific period; it would guide the formulation of country strategies and overall operations in the region. The Regional Strategy would serve as a platform for cooperation and partnership between IFAD and the countries of the region, as well as between IFAD and key regional and subregional organizations. It would also be useful as an accountability framework for IFAD and PI. EVEREST makes five specific recommendations, each one of which is cross-referenced to corresponding sections in the conclusions.

Development of a New Regional Strategy

The strategy should be developed in close consultation with the countries of the region and become a platform for development cooperation and partnership between IFAD and these countries. It should also be used as the basis for defining partnerships with major institutions and development agencies that are active in agriculture and rural development in the region. And the Fund would benefit more broadly if the strategy would build upon the concerns of the variety of countries in the region, that is, both its borrowing and non-borrowing member states, the latter being an important part of the overall enabling environment. Some borrowing countries in the region are particularly interested in gaining access to IFAD's accumulated knowledge and experiences in rural poverty reduction, in addition to the financial investments of the Fund. The Regional Strategy should state how it would address this requirement as well.

A full costing exercise and a thorough analysis of the resource levels required to meet the objectives of the strategy should be a prerequisite up front. In order to gain from the views and knowledge of others, it would be advantageous for IFAD to establish a panel of external peer reviewers during the strategy formulation process.The strategy should indicate how it would retrofit the ongoing country strategies and operations to ensure that IFAD resources are used most effectively and in compliance with the new Regional Strategy. The new Regional Strategy should serve as a management tool for IFAD. In this regard, it should include a results framework and provisions for a mid-term review, and its implementation should be monitored and evaluated.

The directions in the new strategy should be determined following robust analysis of rural poverty and key subsectors in the region. An analysis of emerging opportunities and threats, as well as IFAD's weak and strong points and a detailed study of the role and focus of other major players in agriculture and rural development, should lead to the development and assessment of alternative strategic options before the most promising directions are finally selected, a process that was squarely missing in the development of the 2002 Regional Strategy.

It is recommended that IFAD work in geographical areas where there is a serious opportunity to promote innovations that have the potential of being replicated and scaled up by other partners, in particular, by both the government and donors. This requires, inter alia, a comprehensive institutional analysis up front to ensure that IFAD is able to identify partners who are committed to similar objectives and have the ability to replicate and scale up innovations. Moreover, it is of paramount importance for the Fund to build on its experience, comparative advantage and reputation, as well as focus on supporting those most in need. Another crucial consideration is that the Fund should assess thoroughly the overall governance framework and policy environment of the geographical area within a country where IFAD plans to allocate its resources. In addition, to the extent possible and especially in large countries, it is advisable to concentrate IFAD's assistance geographically, rather than spreading IFAD resources thinly across different parts of the country. This will allow IFAD to develop deeper knowledge and specialization on selected issues and in selected areas, as well as promote greater development results and sustainability.

Continued emphasis on promoting people's participation and building up grass-roots institutions and coalitions of the rural poor should be maintained. In this regard, there is a need to ensure a better balance between empowerment and social capital formation on the one hand and income-generating opportunities (from both on-farm and off-farm sources) on the other. Moreover, women's development is considered important in achieving a deeper impact on rural poverty in the region and should be considered a priority in the process leading up to the selection of future strategic directions. In this regard, greater attention is required in the effort to promote better gender equity and address how gender relations are evolving as a result of women's advancement.

Moreover, given the sizeable involvement of IFAD in the sector, the positive results achieved on the ground, IFAD's thorough understanding of issues affecting development and the continued, widespread poverty among such communities, IFAD should maintain a very special focus on indigenous and tribal peoples in Asia and the Pacific. However, EVEREST underscores that, while focusing on such communities, a broader inclusive approach is necessary to targeting also to ensure that other rural poor living in the same project area play a constructive role in the development operation under consideration.

As in other areas that require priority attention, corruption needs to be tackled explicitly as an overarching theme at the project level, but also at the policy level and in close cooperation with other international development organizations and in consultation with governments, within the broad framework of IFAD's anticorruption policy. Likewise, attention to HIV/AIDS issues would seem to be required in a region in which the number of people affected is large and increasing. However, in this regard, it should be recognized that there are other development organizations at the forefront of HIV/AIDS issues. Hence, IFAD's contribution should naturally be complementary to the work of others, and, in particular, the Fund should only operate in those areas where a gap in assistance might otherwise exist.

Building on its experiences, IFAD can provide support for decentralization, as this would bring development planning and resource allocation closer to the rural poor. In this regard, for IFAD, decentralization should not be considered as an aim in itself, but it should serve as an instrument to enhance the effectiveness of rural poverty reduction programmes. Among other issues, in its future efforts to promote decentralization, IFAD should ensure that locally elected officials and bodies are examined carefully given their central role in development matters at the local level.

Greater attention needs to be devoted to the environment and to natural resource management, where performance is weak, as well as to engaging the private sector proactively in IFAD operations; an IFAD policy has recently been adopted by the Board in this area. Finally, the livestock and rural infrastructure subsectors deserve more attention, given that they constitute important components of IFAD operations that were not adequately included in the 2002 strategy.

The following four recommendations are also considered integral to the new IFAD Regional Strategy in Asia and the Pacific.

Strengthen strategic partnerships and policy dialogue

Building on recent initiatives implemented by the division, EVEREST recommends that a strategic compact with ADB and the World Bank be developed, given the increasing attention and investments of these IFIs to the agriculture and rural sectors. The evaluation recommends that both the PI and IFAD senior management should be closely involved in the development, implementation and review of such a strategic compact. The partnerships would, among other issues, pay attention to cooperation in the areas of policy dialogue and the replication and scaling up of successful innovations promoted by IFAD. Opportunities to develop joint agriculture and rural development strategies within the framework of Poverty Reduction Strategy Papers could be explored in selected countries on a pilot basis. Enhanced cooperation and harmonization with the other two Rome-based UN agencies (especially with FAO, per the Agreement Establishing the Fund) and selected bilateral aid agencies should also be considered, for example, in terms of project design and knowledge sharing.

Enhanced partnership with governments in the region is crucial as well. Greater country ownership in project design (see paragraph 76) would also lead to greater effectiveness and sustainability. For this, IFAD must ensure that its country strategies and operations are firmly anchored in key national strategies and plans for rural poverty reduction. It is, however, important that IFAD work towards engaging a broader range of government institutions in order to benefit from the variety of technical expertise, skills and experiences available at the country level. More effort needs to be invested in increasing government counterpart funding in IFAD operations, as this would be one way of generating added ownership and responsibility.

Private sector engagement needs to be improved in IFAD operations. This is particularly important in areas related to microfinance (such as through the establishment of linkages with commercial banks) and the processing and marketing of farm and non-farm produce, but also in the supply of technical assistance to project implementing agencies (for instance, in setting up and training project staff in monitoring and evaluation systems, in the provision of extension advice, or in undertaking project supervision and implementation support). Strengthened partnership is also needed with civil society and the NGO community be it for advocacy on policy issues, social mobilization, or capacity-building purposes. However, the evaluation experience in various countries has repeatedly highlighted the need to clarify, from the outset, their objectives, roles and responsibilities in order to ensure smooth relations with other project partners.

Finally, it is imperative that policy dialogue be approached in a more systematic manner, anchored in IFAD's operational experiences in the field. Policy dialogue objectives need be to set in a realistic manner, and specific human and financial resources should be allocated for the purpose. IFAD policy dialogue efforts should also be undertaken with similar initiatives by other partners, including international organizations. A clear policy dialogue agenda should be articulated, with indicators that will allow the progress in implementation to be monitored and the corresponding achievements to be evaluated. The role and responsibility of IFAD field presence staff in advancing IFAD's policy dialogue objectives must be clarified, and policy dialogue should be included as a criterion for the year-end performance assessment of staff.

Enhance ownership, accountability and learning through programme cycle management

IFAD should identify appropriate partners through institutional assessments as early as possible in the project design process. The selected implementation partner must designate the project director and a core team to work on the project design process on a full-time basis. Project formulation should be the responsibility mainly of the selected implementing partner, whereas IFAD should continue to be responsible for project appraisal. However, transferring the highly important and time-sensitive formulation process mainly to the implementing partners would leave IFAD facing the risk of not getting projects approved on time. This would be inconsistent with the prevailing situation, wherein staff are rendered accountable for getting loans approved by the Board according to schedules. Therefore, a review of the overall accountability framework in place in the division and in IFAD is required.

Regardless of any changes that might be made in the project design process, problems and lessons that emerge during implementation should be addressed on a regular basis and not kept pending until the mid-term review, which is often a turning point in the development effectiveness of the project. One option for IFAD and its implementing partners would be to conduct joint annual reviews, agree on course corrections, and prepare the annual workplan and budget accordingly. Greater attention and resources should be devoted to project-level monitoring and evaluation systems not only for this reason, but also, more specifically, to improve impact assessment and reporting and contribute more broadly to the division's knowledge management objectives.

Improve impact through better targeting and sustainability

Within the framework of the forthcoming IFAD targeting policy, EVEREST recommends that IFAD should develop a clear and comprehensive approach that would guide IFAD and its partners in: (a) the selection of project areas within a given country; (b) the selection of beneficiary communities within a project area; (c) the targeting of individuals within a community; (d) the matching of interventions to the targeting approach; and (e) regular monitoring and reporting on targeting issues.

Issues of sustainability need to be addressed, for which a range of measures, including the following, should be considered: (i) an exit strategy should be developed in every project at an early stage; (ii) technical standards that are employed in service delivery and infrastructure development should be reviewed, and whether the rural poor can operate and sustain project interventions with the financial, social and human capital available to them should be determined; (iii) in many countries, especially where implementation occurs through government line departments, it may be important to train technical experts to take a more realistic, less technically demanding and more pro-poor approach to sustainability; and (iv) a thorough analysis is required to determine whether agencies charged with operation and maintenance have the capacity to fulfil this aspect of their mandate.

Organizational considerations to achieve the objectives of the regional strategy

Given the multisectoral nature of IFAD's work and the strong focus on country programmes, the continued organization, on a geographical basis, of the core line functions and staff in PI seems appropriate. However, PI should explore the possibility of reorganizing its staff into an appropriate number of subregional teams. There are several examples of such an approach taken by others, including ADB and the World Bank for their operations in Asia and the Pacific. Within IFAD, the Latin America and Caribbean Division has recently adopted a subregional team organizational set-up, and IFAD's operations in Africa are managed by three regional divisions focusing on three subregions of the continent.

With regard to the FPPP, it is recommended that the role and relationships of field presence staff with the division be clarified and accordingly communicated to the main partners at the country level. This includes the countries where IFAD has proxy field presence arrangements. Moreover, as mentioned, a greater delegation of authority to the field presence staff would ensure their better integration in the overall IFAD country team 5concept.

As the work of professional staff and others evolves, the need for training and staff development becomes pressing. In light of the limited resources available in the Human Resources Division for decentralized training purposes, PI training needs will need to be fully costed, and budgets will need to be allocated for the implementation of the training.


1/ Bangladesh, China, India, Indonesia, Laos, Mongolia, Nepal, Pakistan, Papua New Guinea, the Philippines, Sri Lanka and Vietnam.

2/ Prepared on China, India, the Philippines and Pakistan.

3/ Including Indonesia, Laos, Mongolia, Nepal, Papua New Guinea, Sri Lanka and Vietnam.

4/  It would be worthwhile to include on the peer review panel a representative of another IFAD regional operations division.

5/ In fact, granting greater "delegation of authority to donors' field staff" is a key commitment included in the Paris Declaration on Aid Effectiveness.

LANGUAGES: English

Evaluation of IFAD’s regional strategy in Asia and the Pacific (EVEREST)

julio 2006

Background

The International Fund for Agricultural Development (IFAD) adopted its Regional Strategy in Asia and the Pacific in March 2002, which was the first time that the Fund coherently developed and formally adopted a Regional Strategy. During its session in December 2004, IFAD's Executive Board decided that the Office of Evaluation (OE) should undertake, in 2005/06, the Evaluation of IFAD's Regional Strategy in Asia and the Pacific (EVEREST), which would represent the first such type of evaluation conducted by the Fund.

EVEREST, which was conducted under the overall provisions of the IFAD evaluation policy, is expected to provide the key building blocks for the preparation of the new Regional Strategy in Asia and the Pacific that IFAD plans to present to the Board in 2007.

Evaluation objectives and methodology

The main objectives of the evaluation were to: (a) assess IFAD's performance and impact in the region during 1996-2005, with particular reference to the 2002 Regional Strategy in Asia and the Pacific; and (b) develop a series of findings and recommendations that would serve as building blocks for formulating the next IFAD Regional Strategy in Asia and the Pacific. In order to achieve its main objectives, the EVEREST defined the following three key questions that the evaluation would address: (a) What has been the performance and impact of IFAD in the region? (b)Has IFAD made the appropriate strategic choices in the region, including its operational emphases and the choice of niche, instruments, target group, partners, interventions and implementation modalities? and (c) How have IFAD business and management processes influenced performance and impact and the strategic objectives of IFAD in the region?

For each of these questions, more detailed questions were formulated and information sources identified by the evaluation team at the outset of the process. In order to facilitate the work of the evaluation team and to ensure that various stakeholders were able to gain an overview of and comment on the main issues being covered by the evaluation, OE prepared a detailed evaluation framework that captured the main questions the evaluation would cover and linked them to each of the three overarching questions listed above.

Evaluation approach and process

The evaluation approach paper was discussed in two subregional EVEREST launching workshops, one held in Islamabad, and the other in Bangkok, in July 2005. The evaluation benefited from the views and experience of staff in the Asia and the Pacific Division (PI) throughout the process. Discussions were also held with other IFAD staff, including the three Assistant Presidents, various directors in the Programme Management Department, the director and staff in the Office of Human Resources and staff in the Strategic Planning and Budget Division.

A two-person senior advisory panel was constituted at the outset of EVEREST, including Professor M. S. Swaminathan and Dr Robert Picciotto. Their role was to provide comments and guidance on the overall evaluation objectives, methodology and process, as well as to review key deliverables generated throughout EVEREST. Their final report is included in this document (see Appendix 4).

The evaluation noted that the 2002 Regional Strategy formalized a strategy that IFAD had already been following in the region for a number of years before 2002. Given this, it was decided that the evaluation period for the EVEREST analysis would run from 1996 to 2005.

It is important to note that, as a contribution to EVEREST, PI undertook a comprehensive "Self-Assessment of PI Performance in Implementing the Regional Strategy in Asia and the Pacific". The objective of this self-assessment for IFAD was to take stock of the Fund's overall experiences in implementing the Regional Strategy. The self-assessment served as a useful input in EVEREST, inter alia, in collecting and analysing data on the alignment of project design with the strategic directions of the Regional Strategy and on cofinancing and counterpart funding.

The five evaluation phases. The evaluation was organized in five phases. First, an inception phase served to fine-tune the methodology, finalize the selection of countries to be covered, define in detail the workplan and develop the instruments for data collection. Second, a desk review was undertaken in the 12 countries 1 included in EVEREST. More than 25 existing OE evaluation reports were reviewed, and extensive use was made also of the reports prepared during the External Review of the Results and Impact of IFAD Operations (2002) and the Independent External Evaluation of IFAD (2004/05). In addition to the aforementioned, all the IFAD country opportunities strategic papers produced during the evaluation period were reviewed, including the subregional strategies for Central Asia and the Pacific Islands, various country assistance strategies and evaluations of other international organizations, several government documents and so on.

Third, the evaluation team undertook country visits to China, India, Pakistan and the Philippines. In addition, the team visited the headquarters of the ADB twice and met with staff in the regional offices of the Food and Agriculture Organization of the United Nations (FAO) and the United Nations Office for Project Services. In all countries visited, the evaluation team also met with senior government officials, representatives of other international financial institutions (IFIs), selected UN organizations represented at the country level (FAO, the United Nations Development Programme (UNDP) and the World Food Programme) and bilateral aid organizations, as well as academic and research institutions, NGOs, IFAD consultants and project staff.

Fourth, a meeting was held on 10 May 2006 at the M. S. Swaminathan Foundation in Chennai to discuss the draft evaluation report with representatives of IFAD and selected countries in the region. Fifth, OE co-organized with ADB a regional EVEREST workshop in Manila on 28-29 June. This workshop focused on learning from EVEREST with a view to enhancing the effectiveness of IFAD's future strategy in Asia and the Pacific and to pave the way for the production of the Agreement at Completion Point for the evaluation.

It is important to highlight that the Board approved 68 loan projects in the region during the evaluation period, and the analysis conducted for the country working papers2 and cross-country analysis3 extends mainly to 22 of these, plus eight approved before 1996 for which OE evaluations have been completed since 2000. Of the 68 loan projects, 17 are not yet signed or effective, only one of which is included in EVEREST. This means that the evaluation includes 41% of the effective projects approved during the evaluation period. The evaluation also reviewed more than 25 (mainly large) technical assistance grants.

Rural poverty and agriculture in the region4

Poverty in Asia and the Pacific. The Asia and the Pacific Region's track record in reducing income poverty in the 1990s was impressive. Poverty incidence, using a dollar-a-day standard, declined by about 30% over the decade. In 1990, about 32% of people in the region were living below the poverty line. By 2000, this proportion had come down to 22%. Rural poverty in the region declined from 39% to 28%. The absolute number of the poor also declined, by around 180 million, from 900 million in 1990 to 720 million by the end of the decade. This was achieved in spite of a major financial crisis in 1997 that depressed regional growth and increased poverty incidence.

Given continued growth, the Asia and the Pacific Region appears to be on track to meet the Millennium Development Goal for income poverty. However, several countries are likely to be left behind, and others that are likely to meet the target will still have to deal with pockets of poverty. In addition, worsening income distribution patterns are accompanying the economic growth, and the non-income dimensions of poverty have seen much less progress in the Asia and the Pacific Region. In particular, South Asia's progress in reducing the proportion of undernourished children, expanding immunization coverage, increasing the number of births attended by skilled health staff, and combating HIV/AIDS has also been slow. These problems reflect a parallel lag in the progress made towards the gender equality and empowerment goal within this subregion.

Agriculture in Asia and the Pacific. Although the agriculture and rural sectors in the region continue to grow, the sectors are declining in relative importance in Asia, both in terms of their contribution to GDP and their share of the labour force. Urbanization is increasing, and farm households are diversifying their sources of income beyond agriculture. This relative decline of agriculture is inevitable in countries that experience economic growth, which has been widespread in the region. Nevertheless, more than half the economically active population is still involved in agriculture in the region, and agricultural employment is especially important among the livelihoods of the rural poor. In addition, agriculture remains a major sector in nearly all Asian economies even before adding the value generated through downstream processing. Agriculture also serves as a buffer and a safety net by providing employment in the face of large economic shocks, such as the financial crisis in 1997-98.

The importance of agriculture to the macroeconomy, the labour force and the rural poor suggests that investment in agriculture should continue. Indeed, studies in China and India have shown that, dollar for dollar, agricultural research has historically been one of the most effective means of government spending to reduce poverty. Other research has shown that agricultural growth in Asia is typically more pro-poor than growth in other sectors. Unfortunately, public funding for agriculture is declining in many countries, although there are notable exceptions, such as China. Furthermore, much of the funding for agriculture often goes to subsidies that are not targeted on the rural poor, instead of being allocated for activities that increase productivity.

In sum, the Asia and the Pacific Region is home to many of the world's most dynamic economies, but it is also home to a majority of the world's poor. Domestic agriculture provides the bulk of food for the poor in both rural and urban areas, and it is a key provider of jobs in rural areas, where the majority of the poor live. In order to promote agricultural development and growth, deeper attention and more resources will need to be invested in the agriculture sector by governments and the international community so as to ensure that rural poverty does indeed become history.

The role and experiences of other IFIs and UN organizations in the region. Both ADB and the World Bank have increased their attention on and their investments in agriculture and rural development in 2005 and 2006. ADB issued a new poverty reduction strategy at the end of 2004, while the World Bank developed a global rural development strategy in 2002, as well as two subregional strategies for South Asia and East Asia-Pacific shortly before that.

Numerous evaluations examined by the EVEREST team had been prepared by the respective independent evaluation offices of ADB and the World Bank. ADB notes that high performance and impact are difficult to achieve in the agriculture and rural sectors relative to other sectors for a variety of reasons, including complexity in project design, overoptimistic project objectives and inadequate beneficiary participation. The Annual Evaluation Review of ADB (2005) mentions that there is a low probability of success in agriculture and natural resource projects. In comparison, however, according to the 2004 Annual Review of Development Effectiveness of the World Bank, close to 85% of the World Bank's operations globally showed satisfactory outcomes in the agriculture and rural sectors from 2000 to 2004, up from around 75% between 1995 and 1999, with operations in all sectors in Asia and the Pacific performing better than operations in other regions. Interestingly, according to the World Bank, the general perception that rural projects are riskier is not supported by the Bank's quantitative data analysis.

FAO has a regional strategic framework for 2004-06 for Asia and the Pacific, which puts food security and small farmers at the centre of its regional priorities. FAO does not produce a report providing an overview of the organization's development effectiveness and has not undertaken any evaluation of its Regional Strategy in Asia and the Pacific. The UNDP has a Regional Cooperation Framework (2002-06) for Asia and the Pacific, in which agriculture or rural development is not a thematic priority. However, the UNDP does focus on promoting gender equality and tackling HIV/AIDS. Although it produces a development effectiveness report, given the nature of its mandate and operations, comparisons with the UNDP may not prove to be of much significance to IFAD.

IFAD 's strategy in Asia and the Pacific

Strategy preparation process, coverage and format. The process leading up to the development of the strategy was not clearly defined, nor did the preparation include the active participation of persons from all subregions and countries across Asia and the Pacific. The strategy document did not clearly articulate its purpose and intended audience. Moreover, in spite of the dissemination efforts, important partners in the region at the country level and other key development organizations, such as ADB, conveyed to the EVEREST team that they were not familiar with the IFAD Regional Strategy.

However, feedback from IFAD staff, the Executive Directors and others highlighted the usefulness of the document as a reference point for the development of country strategies and operations, as well as for general communication purposes. That said, the evaluative content of the document is weak, as it does not refer adequately to operational experiences or the extensive range of lessons learned generated through OE evaluations. Moreover, it does not include a results framework or a timeline and lacks consideration of the issue of retrofitting ongoing country strategies and operations so as to bring them into conformity with the Regional Strategy. A full costing of the strategy, including human resources and organizational issues, was left out entirely.

Nor does the strategy articulate how it would engage donors and non-borrowing country partners in the region, and it does not sufficiently consider the diversity of the subregions in Asia and the Pacific. In sum, the Regional Strategy does not serve fully as an effective management tool.

Relevance of the strategic directions of the regional strategy. Three of the five strategic directions covered in the document are considered relevant both at the time of the strategy's development and in the context of today's development challenges and opportunities for rural poverty reduction in the region. The development of indigenous peoples, enhancing the capabilities of women and building coalitions of the poor are appropriate as IFAD's strategic directions in Asia and the Pacific. However, in light of IFAD's overarching mandate to promote replicable innovations, its experience and knowledge, and the role of other development organizations, the EVEREST team believes that the development of less favoured areas and enhancing peace for poverty reduction should be reconsidered as strategic directions. The following sections underline some of the key reasons for the findings mentioned in this paragraph.

It is known that large numbers of indigenous peoples are living in the region. These peoples are among the most impoverished and disadvantaged parts of society, and they require special attention and assistance. IFAD has many achievements to highlight in supporting indigenous peoples in Asia and the Pacific. However, while focusing on indigenous peoples, IFAD must also find ways to support other rural poor people living in the same project and programme areas. According to the EVEREST team, this strategic direction, along with the above qualifications, was and remains pertinent for IFAD in the region. This is so because governments and donors alike expect IFAD, given its accumulated experiences, successes and comparative advantage, to take the lead in addressing rural poverty among indigenous peoples. This would also be consistent with and contribute to furthering IFAD's commitment in relation to the Paris Declaration on Aid Effectiveness, which states that "Donors commit to make full use of their respective comparative advantage at sector or country level by delegating, where appropriate, authority to lead donors for the execution of programme, activities and tasks".

EVEREST finds that the emphasis devoted in the Regional Strategy to enhancing the livelihoods of rural women was equally appropriate. For example, women in the region have suffered and, generally, continue to suffer from heavy workloads both at home and in support of family incomes, less nutritional intake than other family members, limited access to assets such as land and housing, domestic violence and lower levels of education. They continue also to have a limited role relative to men in key decision-making processes, especially, but not only in formal institutions and political bodies at all levels. That said, the EVEREST team notes that more attention should have been devoted to gender roles and relations, that is, the consequences produced by the attention on women's development in terms of the overall social and cultural fabric need greater analysis and consideration.

Building coalitions of the poor, including the mobilization and participation of poor people in projects and programmes, is at the centre of IFAD's strategy and operations in the region. It is an underlying condition for achieving results and sustainability. This is so within today's context as much as it was so in 2002 at the time when the Regional Strategy was adopted. The Fund has generally been quite successful in social mobilization, promoting participation and contributing to building grass-roots institutions throughout the region. Many good examples can be cited. At the same time, there are three specific points that need consideration. First, various evaluations have found that greater attention needs to be paid to understanding the types and capabilities of existing institutions at the local level. Second, while emphasis on social mobilization, people's participation and training and on grass-roots institution-building is important in empowering the rural poor, the EVEREST team found that, in many instances, this objective needs to be better integrated with agricultural and non-agricultural productive activities that can contribute to raising incomes. Third, various evaluations call attention to the fact that governments are not always the most appropriate channel for promoting participation, especially in the remote geographical areas covered by IFAD operations. In addition to the above, EVEREST notes that building coalitions of the rural poor has been difficult in some cases, partly because processes to link groups or form associations are not sufficiently well defined in the project designs.

Focusing on the development of less favoured areas, another strategic direction, is a topic that requires careful reflection. Selecting less favoured areas as a precondition to IFAD's assistance could compromise what EVEREST sees as IFAD's central mission: the promotion of innovative approaches to rural poverty reduction that can be replicated and scaled up by others. Less favoured and, especially, remote and marginalized areas are not always a priority for governments and donors, and finding partner institutions that have a track record in innovations and in working in such geographical locations is therefore challenging. Furthermore, less favoured areas are often characterized by poor soils, a harsh climate, weak institutions and limited marketing opportunities. As a result, they may have limited agricultural development potential. Thus, they may represent major risks and limited rewards for the IFAD operations focused on agricultural innovation. On the other hand, it is undeniable that some of the poorest people, including many indigenous people, are living in the less favoured areas, and one might argue that it is IFAD's mandate to contribute to livelihood development among such people not only as a solo agent, but also by demonstrating to other donors that there are ways to assist the poorest even if they are living in the harsh conditions of less favoured areas. Such a policy stance would help in the reduction of regional inequalities. In turn, more equitable distribution of the benefits of growth may yield a reduction in conflicts. On balance, according to EVEREST, investment in less favoured areas should have a place in IFAD's lending priorities, but an exclusive focus on such areas should not determine where IFAD should work within a given country. Rather, IFAD should channel its assistance to those parts of a country where the operating environment is conducive to promoting innovative approaches to agricultural development and rural poverty reduction with a view to their replication and scaling up by others.

The fifth strategic thrust, enhancing peace for poverty reduction, is innovative and bold, given that numerous countries were faced with serious problems of civil unrest and that these problems had not been factored into the strategies of other donors. However, it was not possible for the EVEREST team to assess IFAD's effectiveness in pursuing this policy direction since few projects funded by IFAD in the region focused on peace building. Thus, the self-assessment report prepared by PI notes that only 3% of lending following the adoption of the Regional Strategy was assigned to enhancing peace for poverty reduction. This limited involvement can be traced to a host of constraints on IFAD's capacity to operate in insecure environments. Recently, the development community has begun to give higher priority to peace building activities, including greater support for fragile states, so that IFAD may be called upon to play a pioneering role in peace building, e.g., in community-based projects geared to the reintegration of former combatants into the rural economy. Hence, the question arises whether IFAD should gear up with adequate capacity to work in conflict-ridden and conflict-prone areas, and whether, in this context, it should begin nurturing alliances with development and humanitarian organizations better equipped and specifically mandated to focus on this important theme. Until this policy shift takes place, a key message of the Chennai workshop ("IFAD cannot do everything and needs to be selective in addressing rural poverty") will remain relevant.

Other issues related to strategic directions. The importance attributed to microfinance as an overarching feature in the Regional Strategy is highly relevant, and the results have been positive even though there is room for improvement in targeting the poor and building the sustainability of grass-roots institutions involved in microfinance. However, the strategy did not consider adequately a number of other development issues. First, the centrality of promoting innovations and having demonstration effects for replication and for purposes of scaling up through IFAD-funded operations was not given due treatment. This issue is important throughout the region, but even more so in the larger economies, where IFAD financial resources are not the only attraction available to borrowing countries.

In addition, the strategy, surprisingly, paid limited attention to rural infrastructure and the livestock subsector, which are important subsectors in the region. The issue of how IFAD might complement the work of the main institutions combating HIV/AIDS was not covered. Likewise, in spite of the good experiences in various countries, the Regional Strategy paid limited attention to decentralization and failed to address corruption seriously both at the time of the development of the strategy and following the approval of IFAD's anticorruption policy. The role of the private sector, migration and remittances, their effects on the rural economy and ways in which the Fund could cost-effectively engage in the Pacific Islands were also not covered.

Alignment of the country strategic opportunities papers and operations with the Regional Strategy. By and large, the country strategies and projects formulated by IFAD are in line with the main elements of the Regional Strategy. Depending on the country strategy or project objectives, there is a stronger focus on some strategic directions than others. However, as mentioned elsewhere above, only one country strategy and very few operations have focused on enhancing peace for poverty reduction. Loan resources have been, by and large, allocated in line with the main priorities of the strategy, which are separate from the area of enhancing peace for poverty reduction.

Assessment of resource allocation and results

Resource allocation. Since its establishment, IFAD has provided around USD 2.8 billion to 21 countries in the region in the form of loans. During the evaluation period, IFAD loans in the region stood at around USD 1.2 billion through 68 projects and programmes. This represents 41% of the commitments and 37% of the projects approved for the region since IFAD was established in 1978. Total project and programme costs during the evaluation period were nearly USD 3 billion, and all lending since 1998 has been on highly concessional terms.

According to the regional lending allocations established by IFAD in 1994, the share of Asia and the Pacific Region was set at about 31% of the total for IFAD. Before the introduction of the performance-based allocation system (PBAS), PI worked out country-level allocations largely on the basis of precedent. Seen in conjunction with the current regional shares, these outcomes suggest that, prior to the introduction of the PBAS, IFAD's actual lending in the Asia and the Pacific Region, as also in other regions, was driven by legacy rather than strategy, except that the addition of new borrowers diluted the share of old ones. These are some of the reasons for the introduction of the PBAS, which is expected to establish a more systematic and transparent resource allocation process.

The addition of new borrowers (e.g., Cambodia, Laos and Vietnam) tilted IFAD lending towards East and South-East Asia and away from South Asia. Perhaps the biggest loss was experienced by the Pacific Islands, however; these went from a 2.1% share of the PI portfolio to zero. The introduction of the PBAS is more likely to safeguard the interests of the smallest states if the minimum lending stipulated in the PBAS (USD 1 million per country) is followed. Moreover, because of the weight given to (rural) population in the PBAS, country allocations will shift in favour of South Asia.

EVEREST also notes that, within many of the countries in the region, IFAD support and resources are dispersed in various geographical areas, which, inter alia, militates against promoting synergies across the programme at large and limits the Fund's ability to build deep knowledge on specific rural poverty issues. With regard to grants, the division's current portfolio includes 47 grants, worth USD 17.3 million, that are divided into large and small grants and also regional and country-specific grants. The reported grant portfolio in the region is inexplicably only about 1.5% of the loan portfolio (excluding the programme development financing facility), which seriously limits the division's ability to, inter alia, promote innovation, undertake knowledge management and conduct research through grant-funded initiatives. Finally, it is noted that there have been increases in grant allocations recently for partnership building, policy dialogue, and impact assessment.

Cofinancing and counterpart funding. There has been a drastic reduction in cofinancing in the past four or five years. Cofinancing as a percentage of IFAD lending varied between 65% and 68% in 2001-02 and only 3%-4% in 2003-04. There are various reasons for this, including the organizational decentralization of donor agencies, the increase in the use of other instruments to channel aid (such as sector-wide approaches and budget support) and limited knowledge among potential cofinanciers at the country level of IFAD's programmes and experiences. Furthermore, there is little management guidance, and the incentives for mobilizing cofinancing are few. The lack of continuity in cofinancing partnerships is also a concern: 35 different partners have cofinanced IFAD projects in Asia since 1978, but only 11 of these were actively cofinancing with IFAD in 2002-05. New cofinancing with ADB and the World Bank has been zero since 2002 (see Main Report, Tables 10 and 11). The level of counterpart funding has also fallen markedly, from 45% of total cost between 1978 and 1995 to 19% between 2002 and 2005.

Project portfolio performance (see Main Report, Table 3). The ratings of the 14 projects examined in EVEREST show more or less consistently better results than those reported for IFAD as a whole in the Annual Report on the Results and Impact of IFAD Operations (ARRI) during 2002-04. In 100% of the sample, project objectives are highly or substantially relevant in Asia and the Pacific. In terms of effectiveness, projects in the Asia and the Pacific Region achieved far better results than the projects considered in the ARRI. Some reasons for this include the following: (a) IFAD operations in the region benefit from sounder institutions, well-trained human resources and a relatively favourable policy environment prevalent generally in the region. This is reflected in the high levels of economic growth and relatively good development performance of the region. In fact, the outcomes for projects funded by the World Bank are better in Asia and the Pacific than they are in other regions, and they are above Bank-wide averages. (b) The extensive involvement and quality of NGOs and community-based organizations, as well as the enhanced participation of women in development activities, help explain the higher effectiveness. (c) The project implementation indicators are better relative to those in other IFAD regions, for example, in terms of disbursements and time taken from loan approval to effectiveness. As far as efficiency is concerned, the ratings of projects in the Asia and the Pacific Region are a little better than the ARRI figures. The measurement of efficiency is not facilitated by the unsystematic data collection and infrequent estimates of cost-benefit indicators in IFAD-funded projects.

Impact on rural poverty (see Main Report, Table 4). In terms of the impact of projects on rural poverty, it is notable that the EVEREST sample yields higher ratings than the ARRI sample in all but one of the impact indicators (environment and natural resources). A possible reason for this is that 34% of the ARRI sample consists of projects from middle-income countries, where the ARRI has found relatively poor impact in the past. The EVEREST sample does not include any representation of middle-income countries. As noted by the ARRI, policy and institutional environments have a bearing on the rural poverty impact of projects. In this regard, as also indicated above, the Asia and the Pacific Region provides a better-than-average operating environment. Nonetheless, in spite of the good project results and impact, important concerns remain about targeting, innovations, sustainability in general terms and impact on the environment and natural resources in particular. Few ratings are available for policy dialogue, donor coordination and partnership strengthening in the 14 projects covered by EVEREST. However, those available show a generally weak impact in these areas, for example, in Bangladesh and Indonesia, but also in other countries. This is a key constraint on IFAD's ability to promote innovative approaches to rural poverty alleviation that can be replicated and scaled up by others.

Assessment of key business processes

The EVEREST team assessed a number of business processes that are crucial to ensuring that the implementation of the Regional Strategy is smooth. These processes include overall PI organization, human resources, budgets, knowledge management, project and programme cycle management, policy dialogue, partnership building, and donor coordination and harmonization.

Following the adoption of the Regional Strategy, the evaluation found that no particular steps had been taken by the division to reposition its organizational structure and human and financial resources to facilitate the implementation of the strategy. In general, the evaluation notes the need for more systematic management of human and financial resources. For example, a number of vacancy announcements issued for country programme managers have been left unfilled for close to two years, and there is inadequate representation of women and staff from developing countries of the region in the regular professional positions. Due to limited resources, staff had few opportunities for training. The allocation of country programmes has not always been made based on experience, competences and staff skills. Attention is required to the management of consultants. For example, only 20% of consultants are women, and the ratio of PI professional staff to consultants (one to 3.5) is very high, compared to the rest of IFAD, which is around one to one. Finally, field presence staff, as part of the FPPP, have not been sufficiently integrated in the overall activities of the division, and insufficient attention and budgets have been allocated to activities like policy dialogue and partnership building during the overall evaluation period.

The division has recently set up Economics and Results teams. Among other useful tasks, the economics team is responsible for undertaking thematic studies on emerging issues, for regional planning and for the production of the new Regional Strategy. The results team is responsible, inter alia, for budget management, the divisional portfolio review and knowledge management. As these teams became fully operational only at the beginning of 2006, it is premature for EVEREST to make meaningful comments on their operations.

Country programme managers have responsibility for two or more countries spread across the region. EVEREST believes that this model may not be the most appropriate organizational set-up for country programme management in Asia and the Pacific Region for a variety of reasons. For example, although each country situation is unique, countries within subregions are more likely to face similar issues and opportunities, and, given the vastness and diversity of the region, there are considerable subregional differences. Moreover, the sorts of issues of interest and the cooperation among countries predominantly revolve around subregions. If country programme managers were to focus their work on one subregion, they would contribute to building more specialized knowledge and networks of subregional consultants and partners, and they would reduce administrative costs. Finally, many country programme managers feel that, professionally, they are working alone in their "country-programme-manager silos" with little horizontal communication and support, and, despite the introduction of the country programme teams, there is no concept in PI of subregional teams or managers that could, among other issues, be responsible for a subregion and for supporting, coaching and supervising less experienced country programme managers and other staff assigned to the subregional teams.

There have been some useful initiatives to promote knowledge sharing, such as the organization of workshops and exchange visits across countries, as well as the recent introduction of peer reviews. During the evaluation period, the knowledge networking for rural development in Asia/Pacific Region(ENRAP) was one of the key vehicles for knowledge management, in fact, devoting much emphasis on the promotion of information and communication technologies, rather than on learning content related to rural poverty reduction. Also, the division has not made proactive efforts to contribute to and benefit from interdivisional learning, and, broadly speaking, the division has not fully utilized OE evaluation results. Generally, few resources and little time have been allocated to the task, and staff do not appear to have the necessary incentives to engage in knowledge sharing activities. Between the country level and headquarters, the knowledge flows are poor, partly due to weak monitoring and evaluation systems in the projects and the lack of a permanent presence in the field (notwithstanding the recent arrangements set up under the FPPP).

Limited ownership and responsibility at the country level among key stakeholders, especially in project and programme design, are an underlying reason for the problems associated with sustainability and are a constraint on even better project performance. The mid-term reviews have proved to be an effective instrument for improving project implementation, but this leads to deferred project effectiveness because important decisions related to potential project redesign are left until the mid-term review exercise, which normally takes place several years after the start of project implementation. The divisional annual portfolio review process has been well implemented, and the attention to self-evaluation processes is increasing.

Notwithstanding recent efforts, limited consideration and resources have been devoted to building strategic partnerships and policy dialogue, as well as donor coordination and harmonization. Partnerships and coordination with key institutions, such as ADB, FAO and the World Bank, are weak even though efforts have been deployed recently to redress the situation. While there is evidence of successful policy dialogue in some countries, this was being pursued mainly through project-related processes, rather than the implementation of coherent policy agendas as a distinct non-lending activity. Limitations in staff time and competencies, as well as management guidance, have been constraints in achieving better results in these areas.

Conclusions

Overview. The storyline emerging from the evaluation of IFAD's Regional Strategy in Asia and the Pacific illustrates that portfolio performance in the region has been good, especially as compared to the IFAD-wide ratings reported in the ARRI. But concerns remain about the systematic promotion of innovations, replication, scaling up, targeting and sustainability, which the evaluation recognizes as areas in need of improvement.

EVEREST concludes that portfolio performance and overall results and impact could have been even greater if better business process management had been in place. For example, changes in key aspects of project and programme cycle management would be beneficial, for instance, by giving more responsibility to the country for project formulation and introducing a thorough annual review of projects/programmes to improve effectiveness, rather than waiting for the mid-term review exercise. Moreover, a more effective approach to human resources and budget management (e.g., by ensuring that country programmes are allocated according to staff experience, skills and competencies or by earmarking sufficient budgets to implement key aspects of the Regional Strategy, such as policy dialogue and impact assessment) and the creation of incentives for knowledge management with stronger linkages to and from the field, including a more systematic use of evaluation results, are likely to contribute to enhanced results.

Performance and efforts in building strategic partnerships with key institutions, the mobilizing of cofinancing and counterpart funds, the promotion of innovations, policy dialogue, and IFAD's participation in donor co-ordination and harmonisation have been overall weak. Low achievements in these critical areas reveal the ‘go it alone' characteristic of IFAD's operational posture.

In sum, results have been good in terms of portfolio performance and impact, which is IFAD's traditional area of business. But, there is no room for complacency given the size and complexity of rural poverty in the region, which IFAD wants to help reduce.

The regional strategy. The preparation process of the Regional Strategy was not well defined. It was not sufficiently inclusive of all key countries in the region, nor did the strategy document fully meet the requirements to allow it to be used as a management tool (see paragraphs 21 and 23). Even though the purpose and audience of the strategy were not articulated clearly, the document was considered by IFAD staff and others as a useful instrument and as an external communication vehicle.

According to EVEREST, the main directions covered in the strategy are mostly relevant. The focus on indigenous peoples is very important, given the poverty conditions faced by these people and the good results and experience accumulated by IFAD in supporting such communities. Likewise, the emphasis devoted to enhancing the capabilities of women and building coalitions of the poor is appropriate. The Fund has achieved positive results in empowering women, even though greater attention could have been given to analysing gender roles and relations, especially in a context where the role and status of women are constantly evolving. EVEREST found, however, that IFAD has devoted disproportionate attention to social capital formation and capacity-building and much less to enhancing productivity and incomes. Good results were achieved in using microfinance as an instrument for rural poverty reduction, even though there is scope for improving targeting and the sustainability of microfinance activities.

IFAD's main objective in the region is to play a catalytic role in addressing rural poverty issues and assisting those rural poor people and their communities that are marginalized and voiceless. This means that, in terms of geographical focus, the Regional Strategy should identify areas where there is a high incidence of marginalized poor people and where there is a realistic potential for IFAD to promote innovative development approaches, with the ultimate objective of allowing others to replicate them and scale them up, whether in less favoured areas or not. This would, inter alia, require: (a) the actual or potential presence in the selected geographical area of organizations committed to developing innovative solutions in agriculture and rural development; (b) plausible indications that the selected geographical area will not be shunned by the government and key donors that have the capability to replicate and scale up successful innovations tested on the ground; and (c) the availability of technical packages or institutional innovations that have good prospects of enhancing agricultural productivity and sustainable livelihoods and that take development experience and knowledge into account in similar geographical areas and with relevant target groups. Such operational criteria in defining IFAD's geographical focus in Asia and the Pacific would facilitate risk management and allow the Fund to make a difference by investing in those areas where others would not otherwise venture, while giving a voice to the many voiceless rural poor. Finally, while recognizing the importance of enhancing peace for poverty reduction, EVEREST concludes that this is not an area in which IFAD currently has a particular comparative advantage.

At the time of preparation, the strategy did not consider adequately several important development aspects. For instance, it did not address the issue of corruption. Likewise, in spite of the successful experiences of IFAD in this area in the region, the strategy did not underline the role IFAD would play in promoting decentralized development. Other important areas not given due treatment included the way IFAD would engage the private sector in its operations and the specific contribution IFAD could make in combating HIV/AIDS in collaboration with others (e.g., the Joint United Nations Programme on HIV/AIDS), for which this would represent a core area of mandate.

The strategy surprisingly underemphasized the rural infrastructure, livestock and fisheries subsectors, to which IFAD has allocated a fair amount of resources (and continues to do so), given their importance for food security and poverty reduction. The issue of migration out of rural areas and the impact of remittances on rural economies, as well as the way in which the Fund could engage cost effectively in the Pacific Islands, were similarly left out of the strategy.

Resource allocation. There is broad consistency between resource allocation and the five main strategic directions in the Regional Strategy, even though resources have been spread thinly across geographical areas within many countries in the region, and, as mentioned elsewhere above, very few resources had been allocated to enhancing peace for poverty reduction. The division has not made sufficient use of grants, thereby limiting, for example, its ability to promote innovations and to engage in policy dialogue processes. Cofinancing declined dramatically during the evaluation period, and counterpart funding has also been falling over the last decade. Declining cofinancing and the reduction in counterpart funds have not yet been matched by a corresponding increase in IFAD's participation in sector-wide approaches or the mobilization of cofinancing through alternative sources, such as from the private sector.

Performance and impact. The ratings on project performance and rural poverty impact in the region are, in general, consistently better than the IFAD-wide ratings for the same criteria. Part of the difference in results and impact may be explained by the operating environment and project implementation in the region, which are generally better than average. However, the ratings on impact on the environment and the communal resource base are less than half the ones in the ARRI: few projects in the region make environment a priority. Targeting and sustainability are causes for concern. IFAD has had some success in policy dialogue. However, the successes have largely been achieved by addressing policy issues in connection with project design and implementation processes, rather than by efforts at integrated policy dialogue at the country programme level. Partnerships with the IFIs and the UN have generally been weak. There are few examples illustrating that IFAD has engaged in donor coordination and harmonization activities, even though the field presence arrangements are contributing to a gradual improvement in these areas.

Programme and project cycle management. One main conclusion is that there is limited ownership and responsibility at the country level in various phases of the project life cycle. This is partly due to the way the project cycle is presently organized; IFAD takes the lead in most activities related to project design. The limited ownership exercised by borrowing countries constrains the achievement of even greater results and sustainability in the projects supported by IFAD. Moreover, while the mid-term reviews have proved to be an effective instrument for improving project implementation, this comes relatively late in implementation and leads to deferred project effectiveness because important decisions related to potential project redesign are often left until the mid-term review takes place.

Knowledge management and learning. Some interesting initiatives were undertaken in knowledge management, such as promoting South-South exchanges and, more recently, the introduction of divisional peer reviews. However, less attention was given to learning from the experiences of other international or bilateral organizations, and no comprehensive approach was defined until 2006 on how knowledge management would be addressed at headquarters, with the required linkages to the field and to others concerned. ENRAP did not focus sufficiently on building up and making relevant knowledge available to key partners. OE evaluation results have not been fully utilized, and there continue to be few incentives, limited time, and few funds to encourage staff to get involved in knowledge management.

Organization, resources and capacity of the Asia and the Pacific Division. Following the adoption of the Regional Strategy, no major steps were taken to reposition PI's organizational structure and human and financial resources in preparation for the implementation of the strategy. The timely filling of vacancies might have helped improve gender balance and enhance the representation of developing countries of the region among the staff. The recent creation of the economics and results teams has the potential to add value to the divisional and individual country programmes. However, a number of older problems persist, such as limited opportunities for staff training, the by and largely stand-alone mode of operation of country programme managers, the lack of a broader integration and utilization of the IFAD field presence arrangements, and inadequate budgetary allocations to further non-lending activities, even though added funds have been earmarked for this over the past couple of years.

Recommendations

There is a clear need for IFAD to develop a new Regional Strategy in Asia and the Pacific for a specific period; it would guide the formulation of country strategies and overall operations in the region. The Regional Strategy would serve as a platform for cooperation and partnership between IFAD and the countries of the region, as well as between IFAD and key regional and subregional organizations. It would also be useful as an accountability framework for IFAD and PI. EVEREST makes five specific recommendations, each one of which is cross-referenced to corresponding sections in the conclusions.

Development of a New Regional Strategy

The strategy should be developed in close consultation with the countries of the region and become a platform for development cooperation and partnership between IFAD and these countries. It should also be used as the basis for defining partnerships with major institutions and development agencies that are active in agriculture and rural development in the region. And the Fund would benefit more broadly if the strategy would build upon the concerns of the variety of countries in the region, that is, both its borrowing and non-borrowing member states, the latter being an important part of the overall enabling environment. Some borrowing countries in the region are particularly interested in gaining access to IFAD's accumulated knowledge and experiences in rural poverty reduction, in addition to the financial investments of the Fund. The Regional Strategy should state how it would address this requirement as well.

A full costing exercise and a thorough analysis of the resource levels required to meet the objectives of the strategy should be a prerequisite up front. In order to gain from the views and knowledge of others, it would be advantageous for IFAD to establish a panel of external peer reviewers during the strategy formulation process.4 The strategy should indicate how it would retrofit the ongoing country strategies and operations to ensure that IFAD resources are used most effectively and in compliance with the new Regional Strategy. The new Regional Strategy should serve as a management tool for IFAD. In this regard, it should include a results framework and provisions for a mid-term review, and its implementation should be monitored and evaluated.

The directions in the new strategy should be determined following robust analysis of rural poverty and key subsectors in the region. An analysis of emerging opportunities and threats, as well as IFAD's weak and strong points and a detailed study of the role and focus of other major players in agriculture and rural development, should lead to the development and assessment of alternative strategic options before the most promising directions are finally selected, a process that was squarely missing in the development of the 2002 Regional Strategy.

It is recommended that IFAD work in geographical areas where there is a serious opportunity to promote innovations that have the potential of being replicated and scaled up by other partners, in particular, by both the government and donors. This requires, inter alia, a comprehensive institutional analysis up front to ensure that IFAD is able to identify partners who are committed to similar objectives and have the ability to replicate and scale up innovations. Moreover, it is of paramount importance for the Fund to build on its experience, comparative advantage and reputation, as well as focus on supporting those most in need. Another crucial consideration is that the Fund should assess thoroughly the overall governance framework and policy environment of the geographical area within a country where IFAD plans to allocate its resources. In addition, to the extent possible and especially in large countries, it is advisable to concentrate IFAD's assistance geographically, rather than spreading IFAD resources thinly across different parts of the country. This will allow IFAD to develop deeper knowledge and specialization on selected issues and in selected areas, as well as promote greater development results and sustainability.

Continued emphasis on promoting people's participation and building up grass-roots institutions and coalitions of the rural poor should be maintained. In this regard, there is a need to ensure a better balance between empowerment and social capital formation on the one hand and income-generating opportunities (from both on-farm and off-farm sources) on the other. Moreover, women's development is considered important in achieving a deeper impact on rural poverty in the region and should be considered a priority in the process leading up to the selection of future strategic directions. In this regard, greater attention is required in the effort to promote better gender equity and address how gender relations are evolving as a result of women's advancement.

Moreover, given the sizeable involvement of IFAD in the sector, the positive results achieved on the ground, IFAD's thorough understanding of issues affecting development and the continued, widespread poverty among such communities, IFAD should maintain a very special focus on indigenous and tribal peoples in Asia and the Pacific. However, EVEREST underscores that, while focusing on such communities, a broader inclusive approach is necessary to targeting also to ensure that other rural poor living in the same project area play a constructive role in the development operation under consideration.

As in other areas that require priority attention, corruption needs to be tackled explicitly as an overarching theme at the project level, but also at the policy level and in close cooperation with other international development organizations and in consultation with governments, within the broad framework of IFAD's anticorruption policy. Likewise, attention to HIV/AIDS issues would seem to be required in a region in which the number of people affected is large and increasing. However, in this regard, it should be recognized that there are other development organizations at the forefront of HIV/AIDS issues. Hence, IFAD's contribution should naturally be complementary to the work of others, and, in particular, the Fund should only operate in those areas where a gap in assistance might otherwise exist.

Building on its experiences, IFAD can provide support for decentralization, as this would bring development planning and resource allocation closer to the rural poor. In this regard, for IFAD, decentralization should not be considered as an aim in itself, but it should serve as an instrument to enhance the effectiveness of rural poverty reduction programmes. Among other issues, in its future efforts to promote decentralization, IFAD should ensure that locally elected officials and bodies are examined carefully given their central role in development matters at the local level.

Greater attention needs to be devoted to the environment and to natural resource management, where performance is weak, as well as to engaging the private sector proactively in IFAD operations; an IFAD policy has recently been adopted by the Board in this area. Finally, the livestock and rural infrastructure subsectors deserve more attention, given that they constitute important components of IFAD operations that were not adequately included in the 2002 strategy.

The following four recommendations are also considered integral to the new IFAD Regional Strategy in Asia and the Pacific.

Strengthen strategic partnerships and policy dialogue

Building on recent initiatives implemented by the division, EVEREST recommends that a strategic compact with ADB and the World Bank be developed, given the increasing attention and investments of these IFIs to the agriculture and rural sectors. The evaluation recommends that both the PI and IFAD senior management should be closely involved in the development, implementation and review of such a strategic compact. The partnerships would, among other issues, pay attention to cooperation in the areas of policy dialogue and the replication and scaling up of successful innovations promoted by IFAD. Opportunities to develop joint agriculture and rural development strategies within the framework of Poverty Reduction Strategy Papers could be explored in selected countries on a pilot basis. Enhanced cooperation and harmonization with the other two Rome-based UN agencies (especially with FAO, per the Agreement Establishing the Fund) and selected bilateral aid agencies should also be considered, for example, in terms of project design and knowledge sharing.

Enhanced partnership with governments in the region is crucial as well. Greater country ownership in project design (see paragraph 76) would also lead to greater effectiveness and sustainability. For this, IFAD must ensure that its country strategies and operations are firmly anchored in key national strategies and plans for rural poverty reduction. It is, however, important that IFAD work towards engaging a broader range of government institutions in order to benefit from the variety of technical expertise, skills and experiences available at the country level. More effort needs to be invested in increasing government counterpart funding in IFAD operations, as this would be one way of generating added ownership and responsibility.

Private sector engagement needs to be improved in IFAD operations. This is particularly important in areas related to microfinance (such as through the establishment of linkages with commercial banks) and the processing and marketing of farm and non-farm produce, but also in the supply of technical assistance to project implementing agencies (for instance, in setting up and training project staff in monitoring and evaluation systems, in the provision of extension advice, or in undertaking project supervision and implementation support). Strengthened partnership is also needed with civil society and the NGO community be it for advocacy on policy issues, social mobilization, or capacity-building purposes. However, the evaluation experience in various countries has repeatedly highlighted the need to clarify, from the outset, their objectives, roles and responsibilities in order to ensure smooth relations with other project partners.

Finally, it is imperative that policy dialogue be approached in a more systematic manner, anchored in IFAD's operational experiences in the field. Policy dialogue objectives need be to set in a realistic manner, and specific human and financial resources should be allocated for the purpose. IFAD policy dialogue efforts should also be undertaken with similar initiatives by other partners, including international organizations. A clear policy dialogue agenda should be articulated, with indicators that will allow the progress in implementation to be monitored and the corresponding achievements to be evaluated. The role and responsibility of IFAD field presence staff in advancing IFAD's policy dialogue objectives must be clarified, and policy dialogue should be included as a criterion for the year-end performance assessment of staff.

Enhance ownership, accountability and learning through programme cycle management

IFAD should identify appropriate partners through institutional assessments as early as possible in the project design process. The selected implementation partner must designate the project director and a core team to work on the project design process on a full-time basis. Project formulation should be the responsibility mainly of the selected implementing partner, whereas IFAD should continue to be responsible for project appraisal. However, transferring the highly important and time-sensitive formulation process mainly to the implementing partners would leave IFAD facing the risk of not getting projects approved on time. This would be inconsistent with the prevailing situation, wherein staff are rendered accountable for getting loans approved by the Board according to schedules. Therefore, a review of the overall accountability framework in place in the division and in IFAD is required.

Regardless of any changes that might be made in the project design process, problems and lessons that emerge during implementation should be addressed on a regular basis and not kept pending until the mid-term review, which is often a turning point in the development effectiveness of the project. One option for IFAD and its implementing partners would be to conduct joint annual reviews, agree on course corrections, and prepare the annual workplan and budget accordingly. Greater attention and resources should be devoted to project-level monitoring and evaluation systems not only for this reason, but also, more specifically, to improve impact assessment and reporting and contribute more broadly to the division's knowledge management objectives.

Improve impact through better targeting and sustainability

Within the framework of the forthcoming IFAD targeting policy, EVEREST recommends that IFAD should develop a clear and comprehensive approach that would guide IFAD and its partners in: (a) the selection of project areas within a given country; (b) the selection of beneficiary communities within a project area; (c) the targeting of individuals within a community; (d) the matching of interventions to the targeting approach; and (e) regular monitoring and reporting on targeting issues.

Issues of sustainability need to be addressed, for which a range of measures, including the following, should be considered: (i) an exit strategy should be developed in every project at an early stage; (ii) technical standards that are employed in service delivery and infrastructure development should be reviewed, and whether the rural poor can operate and sustain project interventions with the financial, social and human capital available to them should be determined; (iii) in many countries, especially where implementation occurs through government line departments, it may be important to train technical experts to take a more realistic, less technically demanding and more pro-poor approach to sustainability; and (iv) a thorough analysis is required to determine whether agencies charged with operation and maintenance have the capacity to fulfil this aspect of their mandate.

Organizational considerations to achieve the objectives of the regional strategy

Given the multisectoral nature of IFAD's work and the strong focus on country programmes, the continued organization, on a geographical basis, of the core line functions and staff in PI seems appropriate. However, PI should explore the possibility of reorganizing its staff into an appropriate number of subregional teams. There are several examples of such an approach taken by others, including ADB and the World Bank for their operations in Asia and the Pacific. Within IFAD, the Latin America and Caribbean Division has recently adopted a subregional team organizational set-up, and IFAD's operations in Africa are managed by three regional divisions focusing on three subregions of the continent.

With regard to the FPPP, it is recommended that the role and relationships of field presence staff with the division be clarified and accordingly communicated to the main partners at the country level. This includes the countries where IFAD has proxy field presence arrangements. Moreover, as mentioned, a greater delegation of authority to the field presence staff would ensure their better integration in the overall IFAD country team 5 concept.

As the work of professional staff and others evolves, the need for training and staff development becomes pressing. In light of the limited resources available in the Human Resources Division for decentralized training purposes, PI training needs will need to be fully costed, and budgets will need to be allocated for the implementation of the training.


1/ Bangladesh, China, India, Indonesia, Laos, Mongolia, Nepal, Pakistan, Papua New Guinea, the Philippines, Sri Lanka and Vietnam.

2/ Prepared on China, India, the Philippines and Pakistan.

3/ Including Indonesia, Laos, Mongolia, Nepal, Papua New Guinea, Sri Lanka and Vietnam.

4/  It would be worthwhile to include on the peer review panel a representative of another IFAD regional operations division.

5/ In fact, granting greater "delegation of authority to donors' field staff" is a key commitment included in the Paris Declaration on Aid Effectiveness.

 

 

 

 

LANGUAGES: English

Direct supervision pilot programme

noviembre 2005

Background. In February 1997, the IFAD Governing Council adopted resolution 102/XX on Loan Administration and Supervision of Project Implementation, together with a Five-Year Plan of Action. The resolution stated that IFAD "may supervise specific projects and programmes financed by it". According to the action plan, no more than 15 IFAD-initiated projects were to be directly supervised and administered during the five-year period. This initiative, including the 15 projects, has since been referred to as the Direct Supervision Pilot Programme (DSPP). The Governing Council resolution entered into effect on 21 February 1997 and will cease to be operational five years after the date of effectiveness of the last approved project to be directly supervised by IFAD1. The Governing Council also decided that the DSPP should not entail any cost increases and that IFAD should use the same funds that would have been spent on supervision by cooperating institutions (CIs) to cover its direct supervision activities.

In order to assess the results of the action plan, in 2002/2003 the Office of Evaluation (OE) undertook an Evaluation of the Supervision Modalities in IFAD-Supported Projects (ESM). The objectives of the ESM were to "evaluate the effectiveness of current supervision modalities against the minimum supervision requirements [MSRs] … and other indicators of quality, and review the achievements under the Five-Year Plan of Action". It is important to note that most of the directly supervised projects were in their early stages of implementation at the time of the ESM. Hence, although the ESM provided concrete findings on supervision through CI, it provided only an overview of the emerging characteristics of IFAD's direct supervision efforts.

DSPP objectives. The overarching objective for embarking on the DSPP was to enable the Fund to acquire first-hand knowledge from supervision activities and to incorporate lessons learned from ongoing operations more effectively into its project design work. It was also to provide IFAD with "knowledge of the supervision function, of what are the costs of an adequate project supervision and of the development impact and human dimension of the projects in its portfolio. The Fund's involvement in direct supervision would also complement and improve cooperating institutions' own activities (mainly the human dimension of projects/programmes)". Although enhancing development effectiveness2 was not an explicit objective, direct supervision was expected to contribute to improving implementation performance and project impact.

Evaluation objectives. The main objective of the corporate-level evaluation (CLE) was to make an overall assessment of the DSPP's achievement in enhancing the implementation and impact of IFAD-funded operations. More specifically, the evaluation had the following key objectives: (a) compare and contrast direct supervision by IFAD with selected, relevant examples of supervision undertaken by CIs; (b) examine the processes established, alternative approaches and the experiences of country programme managers (CPMs) in undertaking direct supervision. This would include comparison with the approaches, systems and experiences of other international financial institutions (IFIs); (c) assess the efficiency of the direct supervision modality; and (d) examine the systems established to capture the experience and insights from direct supervision and the ways in which this has been of benefit to IFAD's project design processes and implementation support activities.

Evaluation methodology. The approach for the direct supervision evaluation has been to build on the methodology and results of the ESM. The criteria3 used in the CLE to assess the DSPP include: (a) the relevance of direct supervision in relation to the DSPP's objectives at the time of approval but also in today's context; (b) the effectiveness of the DSPP, measured against the achievement of the stated objectives of the programme and using the indicators specified for measurement of the impact of direct supervision;4 (c) the efficiency of direct supervision; and (d) an analysis and comparison of the actual and potential impact of the projects included in the direct supervision pilot programme with those supervised by CIs.

The evaluation process was planned to allow for triangulation of evidence and the views obtained from the main actors in the DSPP (the key government focal point at the national level, together with implementing agencies, beneficiaries, and the IFAD staff concerned). Moreover, OE had completed evaluations of three directly supervised projects5 in the past few years, which provided valuable additional sources of information and assessments. Since 2003, OE has also undertaken three country programme evaluations6 that included assessment of one directly supervised project in each of these countries. Relevant information and reports from the Independent External Evaluation (IEE) of IFAD, which analysed in detail two directly supervised projects7, have also been utilized. Finally, four projects8 in the evaluation control group have either been evaluated by OE or included in the IEE.

Evaluation scope. The evaluation involved a comparison of the directly supervised projects with 15 projects supervised by CIs (the latter therefore represented the evaluation's control group). That is, three CI-supervised projects per region were included in the control group based on a set of selection criteria agreed with the evaluation's core learning partnership (CLP)9. The list of projects included in the DSPP and the control group for the purposes of the evaluation can be seen in Appendix I.

Evaluation process. The CLE process began with the preparation of an approach paper, which provided an opportunity for developing a solid understanding of the evaluation's objectives, scope, methodology, time frames and expected outputs. The evaluation benefited from the views of the CLP. An external reviewer10 with wide experience in project supervision issues was contracted to advise OE at critical phases during the evaluation and to review key evaluation outputs. Moreover, OE undertook a thorough peer-review process within the division to improve the evaluation's overall quality.

The evaluation included the following activities: desk reviews of the 15 directly supervised projects and 15 projects in the control group; interviews with IFAD management and staff; field work in 13 of the 15 directly supervised projects and in eight control group projects11; discussion with the United Nations Office for Project Services (UNOPS) and selected IFIs at their headquarters and at the country level; direct supervision cost analysis; preparation of an early feedback note on an experimental basis, with the objective of sharing the emerging evaluation results and sensitizing IFAD management before the draft evaluation report was ready for discussion; organization of a round-table workshop to discuss the evaluation's results and lay the foundations for the agreement at completion point of the evaluation; and discussion at the September 2005 sessions of the IFAD Evaluation Committee and Executive Board.

The draft evaluation report was shared for review and comments with: staff in all the projects included in the DSPP and the control group, IFAD management and other staff, selected IFIs and CIs, and the government officials concerned at the national level in countries in which IFAD has funded a directly supervised project. Their comments have been included in accordance with the provisions in paragraph 42 of the IFAD Evaluation Policy.

The evolution of direct supervision of IFAD

Origin of direct supervision. The Executive Board reviewed the Joint Review on Supervision Issues for IFAD-Financed Projects in December 1996. The report was submitted to the Governing Council in February 1997, together with the Five-Year Plan of Action for the DSPP. The plan12 included the following actions, together with time frames for their implementation by IFAD:

  • "agreement by Governing Council for IFAD to supervise test projects;
  • criteria to be used for selection of test projects to be submitted to Executive Board;
  • 15 test projects to be determined (3 projects per region);
  • identify and negotiate with reputable private national or international organizations to undertake procurement and financial administration;
  • progress report of test projects to be reported to Executive Board;
  • analytical accounting system to be established to track the actual cost of direct supervision as well as supervision by CIs;
  • mid-term review of supervision of test projects to be submitted to Executive Board; and
  • establish monitoring system to evaluate the test projects."

DSPP modality. One important feature of the DSPP has been the cooperation between IFAD and UNOPS. The latter was contracted in July 1998 by IFAD to undertake the supervision of fiduciary aspects (such as procurement of good and services, disbursements, ensuring compliance with auditing and financial requirements) in the context of the DSPP. UNOPS was paid a standard amount equal to USD 12 000 per year per project for their services in the specified areas. IFAD's specific role in the DSPP was thus to arrange and conduct supervision missions, organize the necessary follow-up and provide the overall implementation support required by borrowers and their projects.

Monitoring and progress reporting. IFAD management presented indicators for the measurement of the impact of direct supervision to the Executive Board in April 1999 (see Chapter II, Box 1). IFAD further committed to reviewing the indicators at least one year after most directly supervised projects had become fully operational. For comparative analysis, the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects explicitly requested that a control group of CI-supervised projects, similar in other respects to the directly supervised projects, be identified and monitored.

In terms of reporting, the Governing Council decided that the progress, lessons learned and results of the pilot programme would be reported annually to the Executive Board. Moreover, IFAD would conduct a mid-term review (MTR) of the supervision of test projects, also to be submitted to the Board. In approving OE's work programme for 2004 during its session in December 2003, the Executive Board requested OE to undertake a CLE on the DSPP. Finally, the Governing Council decided that the President shall submit the results of IFAD's experience and conclusions on the DSPP to the Executive Board for its review before the corresponding resolution would cease to be operational (i.e. in June 2006).

Implications of selected new IFAD initiatives. The growing emphasis on ‘country presence'13 will have important consequences on the modus operandi and costs of both direct supervision and supervision by CIs. For instance, IFAD's country presence will have a role in supervision activities, ranging from simply facilitating the organization of supervision missions to the more substantive role of providing backstopping and follow-up on implementation issues. Furthermore, management is developing a proposal for the Fund's new operating model. There are two elements in this model that are likely to have important implications for IFAD's supervision activities. These are: (a) the shift in the unit of account from the project to the country programme level; and (b) the utilization of the Results and Impact Management System (RIMS) to capture and analyse data systematically according to a core set of indicators. IFAD will need to pay attention to these aspects in its future supervision activities.

Direct supervision analysis

Selection of the pilot projects. Most of the 15 projects were selected according to the criteria presented by IFAD to the Executive Board in September 1997. Two projects (in Gaza and the West Bank and Zimbabwe) have been adversely affected by political instability. While the inclusion of the Zimbabwe project in the DSPP was justified, because the unexpected events in the country could not have been foreseen at the time of Board approval, the inclusion of the Gaza/West Bank programme in an unstable country situation did not reflect an appropriate choice for inclusion in the pilot programme. The evaluation also notes that the Bangladesh project was not an appropriate choice in that it was not innovative, replicating broadly the design of a previous IFAD-financed project in the country.

Approaches to direct supervision. There have been various approaches to direct supervision in the 15 pilot projects. These include: (a) intensive CPM involvement in all steps of direct supervision, including fiduciary aspects. In these cases, the role of UNOPS was largely limited to disbursement processing; (b) the CPM completely delegates fiduciary aspects to UNOPS and focuses on implementation support matters, although s/he may not write mission reports and may not be as closely involved in the annual work programme and budget processes; (c) fiduciary aspects are dealt with by UNOPS, and the CPM appoints consultants who would be largely responsible for leading supervision missions and producing the corresponding documentation. The CPM may participate in key stages of the missions. In this approach, the CPM takes a management role and focuses on mission deployment and ensuring that all outputs are produced as required; and (d) this approach is specific to those countries in which IFAD has some form of local representation that plays a part in direct supervision by participating in supervision missions and following up between missions. The distribution of the pilot projects across the four approaches has been fairly evenly spread, with three to four projects each following approaches (a)-(c) and two projects using approach (d).

Implementation of direct supervision. The evaluation reviewed the implementation experience of direct supervision activities as compared with those under supervision by CIs. Its report observes that supervision planning during preparation of country strategic opportunities papers (COSOPs) and the project design phase was given limited attention in both forms of supervision. On another issue, the evaluation notes that the continuity of CPMs in direct supervision activities is important, and that in some pilot projects, there have been more than three CPMs allocated to the project in the last five years. In terms of supervision mission frequency and duration, the directly supervised projects have received close to two supervision missions per year, against one supervision mission for the control group projects. The average mission days per project per year were 15.2 days for direct supervision as compared with 11.2 days for the control group.

The composition of supervision mission teams in the directly supervised projects and in the control group has not been significantly different, with both groups making use of local consultants. There has also been more consultation between IFAD, the country and the project authorities before deciding what expertise to include in each supervision mission, especially in the context of the DSPP. Direct supervision missions on average comprised 2.6 personnel, compared with 2.5 for the control group.

So far, eight of the projects directly supervised have benefited from MTRs, whereas 11 projects in the control group have had an MTR. This is understandable, given that the average implementation period of the directly supervised projects (4.8 years) is less than that of the control group (6.5). In the case of the DSPP, the MTR has offered an opportunity to formalize a number of decisions made during the regular supervision missions that preceded the MTR. That is, given the more constant and intensive interaction between IFAD and the project authorities in directly supervised projects, CPMs have been able to stay on top of implementation issues and make key decisions within the framework of supervision missions themselves, rather than wait until the MTR stage. Hence, the MTR does not appear to have been critical in the context of the DSPP. It has been a critical instrument in CI-supervised projects, as it has provided CPMs the possibility of conducting a comprehensive review of what has been done and of introducing the required corrective measures to project design and implementation arrangements.

The quality of supervision reports varied across the directly supervised projects. In the terms of reference of and reports on direct supervision, attention largely focused on implementation issues and on physical and financial achievements, and less on lessons learned and impact. The same may be said of supervision reports prepared by CIs. However, supervision reports of the DSPP projects provided better coverage of issues of concern to IFAD, such as gender mainstreaming, monitoring and evaluation, beneficiary participation, building of grass-roots institutions and so on. Moreover, the documentation of the directly supervised projects was more complete and more easily accessible at IFAD than that of the control group.

In terms of feedback, the evaluation concludes that the DSPP did not systematically follow the set procedures established for feedback. Often an informal mode of communication was chosen rather than following official channels. In spite of the above, the evaluation notes that there is generally a faster response to project queries and follow-up on supervision recommendations under direct supervision than under CI supervision. The feedback in supervision undertaken by CIs is also erratic. Few CIs produce all the required documentation. The inadequate type and quality of reports and feedback bring up the critical issue of the current lack of quality assurance systems in the overall supervision activities of IFAD, which will be discussed later.

No notable difference in participatory processes was observed between IFAD direct supervision and supervision by CIs. However, there appears to be a clear preference for the increasing trend of conducting joint review missions (as organized by some IFIs) with the governments concerned, as opposed to traditional supervision missions. The term ‘supervision', itself, was described by many partners at the country level as being top-down, one that did not reflect the partnership ethos between the country concerned and IFAD.

Project status reports (PSRs) are an important instrument in IFAD's overall monitoring and reporting system. In PSRs of directly supervised projects, there has been no analysis or lessons on direct supervision processes or information on costs. In addition, PSRs do not include ratings on the IFAD impact indicators for direct supervision, which were developed at the beginning of the DSPP. For CI-supervised projects, CPMs include an assessment of the CI's performance and assign a corresponding rating. For directly supervised projects, PSRs do not include a rating of IFAD's performance in direct supervision. With regard to the process in the preparation of PSRs, it must be noted that the PSRs of CI-supervised projects are done by the CPM concerned, whereas the same CPM responsible for direct supervision prepares the PSR for the project under his/her direct supervision.

One important message emerging from the evaluation is the very wide support by IFAD partners of direct supervision. Government authorities, development organizations and project authorities were all of the unanimous opinion that there were advantages in having a direct contact with IFAD through CPMs. For example, partners conveyed the view that direct communication and interaction with IFAD contribute to better implementation and a stronger partnership, be it in terms of policy dialogue, identification of future pipeline and cofinancing opportunities or knowledge-sharing. There was one case in which a criticism was not of direct supervision, as an approach, but of the frequent staff changes at the CPM level that the project had experienced. The perceived lack of seniority and qualifications of some staff assigned by IFAD for this task were also raised for some projects.

Fiduciary aspects (under the formal responsibility of UNOPS) of directly supervised projects perform better than those of the control group. This finding is consistent with the ESM finding that UNOPS tends to perform better than other CIs. As UNOPS is the only CI involved in the DSPP, it is not unexpected that there would be a higher level of performance than for the control group, which incorporates a variety of CIs.

Project implementation performance. The evaluation first compared project implementation performance using indicators and ratings contained in the PSRs for both the directly supervised projects and the control group. Thereafter, the evaluation used the same set of indicators included in the PSRs and, based on its own independent ratings, compared the implementation performance of the directly supervised projects with those supervised by CIs.

From the evaluation's analysis, it is evident that the directly supervised projects perform better than CI-supervised projects across the PSR indicators in terms of, for example: compliance with loan covenants (directly supervised projects were rated on average 3.4 as against 3.1 for CI-supervised projects);14performance of M&E systems (3.2 for directly supervised projects against 2.6); availability of counterpart funds (3.5 against 3.2) and so on. One explanation might be more optimistic reporting by CPMs, who, as mentioned previously, are themselves responsible for the preparation of the PSRs. However, the evaluation's independent assessment also demonstrates that, on the whole, directly supervised projects have a better implementation performance as compared with CI-supervised projects across the PSR indicators. All rating scores may be seen in Chapter III, Tables 4 and 5.

The comparison of the directly supervised projects with the 15 projects in the control group was also developed by assessing the average time from approval to effectiveness. The analysis reveals more favourable performance by directly supervised projects. That is, the average time lag is 15.36 months for directly supervised projects as compared with 17.21 months for all other projects in the same country. Moreover, in terms of disbursement performance, the average rating by the evaluation team for directly supervised projects is 2.6 as compared with 1.9 for CI-supervised projects in the control group. For example, the average cumulative disbursement rate for all directly supervised projects in the fifth year of project implementation was around 62% as compared with 43% in the control group. Chapter III, Figure 1 provides more information on disbursement performance.

Finally, the evaluation reviews one indicator that is not included in the PSRs: the time overrun factor in project implementation (number of years/months a project is extended beyond the original completion date). Based on the calculations made by the evaluation, the overall time overrun for directly supervised projects is on average 0.54 years, as compared to 1.4 years for projects in the control group. Time overrun is an important indicator, as it reflects the soundness of design, but also the ability of the supervision process to recommend timely corrections and improvements during implementation, as and when required. Time overrun is also significant because it has an administrative cost implication for the Fund: additional supervision costs and related staff time need to be allocated for the time that projects run beyond their original completion date.

The human (social) dimension in supervision and attention to key areas of IFAD's catalytic role. There is evidence based on the evaluation's analysis that through direct supervision, as compared with supervision by CIs, IFAD has paid more attention to issues such as the targeting of women (rated 3.1 for directly supervised projects, compared with 2.5 for projects supervised by CIs); targeting the poor (rated 3.0 for directly supervised projects against 2.4); beneficiary participation; gender mainstreaming; participatory monitoring and evaluation and so on. These trends are by and large consistent with the ratings included in the PSRs of the directly supervised and control group projects. Direct supervision has also provided IFAD an opportunity to focus more on issues such as innovation (rated 3.2 against 2.5 for CI-supervised projects), partnership, knowledge management and policy dialogue, which according to the IFAD strategic framework contribute to improving project performance and impact. In each country exposed to direct supervision, according to the evaluation and partners at the country level, the presence of the CPM during supervision is seen as an opportunity to advance IFAD's broader objectives, such as those listed above. Chapter III, Figures 2 and 3 provide all comparative rating scores on the aforementioned indicators for the DSPP and projects supervised by CIs.

The evaluation's rating for knowledge management of directly supervised projects (rating 2.7) is only marginally better than that of projects in the control group (2.5). This is partly explained by the fact that, while the knowledge acquired through direct supervision at the individual CPM level was high, there was no systematic effort to document, analyse or share such learning from direct supervision. Moreover, no specific activities were conducted or resources allocated to knowledge management in the directly supervised projects, and their supervision reports did not emphasize lessons learned. The same is true for the projects in the control group. However, this is considered a particular shortcoming in the case of direct supervision, as knowledge management in the broader sense was a specific objective of the DSPP (which is not the case for CI supervision).

Development effectiveness. The first thing to note is that, contrary to the requirement of the action plan adopted by the Governing Council, no specific, ongoing monitoring and reporting mechanisms to trace project impact were established, neither at the individual project level nor for the DSPP as a whole. In addition, contrary to the recommendation in the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects, "for comparative analysis a control group of CI-supervised projects similar in other respects to the directly supervised projects" was not identified and monitored. Nevertheless, the evaluation made efforts to analyse the development effectiveness of the directly supervised projects and compare them with projects in the control group.

The evaluation has undertaken three specific types of comparison that, taken together, provide an overview of the development effectiveness of the directly supervised projects in relation to the CI-supervised projects. Tables 6, 7 and 8 in Chapter III contain the specific rating scores that provide the basis for the analysis.

First, based on three indicators (quality of supervision recommendations, follow-up actions and support provided to the project), the overall quality of supervision was assessed (see Chapter III, Table 6). These indicators are also used by the World Bank's Quality Assurance Group in determining the quality of the Bank's supervision, which is considered an important ingredient in achieving development effectiveness. In fact, in one of their recent evaluations, the Operations Evaluation Department of the World Bank concluded that well-supervised projects are twice as likely to succeed as are poorly supervised ones. The DSPP evaluation's assessment according to the three indicators reveals a positive trend in favour of directly supervised projects. For example, quality of supervision recommendations is rated 3.3 for DSPP, compared with 2.9 for CI-supervised projects.

Second, the evaluation compared directly supervised and control group projects using the 11 IFAD impact indicators for the DSPP.15 By and large, the directly supervised projects do better than those supervised by CIs across all indicators (see Chapter III, Figure 4). For example, "identification of new project concepts for inclusion in the pipeline" is rated 2.9 for DSPP as compared with 2.1 for CI-supervised projects. However, the control group projects perform marginally better in terms of timeliness of reporting (rated 2.9 for projects supervised by CIs, compared with 2.8 for DSPP). The reason is partly that, as subcontractors, CIs pay specific attention to project reporting. This issue has not been given the same level of attention in direct supervision, given the constant contact between IFAD and the project authorities. It can also be seen that the ratings for costs appear to favour CI supervision, as supervision by CIs costs less on average (this will be discussed in more detail later).

Third, the evaluation rated the directly supervised and CI-supervised projects using the six impact domains in the OE methodological framework for project evaluation (MFE). Moreover, the two overarching factors in the MFE (sustainability and innovation/replication) have been included in the analysis (see Chapter III, Table 7). In short, the directly supervised projects are rated better in most impact domains (e.g. food security is rated 2.9 for DSPP, compared with 2.5 for CI-supervised projects), although in two impact domains (environment and sustainability), the directly supervised projects performed less well as compared with the CI-supervised projects.

With regard to the latter issue, it should be noted that 12 of the 15 projects in the control group included a specific objective or component related to environmental matters. Consequently, their supervision missions have normally adequately reviewed project progress in this area. On the other hand, only five directly supervised projects have a specific environmental component and their supervision missions have not always included the required skills to undertake a thorough assessment of environmental issues and to provide the necessary backstopping to project staff. Furthermore, environmental matters have not received the same level of focus within IFAD at large as compared with areas such as gender mainstreaming and social capital formation.

No conclusive statement can be made on project sustainability at this stage. On the one hand, the slightly less positive ratings of the directly supervised projects as compared with the control group may be explained partly by the relatively fewer years of implementation of the directly supervised projects. That is, many of the projects in the DSPP have devoted greater emphasis, at least in the initial years, towards developing grass-roots organizations and promoting participatory processes and less towards productive activities. This may be a cause of the lower sustainability rating at this stage. Moreover, some of the directly supervised projects (in Brazil, India and the Sudan, for example) suffered initial delays in implementation due to compelling political and administrative circumstances, which could be another reason. On the other hand, the IEE concluded that during the early stages of a project there are relatively high expectations that project benefits will endure, but these expectations are modified in the later stages of implementation.

Finally, the overall composite rating16 of the six impact domains is compared with the rating available in the PSR on ‘meeting development objectives'. In both cases, directly supervised projects do better than CI-supervised ones (see Chapter III, Table 8). The MFE composite index is rated 2.9 for the DSPP as compared with 2.5 for CI-supervised projects.

Some general considerations must be made while interpreting the above results. First, it must be acknowledged that the differences in rating scores are relatively small. Nevertheless, the analysis shows that directly supervised projects have performed better, compared with the CI-supervised ones, according to most indicators considered in this evaluation. Additionally, direct supervision has contributed to enhanced results in furthering IFAD's broader objectives, such as policy dialogue, partnership-building and knowledge management. The evaluation also notes that the results of direct supervision could have been even greater had the pilot programme been implemented under more favourable conditions, for example if the CPM workload had been appropriately prioritized or if adequate monitoring and assessment systems had been put in place. However, in analysing the results, one must also take into account the time allocated to direct supervision. More than half the CPMs involved in DSPP said they spend up to double the time on direct supervision than on other projects in the portfolio. There are also issues of cost, which will be discussed later.

Quality assurance. IFAD lacks a continuous quality assurance system for supervision, which would have allowed the Fund to meet more fully the objectives of the pilot programme. As a result, there was limited quality assurance in direct supervision inputs and processes (e.g. mission terms of reference, composition and duration in the field) and in deliverables such as supervision reports. Other IFIs, in particular the African Development Bank (AfDB) and the World Bank, also have quality assurance mechanisms that allow for periodic assessment of supervision activities (e.g. once every two years). Through these quality assurance mechanisms, IFIs are able to take a holistic view of the supervision function and suggest corresponding systemic improvements across the organization. Such quality assurance mechanisms in the aforementioned IFIs are located outside the operations departments. The Inter-American Development Bank (IDB) is in the process of setting up a similar quality assurance system. In addition, IFIs generally have quality assurance mechanisms for supervision built into the operations divisions (e.g. at the African Development Bank, peer reviews are used for quality assurance).

Reporting to the Executive Board. Management has not complied with two key commitments in terms of reporting. First, no MTR of the pilot programme was undertaken in the last quarter of 2000/first quarter of 2001, as had been decided by the Governing Council. The reasons for this are neither evident nor documented. Second, although the Fund has provided progress reports on direct supervision to the Board on an annual basis, these have focused mainly on the deployment of inputs and processes in implementing direct supervision activities. The evaluation notes that, at the same time, the Executive Board could have exercised better oversight of implementation of the action plan and could have demanded more analytical information from management on the development effectiveness of the DSPP, in particular on the costs, results and lessons learned of the pilot programme.17

Operating environment for direct supervision. CPMs did not benefit from a favourable operating environment in which to manage the increasing workload caused by the introduction of direct supervision. More specifically, training was not provided at the outset of the pilot programme, nor were CPMs given any particular recognition or incentives. They accepted the additional responsibilities for direct supervision, which they undertook without managerial support for any reprioritization of their existing workload. Last, but equally important, management engagement appeared to diminish gradually after approval by the Governing Council of the DSPP, as demonstrated, for example, by the non-implementation of three key activities18 included in the action plan.

Learning and knowledge management

By participating in direct supervision activities, CPMs have acquired first-hand experience of the task of supervision and a better appreciation of the issues related to rural poverty reduction. The benefits of their experience are evident, as some CPMs have put their newly acquired knowledge to use in designing new projects and implementing ongoing ones. Still, supervision reports (of both direct supervision and CIs) focus more on implementation issues and less on lessons learned from supervision processes or broad rural development issues. This has led to missed opportunities to incorporate learning into project design across the institution and to improve the supervision activities of cooperating institutions in general. Nonetheless, in spite of the benefits to individual CPMs, there is a need to institutionalize this knowledge and establish systems/platforms through which CPMs involved in direct supervision could share their overall experiences among themselves and within the Fund. Some exchanges have occurred at the CPM level in an informal, unstructured manner. But, overall, institutional support to capture and channel learning from direct supervision to IFAD staff has been inadequate.

IFAD also lacks an adequate reporting and feedback mechanism at the country level: its current system for learning does not allow governments and other development organizations to become familiar with its successful innovative approaches and to learn from its experiences. This needs to be addressed in the light of the Fund's strategic objective of seeking partnerships for replicating and scaling up the activities it finances, as well as of the proposed new operating model. OE recognizes that knowledge management is an institution-wide concern and that direct supervision is only one of the components of the Fund's learning system. However, the unique opportunity for the institution as a whole to learn from the direct supervision pilot programme to enhance IFAD operations has been largely missed.

Efficiency of direct supervision

Based on the calculations of the evaluation, the average cost per year per project of direct supervision is around USD 93 000 as compared with USD 61 000 for CI-supervised projects in the control group. In this regard, it should be noted that the costs incurred go beyond the expectation of the Governing Council, which had decided that in implementing direct supervision activities "there would be no cost increase" to IFAD.

However, there are some points related to the average CI cost that merit being highlighted. First, there is quite a variation in costs across the different CIs. Hence, the average cost for CI supervision calculated above does not reflect the costs that all CIs charge. For instance, supervision through some CIs (e.g. the World Bank) costs over USD 100 000.19 The current overall cost of supervision through UNOPS (the CI with the greatest number of projects under supervision) is around USD 79 000 – see Chapter V for more data on costs. Moreover, according to the evaluation, the current cost attributed to UNOPS needs to be increased if the Fund is to expect them to provide enhanced quality services.20 Next, there are additional costs of CI supervision that are difficult to identify and have thus not been included in calculating the total cost of CI supervision. For instance, the Belgian Survival Fund has an annual allocation for CI supervision in its administrative budget, recorded outside IFAD's accounting system. Along similar lines, more recently, IFAD's Asia and the Pacific Division provided additional funds to the UNOPS Asia Office to augment its supervision-related activities. These and other such costs are not included in the average of USD 61 000 calculated for CI supervision. Finally, it should also be noted that the cost to IFAD for direct supervision includes the fielding of around two supervision missions per year to each project for longer durations than those of the average CI supervision activities.

Furthermore, the evaluation underlines that the longer implementation period of projects in the control group as compared with the directly-supervised projects has administrative cost implications for IFAD, which would raise the overall costs of supervision by CIs and which need to be considered.

The evaluation also argues that there are possibilities of reducing the costs of direct supervision to some extent. For instance, this could be achieved by making use of competent national entities to discharge the fiduciary responsibilities involved in supervision and thus enhancing the role of government in implementation support activities. Greater use of local consultants could also contribute to cost savings. Finally, as CPMs and the other IFAD staff involved acquire the necessary competencies and gain more experience in direct supervision, it is fair to assume that the overall time invested in the associated tasks is likely to reduce to some extent. This will have a corresponding effect on the staff costs component, leading to a reduction in the overall costs of direct supervision.

While the cost of direct supervision may still be higher than that of CI supervision, according to the evaluation it is paramount to assess the cost together with the corresponding benefits to the Fund. The analysis of the evaluation illustrates that direct supervision has contributed to better development effectiveness and has allowed the Fund to further its catalytic objectives of innovation, policy dialogue and partnership development. With regard to the latter point, partners at the country level expressed their preference for building partnerships directly with IFAD rather then managing such processes through proxy institutions such as CIs. Moreover, in the context of knowledge management, although the CPMs involved in direct supervision have acquired better understanding of implementation matters and despite the fact that knowledge from the DSPP has not been properly institutionalized, the evaluation observes that using a CI for supervision introduces an extra layer into the already feeble learning loop of the Fund. On a similar note, the knowledge that the staff or consultants of CIs have acquired by undertaking supervision on behalf of IFAD is largely lost to the Fund. Finally, the common opinion of governments and all other partners at the country level, who clearly favour IFAD's direct involvement in supervision activities, must also be given due consideration.

Supervision systems and experiences of other International Financial Institutions

By and large, the AfDB, the Asian Development Bank (AsDB), the IDB and the World Bank have similar supervision systems, with processes and details applied variously to fit their institutional structures. In the IFIs, generally, supervision is not limited to official missions and formal reports, rather it is a continuing and flexible process, specific to the needs of the particular operation and intended to foster a close partnership among an IFI, its borrower, and the implementing agencies. The planning of supervision is done carefully during the design phase of the project cycle, when appropriate resources and arrangements are put in place to facilitate supervision. The supervision process normally starts after the project is approved and ends when the last disbursement is completed and the project completion report is prepared.

At the Inter-American Development Bank, supervision is largely delegated to the country offices, with participation of staff from headquarters as required. At the Asian Development Bank and the World Bank, headquarters and country offices share responsibility for supervision. That is, in some cases, supervision responsibilities are entirely delegated to the country offices, especially in those countries in which offices have staff with the required sectoral know-how. Under such arrangements, selected staff at headquarters are sometimes asked to join supervision missions. In other cases, the "task managers", if based at headquarters, retain responsibility for supervision and, in turn, involve staff at their country offices in supervision missions. At the African Development Bank, so far, task managers based at headquarters have full responsibility for project supervision. However, with the establishment of 26 country offices by the end of 2006, AfDB expects in-country staff to be involved in one way or another in project supervision and related follow-up, as well as in portfolio management issues. One important aspect of supervision by these IFIs is that certain supervision functions are being increasingly located in their country offices. For example, the review of documents related to fiduciary aspects (such as bidding proposals and accounting matters) is largely handled by the country offices. It is important to realize, however, that the degree to which such field office staff can make decisions varies according to the delegation provided by the responsible task managers. Finally, some IFIs have well-established quality assurance mechanisms for supervision and others are rapidly moving in the same direction.

Conclusions

A consistent trend in the overall evaluation analysis demonstrates that, compared with CI supervision, direct supervision has greater potential to contribute to better development effectiveness at the project level and, at the same time, allows greater attention to IFAD's broader objectives at the country programme level. That is, direct supervision by IFAD can contribute to better and timelier project implementation, which in turn enhances overall results and impact. Moreover, through direct supervision, IFAD has been able to emphasize issues of prime concern such as gender mainstreaming, targeting and the building of grass-roots institutions, which taken together are important elements in ensuring sustainability.

In view of their more frequent and longer presence in countries with direct supervision, CPMs have wider opportunities to advance IFAD's objectives at the country programme level, including policy dialogue and partnership development. CIs do not consider these to be a priority, nor can such activities be effectively advanced through third parties. Although there is no conclusive evidence that new partnerships have resulted from the DSPP, governments and other development partners at the country level have unanimously expressed their deep appreciation of the more frequent contact with CPMs, which has been facilitated by IFAD direct supervision activities. The same partners communicated that they find it more useful to deal directly with IFAD staff rather than with CI representatives. In this regard, for example, partners affirmed that the response rate and follow-up on implementation issues are faster under direct supervision than under supervision by CIs.

Direct supervision has contributed to developing IFAD's knowledgebase. In this regard, in particular, the CPMs responsible for direct supervision have acquired knowledge of supervision processes, project implementation and rural development issues in the countries concerned. This knowledge has enabled them to better design and implement new operations. However, the knowledge gained at the CPM level has not been systematically shared with others, nor has it been institutionalized, which is one of the key shortcomings of the DSPP. The evaluation notes that supervision by CIs also offers possibilities for knowledge generation. However, CI involvement in supervision makes the transmission line of knowledge from the CI to IFAD and the country more cumbersome in the already feeble knowledge systems of the Fund.

The evaluation concludes that direct supervision allows CPMs to strengthen country-level coordination both within the context of IFAD operations and with the development community at large. It also facilitates the strengthening of existing IFAD-funded programmes and the identification of new programmes and cofinancing opportunities, which are mostly available at the country level, given that the majority of our international and bilateral partners have delegated an increasing amount of authority to their country representatives.

Unlike most other IFIs, IFAD lacks a quality assurance system for direct supervision. As a result, the DSPP was approached and implemented in a variety of ways, based on the perception and understanding of individual CPMs. Compliance with the minimum supervision requirements and the direct supervision guidelines was also not monitored. In conclusion, both continuous and periodic quality assurance systems are fundamental if direct supervision activities are to be expanded.

At face value, the average cost of direct supervision (USD 93 300) is higher than the average cost of supervision by CIs (USD 61 461)21. However, the evaluation argues that cost should not be seen in isolation from the benefits the DSPP has evidenced. Moreover, in discussions with UNOPS (the main IFAD CI), it is clear that the costs to IFAD for supervision by UNOPS need to be increased if they are to deliver the type and quality of service IFAD requires in the future. In parallel, the evaluation notes that there is the potential for efficiency gains in direct supervision if, for example, the fiduciary responsibilities related to supervision are entrusted to competent national entities and greater use is made of local consultants for implementation support activities.

The evaluation notes that management's interest has appeared to gradually diminish following approval of the DSPP by the Governing Council. This is illustrated by the fact that management did not fully implement all decisions of the Governing Council. For instance, it did not undertake a mid-term review of the DSPP as required. Neither did it establish an integrated, analytical accounting system to track costs of the DSPP. Nor did it set up a monitoring and assessment system to measure the performance and impact of direct supervision. According to the evaluation, however, neither did the Executive Board exercise adequate oversight to ensure that IFAD management would fulfil all its commitments under the DSPP. The evaluation believes that the outcome of the DSPP would have been even more significant had all the requirements laid down by the Governing Council been implemented.

The evaluation reveals that not all concerned have the same understanding of the notion of supervision. In fact, there is often confusion as to what constitutes supervision missions, implementation support, follow-up activities, fiduciary responsibilities and so on. There is also a lack of clarity on the roles and responsibilities of IFAD, CIs, project staff and government authorities. Additionally, it is worth noting that many partners at the country level felt that the term ‘supervision' – when applied to the implementation aspects of projects rather than to fiduciary aspects – has a paternalistic undertone and they felt uneasy with its continued application. Supervision in that sense reflects a top-down, non-participatory approach to the function, which is inconsistent with the Fund's objectives of promoting ownership and partnership with governments and other institutions.

Recommendations

It is important to state that the evaluation's recommendations have taken into consideration the relevant recommendations contained in the ESM report. The evaluation's five key recommendations, given below, are mutually reinforcing. The evaluation recommends that they all be implemented fully in order to ensure that the desired impact is achieved in IFAD's future efforts in the area of supervision and implementation support.

Recommendation one: definition of supervision

The evaluation recommends that the concept of ‘supervision' as used by IFAD be divided into two distinct operational parts: (i) supervision of fiduciary aspects, including aspects related to procurement review, disbursement processing and compliance with financial and auditing requirements; and (ii) support to programme and project implementation.22 This would include, for example, the organization of periodic ‘implementation support' missions and related follow-up; an assessment of the achievement of programme/project objectives and assistance in identifying remedial solutions for implementation challenges, based on interaction and dialogue with project authorities and other partners at the country and project level; and the provision of guidance in preparing the annual work plans and budgets. It would also include oversight of project and programme implementation, for example, in terms of monitoring the achievement of physical targets.

Recommendation two: develop a comprehensive supervision23 and implementation support policy for IFAD

The Fund should develop a specific overall supervision and implementation support policy for its operations. The policy should reflect the following elements.

Supervision of fiduciary aspects. IFAD should be allowed to decide, on a case by case basis, whether to subcontract a competent national, regional or international entity to perform such functions. Special efforts should be devoted to engaging national entities, as this would have the double effect of building local institutional capacity and reducing costs. The utmost attention should be given to ensuring that there is no conflict of interest between the prospective national entity and the IFAD operation under consideration. In a few and very specific circumstances, IFAD might consider undertaking the supervision of fiduciary aspects itself.

Implementation support. The evaluation recommends that the policy state explicitly that: (i) IFAD should be made responsible for providing direct implementation support to all its operations globally. In this regard, it is important to specify the role of the CPMs, who could either be intensively involved themselves as "implementation backstoppers" or act as implementation-support task managers with more attention to process management, which would also require a degree of direct involvement in activities. In both cases, CPMs would ultimately be responsible for the process, content and outputs of implementation support activities; (ii) such support would cover all aspects of IFAD country programmes, both at the project level and beyond, such as policy dialogue and partnership strengthening; and (iii) the role of partner governments would be specified and given due emphasis, which would contribute to building greater ownership and local capacities, as well as reducing costs.

The above would result in new responsibilities for the Programme Management Department (PMD), which would require the allocation of additional staff and financial resources as well as new competencies and skills. It is fundamental that the additional resource requirements for the implementation of the new policy be clearly articulated in a comprehensive and explicit manner by IFAD. This would require a detailed cost analysis, particularly of the elements in paragraphs 65-66, as well as an assessment of the skills and competency of current CPMs. Based on the aforementioned analysis, IFAD would need to develop a proposal to meet the cost deficits in implementing the new policy. It would also need to develop a plan for enhancing the ability of CPMs to meet the specific requirements of the new policy, recognizing that it may not be possible to enhance these skills and competencies in all cases. Until the required level of financial resources would be made available to the Fund and additional PMD staff recruited and their competencies and skills developed and upgraded, IFAD might consider a phased approach to expanding implementation support in all operations.

The success of the new policy would also be determined by the support provided by IFAD management and the conducive environment it creates for the purpose. For example, management would need to: (i) ensure that appropriate opportunities are introduced for periodic staff training; (ii) establish an incentive framework and platforms for the sharing of knowledge acquired by CPMs; and (iii) allocate the necessary time for reviewing implementation experience under the policy. The Board should also play a proactive role in exercising oversight of implementation of the new policy and in approving administrative budgets for the purpose.

The policy should be evaluable, and in particular include a roll-out and implementation plan, with performance indicators that can be monitored periodically.

A chapter on supervision and implementation support should be included on a standing basis in IFAD's annual Portfolio Performance Report. The chapter would provide an analytical account of the opportunities and challenges in the area, and identify key lessons learned. Moreover, it would provide an indication of the ongoing operational measures introduced by IFAD to address emerging issues.

The undertaking of supervision of fiduciary dimensions and implementation support would require revisiting Article 7, Section 2(g) of the Agreement Establishing IFAD.

Other integral aspects that the policy should consider are contained in recommendations three to five.

Recommendation three: supervision and implementation support in the framework of the COSOP

The evaluation recommends development of an overall approach to supervision and implementation support during the preparation of COSOPs. This would take into account the need to supervise the fiduciary dimensions of all operations, and the provision of implementation support to the country programme, including areas such as policy dialogue, partnership-building and knowledge management, in addition to the support traditionally provided to projects. The COSOP would lay the provisions for the need to develop an annual supervision and implementation support plan for each country, indicating the specific objectives, human and financial resource allocations and expected results. Each PMD regional divisional would set up an electronic monitoring, assessment and reporting system, which would serve as a management tool to track the implementation progress of the plans and to flag issues requiring more immediate follow-up. A template should be developed by PMD for the section on supervision and implementation support that would be included in the COSOPs.

Recommendation four: quality assurance system

Within the framework of an overall enhanced quality assurance system at IFAD, there is a need to introduce quality assurance mechanisms for the supervision of fiduciary dimensions and implementation support activities. The evaluation recommends that IFAD establish a management review committee within PMD, which would review supervision and implementation support activities, results and related operational issues. Quality assurance, as well, needs to be strengthened in the PMD divisions. In this regard, semi-annual reviews of supervision and implementation support activities should be undertaken at the divisional level. Summaries of the discussions at these meetings should be circulated to all PMD divisions. Moreover, IFAD should build on the experience of other IFIs (in particular of the Quality Assurance Group at the World Bank) to establish an IFAD-specific quality assurance group, which would review aspects of supervision and implementation support, in addition to any other aspects related to implementation of the COSOP and its components. The Fund would need to thoroughly reflect upon the most appropriate location within IFAD's organizational structure for such a group, which would ensure the most objective and independent review possible of its supervision and implementation support efforts. The introduction of such a quality assurance group should take into consideration the mandates and performance of existing quality control systems within IFAD, such as the project development team, Technical Review Committee and Operational Strategy and Policy Guidance Committee.

Recommendation five: learning and knowledge management

Incentives should be provided to staff to encourage sharing of the knowledge they acquire. For example, time needs to be carved out in CPMs' workload for sharing the knowledge they generate through supervision and implementation support activities. The documenting and sharing of knowledge should be included as an indicator in assessing the annual performance of CPMs. Specific instruments need to be established to facilitate learning and knowledge-sharing. In particular, time should be reserved on a standing basis in the CPM forum for discussing issues and sharing knowledge generated through supervision-cum-implementation support activities. Each project mid-term review and project completion report should include a specific treatment of supervision and implementation support issues, as should all evaluations undertaken by OE. The project and country status reports should be reformatted to include a narrative section on supervision and implementation support, and ratings must be included in all cases. Other instruments should be introduced, such as peer reviews at the PMD divisional level in relation to implementation support activities.

The monitoring and evaluation systems at the project level need significant strengthening if they are to contribute effectively to learning. Moreover, in line with the new operating model, it is necessary to assist in the development of integrated monitoring and evaluation systems at the country level. This would not only facilitate monitoring and sharing of experience across the entire project/programme portfolio, but would also allow for tracking the implementation of the broader objectives of IFAD country programmes, such as policy dialogue and partnership-building.

IFAD should build on the experiences of other international financial institutions, and make more comprehensive use of information technology for knowledge management purposes in relation to supervision and implementation support. In this regard, it is recommended that the existing PPMS be expanded so that it can carry updated summaries of supervision and implementation support activities at all times. An enhanced PPMS should accordingly be made accessible to external partners through the IFAD internet with immediate effect. Last but not least, an integrated, analytical accounting system should be developed through close cooperation between PMD and the Office of the Controller. This would allow the monitoring and analysis of all costs related to supervision-cum-implementation support, including staff time.


1/ The resolution will cease to be effective in June 2006, as the last project (India) became effective in June 2001.

2/ For the purpose of this evaluation, the term ‘development effectiveness' encompasses the extent to which the DSPP's overall objectives have been met, the efficiency in implementing the pilot programme and the contribution of direct supervision in improving project implementation and potential project impact.

3/ For definitions of the criteria, see Methodological Framework for Project Evaluation (document EC 2003/34/W.P.3).

4/ These impact indicators were developed and presented by IFAD management to the Executive Board in April 1999 (see Box 1 of the present document and document EB 99/66/R.10/Rev.1).

5/ In Armenia, the Gambia and Uganda.

6/ In Bangladesh, Benin and Indonesia.

7/ In Armenia and India.

8/ In Bangladesh, Guinea, Indonesia and Peru.

9/ The role of the CLP was to provide comments and input at several key stages in the evaluation process (for a definition of the CLP, see paragraph 33 in the IFAD Evaluation Policy, document EB 2003/78/R.17/Rev.1). The members of the CLP for the DSPP evaluation were: Mr Nigel Brett, Mr Jim Carruthers, Mr Pablo Glikman, Mr Shyam Khadka, Mr Luciano Lavizzari, Mr John McGhie, Mr Ashwani Muthoo, Ms Rasha Omar, Mr Mohamed Tounessi and Mr Joseph Yayock.

10/ Mr Hans Wyss, former Director of Operations at the World Bank.

11/ Hence field work was undertaken in 13 countries of the DSPP (Gaza and the West Bank and Zimbabwe were excluded).

12/ The action plan can be seen in its original format in Appendix II.

13/ In December 2003, the Executive Board approved the field presence pilot programme in 15 countries globally.

14/ The rating scale used is from 1 to 4, where 1=negligible, 2=modest, 3=substantial and 4=high.

15/ The list of impact indicators may be seen in Chapter II, Box 1.

16/ The composite rating is the average of the six impact domains and the two overarching factors.

17/ In 1997 the GC had, indeed, decided that "progress, lessons learned and results of the test would be reported annually to the Executive Board" – see document GC 20/L.10/Add.1.

18/ Items 6, 7 and 8 under Recommendatio 5 in the action plan – see Appendix II.

19/ A study by the World Bank's Quality Assurance Group in 2003 concluded that, on average, from USD 100 000 to USD 125 000 should be allocated per year for supervision of community-driven development projects.

20/ The IEE report highlighted the same issue – see paragraph 18 on page 16 (document EB 2005/84/R.2).

21/ Although the actual cost per project per year for supervision by UNOPS, the main IFAD CI, is around USD 79 000.

22/ It could also be called ‘implementation support'.

23/ The term ‘supervision', from this point onwards in the document, means ‘supervision of the fiduciary aspects' related to IFAD financing.

Background. In February 1997, the IFAD Governing Council adopted resolution 102/XX on Loan Administration and Supervision of Project Implementation, together with a Five-Year Plan of Action. The resolution stated that IFAD "may supervise specific projects and programmes financed by it". According to the action plan, no more than 15 IFAD-initiated projects were to be directly supervised and administered during the five-year period. This initiative, including the 15 projects, has since been referred to as the Direct Supervision Pilot Programme (DSPP). The Governing Council resolution entered into effect on 21 February 1997 and will cease to be operational five years after the date of effectiveness of the last approved project to be directly supervised by IFAD1. The Governing Council also decided that the DSPP should not entail any cost increases and that IFAD should use the same funds that would have been spent on supervision by cooperating institutions (CIs) to cover its direct supervision activities.

In order to assess the results of the action plan, in 2002/2003 the Office of Evaluation (OE) undertook an Evaluation of the Supervision Modalities in IFAD-Supported Projects (ESM). The objectives of the ESM were to "evaluate the effectiveness of current supervision modalities against the minimum supervision requirements [MSRs] … and other indicators of quality, and review the achievements under the Five-Year Plan of Action". It is important to note that most of the directly supervised projects were in their early stages of implementation at the time of the ESM. Hence, although the ESM provided concrete findings on supervision through CI, it provided only an overview of the emerging characteristics of IFAD's direct supervision efforts.

DSPP objectives. The overarching objective for embarking on the DSPP was to enable the Fund to acquire first-hand knowledge from supervision activities and to incorporate lessons learned from ongoing operations more effectively into its project design work. It was also to provide IFAD with "knowledge of the supervision function, of what are the costs of an adequate project supervision and of the development impact and human dimension of the projects in its portfolio. The Fund's involvement in direct supervision would also complement and improve cooperating institutions' own activities (mainly the human dimension of projects/programmes)". Although enhancing development effectiveness2 was not an explicit objective, direct supervision was expected to contribute to improving implementation performance and project impact.

Evaluation objectives. The main objective of the corporate-level evaluation (CLE) was to make an overall assessment of the DSPP's achievement in enhancing the implementation and impact of IFAD-funded operations. More specifically, the evaluation had the following key objectives: (a) compare and contrast direct supervision by IFAD with selected, relevant examples of supervision undertaken by CIs; (b) examine the processes established, alternative approaches and the experiences of country programme managers (CPMs) in undertaking direct supervision. This would include comparison with the approaches, systems and experiences of other international financial institutions (IFIs); (c) assess the efficiency of the direct supervision modality; and (d) examine the systems established to capture the experience and insights from direct supervision and the ways in which this has been of benefit to IFAD's project design processes and implementation support activities.

Evaluation methodology. The approach for the direct supervision evaluation has been to build on the methodology and results of the ESM. The criteria3 used in the CLE to assess the DSPP include: (a) the relevance of direct supervision in relation to the DSPP's objectives at the time of approval but also in today's context; (b) the effectiveness of the DSPP, measured against the achievement of the stated objectives of the programme and using the indicators specified for measurement of the impact of direct supervision;4 (c) the efficiency of direct supervision; and (d) an analysis and comparison of the actual and potential impact of the projects included in the direct supervision pilot programme with those supervised by CIs.

The evaluation process was planned to allow for triangulation of evidence and the views obtained from the main actors in the DSPP (the key government focal point at the national level, together with implementing agencies, beneficiaries, and the IFAD staff concerned). Moreover, OE had completed evaluations of three directly supervised projects5 in the past few years, which provided valuable additional sources of information and assessments. Since 2003, OE has also undertaken three country programme evaluations6 that included assessment of one directly supervised project in each of these countries. Relevant information and reports from the Independent External Evaluation (IEE) of IFAD, which analysed in detail two directly supervised projects7, have also been utilized. Finally, four projects8 in the evaluation control group have either been evaluated by OE or included in the IEE.

Evaluation scope. The evaluation involved a comparison of the directly supervised projects with 15 projects supervised by CIs (the latter therefore represented the evaluation's control group). That is, three CI-supervised projects per region were included in the control group based on a set of selection criteria agreed with the evaluation's core learning partnership (CLP)9. The list of projects included in the DSPP and the control group for the purposes of the evaluation can be seen in Appendix I.

Evaluation process. The CLE process began with the preparation of an approach paper, which provided an opportunity for developing a solid understanding of the evaluation's objectives, scope, methodology, time frames and expected outputs. The evaluation benefited from the views of the CLP. An external reviewer10 with wide experience in project supervision issues was contracted to advise OE at critical phases during the evaluation and to review key evaluation outputs. Moreover, OE undertook a thorough peer-review process within the division to improve the evaluation's overall quality.

The evaluation included the following activities: desk reviews of the 15 directly supervised projects and 15 projects in the control group; interviews with IFAD management and staff; field work in 13 of the 15 directly supervised projects and in eight control group projects11; discussion with the United Nations Office for Project Services (UNOPS) and selected IFIs at their headquarters and at the country level; direct supervision cost analysis; preparation of an early feedback note on an experimental basis, with the objective of sharing the emerging evaluation results and sensitizing IFAD management before the draft evaluation report was ready for discussion; organization of a round-table workshop to discuss the evaluation's results and lay the foundations for the agreement at completion point of the evaluation; and discussion at the September 2005 sessions of the IFAD Evaluation Committee and Executive Board.

The draft evaluation report was shared for review and comments with: staff in all the projects included in the DSPP and the control group, IFAD management and other staff, selected IFIs and CIs, and the government officials concerned at the national level in countries in which IFAD has funded a directly supervised project. Their comments have been included in accordance with the provisions in paragraph 42 of the IFAD Evaluation Policy.

The evolution of direct supervision of IFAD

Origin of direct supervision. The Executive Board reviewed the Joint Review on Supervision Issues for IFAD-Financed Projects in December 1996. The report was submitted to the Governing Council in February 1997, together with the Five-Year Plan of Action for the DSPP. The plan12 included the following actions, together with time frames for their implementation by IFAD:

  • "agreement by Governing Council for IFAD to supervise test projects;
  • criteria to be used for selection of test projects to be submitted to Executive Board;
  • 15 test projects to be determined (3 projects per region);
  • identify and negotiate with reputable private national or international organizations to undertake procurement and financial administration;
  • progress report of test projects to be reported to Executive Board;
  • analytical accounting system to be established to track the actual cost of direct supervision as well as supervision by CIs;
  • mid-term review of supervision of test projects to be submitted to Executive Board; and
  • establish monitoring system to evaluate the test projects."

DSPP modality. One important feature of the DSPP has been the cooperation between IFAD and UNOPS. The latter was contracted in July 1998 by IFAD to undertake the supervision of fiduciary aspects (such as procurement of good and services, disbursements, ensuring compliance with auditing and financial requirements) in the context of the DSPP. UNOPS was paid a standard amount equal to USD 12 000 per year per project for their services in the specified areas. IFAD's specific role in the DSPP was thus to arrange and conduct supervision missions, organize the necessary follow-up and provide the overall implementation support required by borrowers and their projects.

Monitoring and progress reporting. IFAD management presented indicators for the measurement of the impact of direct supervision to the Executive Board in April 1999 (see Chapter II, Box 1). IFAD further committed to reviewing the indicators at least one year after most directly supervised projects had become fully operational. For comparative analysis, the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects explicitly requested that a control group of CI-supervised projects, similar in other respects to the directly supervised projects, be identified and monitored.

In terms of reporting, the Governing Council decided that the progress, lessons learned and results of the pilot programme would be reported annually to the Executive Board. Moreover, IFAD would conduct a mid-term review (MTR) of the supervision of test projects, also to be submitted to the Board. In approving OE's work programme for 2004 during its session in December 2003, the Executive Board requested OE to undertake a CLE on the DSPP. Finally, the Governing Council decided that the President shall submit the results of IFAD's experience and conclusions on the DSPP to the Executive Board for its review before the corresponding resolution would cease to be operational (i.e. in June 2006).

Implications of selected new IFAD initiatives. The growing emphasis on ‘country presence'13 will have important consequences on the modus operandi and costs of both direct supervision and supervision by CIs. For instance, IFAD's country presence will have a role in supervision activities, ranging from simply facilitating the organization of supervision missions to the more substantive role of providing backstopping and follow-up on implementation issues. Furthermore, management is developing a proposal for the Fund's new operating model. There are two elements in this model that are likely to have important implications for IFAD's supervision activities. These are: (a) the shift in the unit of account from the project to the country programme level; and (b) the utilization of the Results and Impact Management System (RIMS) to capture and analyse data systematically according to a core set of indicators. IFAD will need to pay attention to these aspects in its future supervision activities.

Direct supervision analysis

Selection of the pilot projects. Most of the 15 projects were selected according to the criteria presented by IFAD to the Executive Board in September 1997. Two projects (in Gaza and the West Bank and Zimbabwe) have been adversely affected by political instability. While the inclusion of the Zimbabwe project in the DSPP was justified, because the unexpected events in the country could not have been foreseen at the time of Board approval, the inclusion of the Gaza/West Bank programme in an unstable country situation did not reflect an appropriate choice for inclusion in the pilot programme. The evaluation also notes that the Bangladesh project was not an appropriate choice in that it was not innovative, replicating broadly the design of a previous IFAD-financed project in the country.

Approaches to direct supervision. There have been various approaches to direct supervision in the 15 pilot projects. These include: (a) intensive CPM involvement in all steps of direct supervision, including fiduciary aspects. In these cases, the role of UNOPS was largely limited to disbursement processing; (b) the CPM completely delegates fiduciary aspects to UNOPS and focuses on implementation support matters, although s/he may not write mission reports and may not be as closely involved in the annual work programme and budget processes; (c) fiduciary aspects are dealt with by UNOPS, and the CPM appoints consultants who would be largely responsible for leading supervision missions and producing the corresponding documentation. The CPM may participate in key stages of the missions. In this approach, the CPM takes a management role and focuses on mission deployment and ensuring that all outputs are produced as required; and (d) this approach is specific to those countries in which IFAD has some form of local representation that plays a part in direct supervision by participating in supervision missions and following up between missions. The distribution of the pilot projects across the four approaches has been fairly evenly spread, with three to four projects each following approaches (a)-(c) and two projects using approach (d).

Implementation of direct supervision. The evaluation reviewed the implementation experience of direct supervision activities as compared with those under supervision by CIs. Its report observes that supervision planning during preparation of country strategic opportunities papers (COSOPs) and the project design phase was given limited attention in both forms of supervision. On another issue, the evaluation notes that the continuity of CPMs in direct supervision activities is important, and that in some pilot projects, there have been more than three CPMs allocated to the project in the last five years. In terms of supervision mission frequency and duration, the directly supervised projects have received close to two supervision missions per year, against one supervision mission for the control group projects. The average mission days per project per year were 15.2 days for direct supervision as compared with 11.2 days for the control group.

The composition of supervision mission teams in the directly supervised projects and in the control group has not been significantly different, with both groups making use of local consultants. There has also been more consultation between IFAD, the country and the project authorities before deciding what expertise to include in each supervision mission, especially in the context of the DSPP. Direct supervision missions on average comprised 2.6 personnel, compared with 2.5 for the control group.

So far, eight of the projects directly supervised have benefited from MTRs, whereas 11 projects in the control group have had an MTR. This is understandable, given that the average implementation period of the directly supervised projects (4.8 years) is less than that of the control group (6.5). In the case of the DSPP, the MTR has offered an opportunity to formalize a number of decisions made during the regular supervision missions that preceded the MTR. That is, given the more constant and intensive interaction between IFAD and the project authorities in directly supervised projects, CPMs have been able to stay on top of implementation issues and make key decisions within the framework of supervision missions themselves, rather than wait until the MTR stage. Hence, the MTR does not appear to have been critical in the context of the DSPP. It has been a critical instrument in CI-supervised projects, as it has provided CPMs the possibility of conducting a comprehensive review of what has been done and of introducing the required corrective measures to project design and implementation arrangements.

The quality of supervision reports varied across the directly supervised projects. In the terms of reference of and reports on direct supervision, attention largely focused on implementation issues and on physical and financial achievements, and less on lessons learned and impact. The same may be said of supervision reports prepared by CIs. However, supervision reports of the DSPP projects provided better coverage of issues of concern to IFAD, such as gender mainstreaming, monitoring and evaluation, beneficiary participation, building of grass-roots institutions and so on. Moreover, the documentation of the directly supervised projects was more complete and more easily accessible at IFAD than that of the control group.

In terms of feedback, the evaluation concludes that the DSPP did not systematically follow the set procedures established for feedback. Often an informal mode of communication was chosen rather than following official channels. In spite of the above, the evaluation notes that there is generally a faster response to project queries and follow-up on supervision recommendations under direct supervision than under CI supervision. The feedback in supervision undertaken by CIs is also erratic. Few CIs produce all the required documentation. The inadequate type and quality of reports and feedback bring up the critical issue of the current lack of quality assurance systems in the overall supervision activities of IFAD, which will be discussed later.

No notable difference in participatory processes was observed between IFAD direct supervision and supervision by CIs. However, there appears to be a clear preference for the increasing trend of conducting joint review missions (as organized by some IFIs) with the governments concerned, as opposed to traditional supervision missions. The term ‘supervision', itself, was described by many partners at the country level as being top-down, one that did not reflect the partnership ethos between the country concerned and IFAD.

Project status reports (PSRs) are an important instrument in IFAD's overall monitoring and reporting system. In PSRs of directly supervised projects, there has been no analysis or lessons on direct supervision processes or information on costs. In addition, PSRs do not include ratings on the IFAD impact indicators for direct supervision, which were developed at the beginning of the DSPP. For CI-supervised projects, CPMs include an assessment of the CI's performance and assign a corresponding rating. For directly supervised projects, PSRs do not include a rating of IFAD's performance in direct supervision. With regard to the process in the preparation of PSRs, it must be noted that the PSRs of CI-supervised projects are done by the CPM concerned, whereas the same CPM responsible for direct supervision prepares the PSR for the project under his/her direct supervision.

One important message emerging from the evaluation is the very wide support by IFAD partners of direct supervision. Government authorities, development organizations and project authorities were all of the unanimous opinion that there were advantages in having a direct contact with IFAD through CPMs. For example, partners conveyed the view that direct communication and interaction with IFAD contribute to better implementation and a stronger partnership, be it in terms of policy dialogue, identification of future pipeline and cofinancing opportunities or knowledge-sharing. There was one case in which a criticism was not of direct supervision, as an approach, but of the frequent staff changes at the CPM level that the project had experienced. The perceived lack of seniority and qualifications of some staff assigned by IFAD for this task were also raised for some projects.

Fiduciary aspects (under the formal responsibility of UNOPS) of directly supervised projects perform better than those of the control group. This finding is consistent with the ESM finding that UNOPS tends to perform better than other CIs. As UNOPS is the only CI involved in the DSPP, it is not unexpected that there would be a higher level of performance than for the control group, which incorporates a variety of CIs.

Project implementation performance. The evaluation first compared project implementation performance using indicators and ratings contained in the PSRs for both the directly supervised projects and the control group. Thereafter, the evaluation used the same set of indicators included in the PSRs and, based on its own independent ratings, compared the implementation performance of the directly supervised projects with those supervised by CIs.

From the evaluation's analysis, it is evident that the directly supervised projects perform better than CI-supervised projects across the PSR indicators in terms of, for example: compliance with loan covenants (directly supervised projects were rated on average 3.4 as against 3.1 for CI-supervised projects);14performance of M&E systems (3.2 for directly supervised projects against 2.6); availability of counterpart funds (3.5 against 3.2) and so on. One explanation might be more optimistic reporting by CPMs, who, as mentioned previously, are themselves responsible for the preparation of the PSRs. However, the evaluation's independent assessment also demonstrates that, on the whole, directly supervised projects have a better implementation performance as compared with CI-supervised projects across the PSR indicators. All rating scores may be seen in Chapter III, Tables 4 and 5.

The comparison of the directly supervised projects with the 15 projects in the control group was also developed by assessing the average time from approval to effectiveness. The analysis reveals more favourable performance by directly supervised projects. That is, the average time lag is 15.36 months for directly supervised projects as compared with 17.21 months for all other projects in the same country. Moreover, in terms of disbursement performance, the average rating by the evaluation team for directly supervised projects is 2.6 as compared with 1.9 for CI-supervised projects in the control group. For example, the average cumulative disbursement rate for all directly supervised projects in the fifth year of project implementation was around 62% as compared with 43% in the control group. Chapter III, Figure 1 provides more information on disbursement performance.

Finally, the evaluation reviews one indicator that is not included in the PSRs: the time overrun factor in project implementation (number of years/months a project is extended beyond the original completion date). Based on the calculations made by the evaluation, the overall time overrun for directly supervised projects is on average 0.54 years, as compared to 1.4 years for projects in the control group. Time overrun is an important indicator, as it reflects the soundness of design, but also the ability of the supervision process to recommend timely corrections and improvements during implementation, as and when required. Time overrun is also significant because it has an administrative cost implication for the Fund: additional supervision costs and related staff time need to be allocated for the time that projects run beyond their original completion date.

The human (social) dimension in supervision and attention to key areas of IFAD's catalytic role. There is evidence based on the evaluation's analysis that through direct supervision, as compared with supervision by CIs, IFAD has paid more attention to issues such as the targeting of women (rated 3.1 for directly supervised projects, compared with 2.5 for projects supervised by CIs); targeting the poor (rated 3.0 for directly supervised projects against 2.4); beneficiary participation; gender mainstreaming; participatory monitoring and evaluation and so on. These trends are by and large consistent with the ratings included in the PSRs of the directly supervised and control group projects. Direct supervision has also provided IFAD an opportunity to focus more on issues such as innovation (rated 3.2 against 2.5 for CI-supervised projects), partnership, knowledge management and policy dialogue, which according to the IFAD strategic framework contribute to improving project performance and impact. In each country exposed to direct supervision, according to the evaluation and partners at the country level, the presence of the CPM during supervision is seen as an opportunity to advance IFAD's broader objectives, such as those listed above. Chapter III, Figures 2 and 3 provide all comparative rating scores on the aforementioned indicators for the DSPP and projects supervised by CIs.

The evaluation's rating for knowledge management of directly supervised projects (rating 2.7) is only marginally better than that of projects in the control group (2.5). This is partly explained by the fact that, while the knowledge acquired through direct supervision at the individual CPM level was high, there was no systematic effort to document, analyse or share such learning from direct supervision. Moreover, no specific activities were conducted or resources allocated to knowledge management in the directly supervised projects, and their supervision reports did not emphasize lessons learned. The same is true for the projects in the control group. However, this is considered a particular shortcoming in the case of direct supervision, as knowledge management in the broader sense was a specific objective of the DSPP (which is not the case for CI supervision).

Development effectiveness. The first thing to note is that, contrary to the requirement of the action plan adopted by the Governing Council, no specific, ongoing monitoring and reporting mechanisms to trace project impact were established, neither at the individual project level nor for the DSPP as a whole. In addition, contrary to the recommendation in the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects, "for comparative analysis a control group of CI-supervised projects similar in other respects to the directly supervised projects" was not identified and monitored. Nevertheless, the evaluation made efforts to analyse the development effectiveness of the directly supervised projects and compare them with projects in the control group.

The evaluation has undertaken three specific types of comparison that, taken together, provide an overview of the development effectiveness of the directly supervised projects in relation to the CI-supervised projects. Tables 6, 7 and 8 in Chapter III contain the specific rating scores that provide the basis for the analysis.

First, based on three indicators (quality of supervision recommendations, follow-up actions and support provided to the project), the overall quality of supervision was assessed (see Chapter III, Table 6). These indicators are also used by the World Bank's Quality Assurance Group in determining the quality of the Bank's supervision, which is considered an important ingredient in achieving development effectiveness. In fact, in one of their recent evaluations, the Operations Evaluation Department of the World Bank concluded that well-supervised projects are twice as likely to succeed as are poorly supervised ones. The DSPP evaluation's assessment according to the three indicators reveals a positive trend in favour of directly supervised projects. For example, quality of supervision recommendations is rated 3.3 for DSPP, compared with 2.9 for CI-supervised projects.

Second, the evaluation compared directly supervised and control group projects using the 11 IFAD impact indicators for the DSPP.15 By and large, the directly supervised projects do better than those supervised by CIs across all indicators (see Chapter III, Figure 4). For example, "identification of new project concepts for inclusion in the pipeline" is rated 2.9 for DSPP as compared with 2.1 for CI-supervised projects. However, the control group projects perform marginally better in terms of timeliness of reporting (rated 2.9 for projects supervised by CIs, compared with 2.8 for DSPP). The reason is partly that, as subcontractors, CIs pay specific attention to project reporting. This issue has not been given the same level of attention in direct supervision, given the constant contact between IFAD and the project authorities. It can also be seen that the ratings for costs appear to favour CI supervision, as supervision by CIs costs less on average (this will be discussed in more detail later).

Third, the evaluation rated the directly supervised and CI-supervised projects using the six impact domains in the OE methodological framework for project evaluation (MFE). Moreover, the two overarching factors in the MFE (sustainability and innovation/replication) have been included in the analysis (see Chapter III, Table 7). In short, the directly supervised projects are rated better in most impact domains (e.g. food security is rated 2.9 for DSPP, compared with 2.5 for CI-supervised projects), although in two impact domains (environment and sustainability), the directly supervised projects performed less well as compared with the CI-supervised projects.

With regard to the latter issue, it should be noted that 12 of the 15 projects in the control group included a specific objective or component related to environmental matters. Consequently, their supervision missions have normally adequately reviewed project progress in this area. On the other hand, only five directly supervised projects have a specific environmental component and their supervision missions have not always included the required skills to undertake a thorough assessment of environmental issues and to provide the necessary backstopping to project staff. Furthermore, environmental matters have not received the same level of focus within IFAD at large as compared with areas such as gender mainstreaming and social capital formation.

No conclusive statement can be made on project sustainability at this stage. On the one hand, the slightly less positive ratings of the directly supervised projects as compared with the control group may be explained partly by the relatively fewer years of implementation of the directly supervised projects. That is, many of the projects in the DSPP have devoted greater emphasis, at least in the initial years, towards developing grass-roots organizations and promoting participatory processes and less towards productive activities. This may be a cause of the lower sustainability rating at this stage. Moreover, some of the directly supervised projects (in Brazil, India and the Sudan, for example) suffered initial delays in implementation due to compelling political and administrative circumstances, which could be another reason. On the other hand, the IEE concluded that during the early stages of a project there are relatively high expectations that project benefits will endure, but these expectations are modified in the later stages of implementation.

Finally, the overall composite rating16 of the six impact domains is compared with the rating available in the PSR on ‘meeting development objectives'. In both cases, directly supervised projects do better than CI-supervised ones (see Chapter III, Table 8). The MFE composite index is rated 2.9 for the DSPP as compared with 2.5 for CI-supervised projects.

Some general considerations must be made while interpreting the above results. First, it must be acknowledged that the differences in rating scores are relatively small. Nevertheless, the analysis shows that directly supervised projects have performed better, compared with the CI-supervised ones, according to most indicators considered in this evaluation. Additionally, direct supervision has contributed to enhanced results in furthering IFAD's broader objectives, such as policy dialogue, partnership-building and knowledge management. The evaluation also notes that the results of direct supervision could have been even greater had the pilot programme been implemented under more favourable conditions, for example if the CPM workload had been appropriately prioritized or if adequate monitoring and assessment systems had been put in place. However, in analysing the results, one must also take into account the time allocated to direct supervision. More than half the CPMs involved in DSPP said they spend up to double the time on direct supervision than on other projects in the portfolio. There are also issues of cost, which will be discussed later.

Quality assurance. IFAD lacks a continuous quality assurance system for supervision, which would have allowed the Fund to meet more fully the objectives of the pilot programme. As a result, there was limited quality assurance in direct supervision inputs and processes (e.g. mission terms of reference, composition and duration in the field) and in deliverables such as supervision reports. Other IFIs, in particular the African Development Bank (AfDB) and the World Bank, also have quality assurance mechanisms that allow for periodic assessment of supervision activities (e.g. once every two years). Through these quality assurance mechanisms, IFIs are able to take a holistic view of the supervision function and suggest corresponding systemic improvements across the organization. Such quality assurance mechanisms in the aforementioned IFIs are located outside the operations departments. The Inter-American Development Bank (IDB) is in the process of setting up a similar quality assurance system. In addition, IFIs generally have quality assurance mechanisms for supervision built into the operations divisions (e.g. at the African Development Bank, peer reviews are used for quality assurance).

Reporting to the Executive Board. Management has not complied with two key commitments in terms of reporting. First, no MTR of the pilot programme was undertaken in the last quarter of 2000/first quarter of 2001, as had been decided by the Governing Council. The reasons for this are neither evident nor documented. Second, although the Fund has provided progress reports on direct supervision to the Board on an annual basis, these have focused mainly on the deployment of inputs and processes in implementing direct supervision activities. The evaluation notes that, at the same time, the Executive Board could have exercised better oversight of implementation of the action plan and could have demanded more analytical information from management on the development effectiveness of the DSPP, in particular on the costs, results and lessons learned of the pilot programme.17

Operating environment for direct supervision. CPMs did not benefit from a favourable operating environment in which to manage the increasing workload caused by the introduction of direct supervision. More specifically, training was not provided at the outset of the pilot programme, nor were CPMs given any particular recognition or incentives. They accepted the additional responsibilities for direct supervision, which they undertook without managerial support for any reprioritization of their existing workload. Last, but equally important, management engagement appeared to diminish gradually after approval by the Governing Council of the DSPP, as demonstrated, for example, by the non-implementation of three key activities18 included in the action plan.

Learning and knowledge management

By participating in direct supervision activities, CPMs have acquired first-hand experience of the task of supervision and a better appreciation of the issues related to rural poverty reduction. The benefits of their experience are evident, as some CPMs have put their newly acquired knowledge to use in designing new projects and implementing ongoing ones. Still, supervision reports (of both direct supervision and CIs) focus more on implementation issues and less on lessons learned from supervision processes or broad rural development issues. This has led to missed opportunities to incorporate learning into project design across the institution and to improve the supervision activities of cooperating institutions in general. Nonetheless, in spite of the benefits to individual CPMs, there is a need to institutionalize this knowledge and establish systems/platforms through which CPMs involved in direct supervision could share their overall experiences among themselves and within the Fund. Some exchanges have occurred at the CPM level in an informal, unstructured manner. But, overall, institutional support to capture and channel learning from direct supervision to IFAD staff has been inadequate.

IFAD also lacks an adequate reporting and feedback mechanism at the country level: its current system for learning does not allow governments and other development organizations to become familiar with its successful innovative approaches and to learn from its experiences. This needs to be addressed in the light of the Fund's strategic objective of seeking partnerships for replicating and scaling up the activities it finances, as well as of the proposed new operating model. OE recognizes that knowledge management is an institution-wide concern and that direct supervision is only one of the components of the Fund's learning system. However, the unique opportunity for the institution as a whole to learn from the direct supervision pilot programme to enhance IFAD operations has been largely missed.

Efficiency of direct supervision

Based on the calculations of the evaluation, the average cost per year per project of direct supervision is around USD 93 000 as compared with USD 61 000 for CI-supervised projects in the control group. In this regard, it should be noted that the costs incurred go beyond the expectation of the Governing Council, which had decided that in implementing direct supervision activities "there would be no cost increase" to IFAD.

However, there are some points related to the average CI cost that merit being highlighted. First, there is quite a variation in costs across the different CIs. Hence, the average cost for CI supervision calculated above does not reflect the costs that all CIs charge. For instance, supervision through some CIs (e.g. the World Bank) costs over USD 100 000.19 The current overall cost of supervision through UNOPS (the CI with the greatest number of projects under supervision) is around USD 79 000 – see Chapter V for more data on costs. Moreover, according to the evaluation, the current cost attributed to UNOPS needs to be increased if the Fund is to expect them to provide enhanced quality services.20 Next, there are additional costs of CI supervision that are difficult to identify and have thus not been included in calculating the total cost of CI supervision. For instance, the Belgian Survival Fund has an annual allocation for CI supervision in its administrative budget, recorded outside IFAD's accounting system. Along similar lines, more recently, IFAD's Asia and the Pacific Division provided additional funds to the UNOPS Asia Office to augment its supervision-related activities. These and other such costs are not included in the average of USD 61 000 calculated for CI supervision. Finally, it should also be noted that the cost to IFAD for direct supervision includes the fielding of around two supervision missions per year to each project for longer durations than those of the average CI supervision activities.

Furthermore, the evaluation underlines that the longer implementation period of projects in the control group as compared with the directly-supervised projects has administrative cost implications for IFAD, which would raise the overall costs of supervision by CIs and which need to be considered.

The evaluation also argues that there are possibilities of reducing the costs of direct supervision to some extent. For instance, this could be achieved by making use of competent national entities to discharge the fiduciary responsibilities involved in supervision and thus enhancing the role of government in implementation support activities. Greater use of local consultants could also contribute to cost savings. Finally, as CPMs and the other IFAD staff involved acquire the necessary competencies and gain more experience in direct supervision, it is fair to assume that the overall time invested in the associated tasks is likely to reduce to some extent. This will have a corresponding effect on the staff costs component, leading to a reduction in the overall costs of direct supervision.

While the cost of direct supervision may still be higher than that of CI supervision, according to the evaluation it is paramount to assess the cost together with the corresponding benefits to the Fund. The analysis of the evaluation illustrates that direct supervision has contributed to better development effectiveness and has allowed the Fund to further its catalytic objectives of innovation, policy dialogue and partnership development. With regard to the latter point, partners at the country level expressed their preference for building partnerships directly with IFAD rather then managing such processes through proxy institutions such as CIs. Moreover, in the context of knowledge management, although the CPMs involved in direct supervision have acquired better understanding of implementation matters and despite the fact that knowledge from the DSPP has not been properly institutionalized, the evaluation observes that using a CI for supervision introduces an extra layer into the already feeble learning loop of the Fund. On a similar note, the knowledge that the staff or consultants of CIs have acquired by undertaking supervision on behalf of IFAD is largely lost to the Fund. Finally, the common opinion of governments and all other partners at the country level, who clearly favour IFAD's direct involvement in supervision activities, must also be given due consideration.

Supervision systems and experiences of other International Financial Institutions

By and large, the AfDB, the Asian Development Bank (AsDB), the IDB and the World Bank have similar supervision systems, with processes and details applied variously to fit their institutional structures. In the IFIs, generally, supervision is not limited to official missions and formal reports, rather it is a continuing and flexible process, specific to the needs of the particular operation and intended to foster a close partnership among an IFI, its borrower, and the implementing agencies. The planning of supervision is done carefully during the design phase of the project cycle, when appropriate resources and arrangements are put in place to facilitate supervision. The supervision process normally starts after the project is approved and ends when the last disbursement is completed and the project completion report is prepared.

At the Inter-American Development Bank, supervision is largely delegated to the country offices, with participation of staff from headquarters as required. At the Asian Development Bank and the World Bank, headquarters and country offices share responsibility for supervision. That is, in some cases, supervision responsibilities are entirely delegated to the country offices, especially in those countries in which offices have staff with the required sectoral know-how. Under such arrangements, selected staff at headquarters are sometimes asked to join supervision missions. In other cases, the "task managers", if based at headquarters, retain responsibility for supervision and, in turn, involve staff at their country offices in supervision missions. At the African Development Bank, so far, task managers based at headquarters have full responsibility for project supervision. However, with the establishment of 26 country offices by the end of 2006, AfDB expects in-country staff to be involved in one way or another in project supervision and related follow-up, as well as in portfolio management issues. One important aspect of supervision by these IFIs is that certain supervision functions are being increasingly located in their country offices. For example, the review of documents related to fiduciary aspects (such as bidding proposals and accounting matters) is largely handled by the country offices. It is important to realize, however, that the degree to which such field office staff can make decisions varies according to the delegation provided by the responsible task managers. Finally, some IFIs have well-established quality assurance mechanisms for supervision and others are rapidly moving in the same direction.

Conclusions

A consistent trend in the overall evaluation analysis demonstrates that, compared with CI supervision, direct supervision has greater potential to contribute to better development effectiveness at the project level and, at the same time, allows greater attention to IFAD's broader objectives at the country programme level. That is, direct supervision by IFAD can contribute to better and timelier project implementation, which in turn enhances overall results and impact. Moreover, through direct supervision, IFAD has been able to emphasize issues of prime concern such as gender mainstreaming, targeting and the building of grass-roots institutions, which taken together are important elements in ensuring sustainability.

In view of their more frequent and longer presence in countries with direct supervision, CPMs have wider opportunities to advance IFAD's objectives at the country programme level, including policy dialogue and partnership development. CIs do not consider these to be a priority, nor can such activities be effectively advanced through third parties. Although there is no conclusive evidence that new partnerships have resulted from the DSPP, governments and other development partners at the country level have unanimously expressed their deep appreciation of the more frequent contact with CPMs, which has been facilitated by IFAD direct supervision activities. The same partners communicated that they find it more useful to deal directly with IFAD staff rather than with CI representatives. In this regard, for example, partners affirmed that the response rate and follow-up on implementation issues are faster under direct supervision than under supervision by CIs.

Direct supervision has contributed to developing IFAD's knowledgebase. In this regard, in particular, the CPMs responsible for direct supervision have acquired knowledge of supervision processes, project implementation and rural development issues in the countries concerned. This knowledge has enabled them to better design and implement new operations. However, the knowledge gained at the CPM level has not been systematically shared with others, nor has it been institutionalized, which is one of the key shortcomings of the DSPP. The evaluation notes that supervision by CIs also offers possibilities for knowledge generation. However, CI involvement in supervision makes the transmission line of knowledge from the CI to IFAD and the country more cumbersome in the already feeble knowledge systems of the Fund.

The evaluation concludes that direct supervision allows CPMs to strengthen country-level coordination both within the context of IFAD operations and with the development community at large. It also facilitates the strengthening of existing IFAD-funded programmes and the identification of new programmes and cofinancing opportunities, which are mostly available at the country level, given that the majority of our international and bilateral partners have delegated an increasing amount of authority to their country representatives.

Unlike most other IFIs, IFAD lacks a quality assurance system for direct supervision. As a result, the DSPP was approached and implemented in a variety of ways, based on the perception and understanding of individual CPMs. Compliance with the minimum supervision requirements and the direct supervision guidelines was also not monitored. In conclusion, both continuous and periodic quality assurance systems are fundamental if direct supervision activities are to be expanded.

At face value, the average cost of direct supervision (USD 93 300) is higher than the average cost of supervision by CIs (USD 61 461)21. However, the evaluation argues that cost should not be seen in isolation from the benefits the DSPP has evidenced. Moreover, in discussions with UNOPS (the main IFAD CI), it is clear that the costs to IFAD for supervision by UNOPS need to be increased if they are to deliver the type and quality of service IFAD requires in the future. In parallel, the evaluation notes that there is the potential for efficiency gains in direct supervision if, for example, the fiduciary responsibilities related to supervision are entrusted to competent national entities and greater use is made of local consultants for implementation support activities.

The evaluation notes that management's interest has appeared to gradually diminish following approval of the DSPP by the Governing Council. This is illustrated by the fact that management did not fully implement all decisions of the Governing Council. For instance, it did not undertake a mid-term review of the DSPP as required. Neither did it establish an integrated, analytical accounting system to track costs of the DSPP. Nor did it set up a monitoring and assessment system to measure the performance and impact of direct supervision. According to the evaluation, however, neither did the Executive Board exercise adequate oversight to ensure that IFAD management would fulfil all its commitments under the DSPP. The evaluation believes that the outcome of the DSPP would have been even more significant had all the requirements laid down by the Governing Council been implemented.

The evaluation reveals that not all concerned have the same understanding of the notion of supervision. In fact, there is often confusion as to what constitutes supervision missions, implementation support, follow-up activities, fiduciary responsibilities and so on. There is also a lack of clarity on the roles and responsibilities of IFAD, CIs, project staff and government authorities. Additionally, it is worth noting that many partners at the country level felt that the term ‘supervision' – when applied to the implementation aspects of projects rather than to fiduciary aspects – has a paternalistic undertone and they felt uneasy with its continued application. Supervision in that sense reflects a top-down, non-participatory approach to the function, which is inconsistent with the Fund's objectives of promoting ownership and partnership with governments and other institutions.

Recommendations

It is important to state that the evaluation's recommendations have taken into consideration the relevant recommendations contained in the ESM report. The evaluation's five key recommendations, given below, are mutually reinforcing. The evaluation recommends that they all be implemented fully in order to ensure that the desired impact is achieved in IFAD's future efforts in the area of supervision and implementation support.

Recommendation one: definition of supervision

The evaluation recommends that the concept of ‘supervision' as used by IFAD be divided into two distinct operational parts: (i) supervision of fiduciary aspects, including aspects related to procurement review, disbursement processing and compliance with financial and auditing requirements; and (ii) support to programme and project implementation.22 This would include, for example, the organization of periodic ‘implementation support' missions and related follow-up; an assessment of the achievement of programme/project objectives and assistance in identifying remedial solutions for implementation challenges, based on interaction and dialogue with project authorities and other partners at the country and project level; and the provision of guidance in preparing the annual work plans and budgets. It would also include oversight of project and programme implementation, for example, in terms of monitoring the achievement of physical targets.

Recommendation two: develop a comprehensive supervision23 and implementation support policy for IFAD

The Fund should develop a specific overall supervision and implementation support policy for its operations. The policy should reflect the following elements.

Supervision of fiduciary aspects. IFAD should be allowed to decide, on a case by case basis, whether to subcontract a competent national, regional or international entity to perform such functions. Special efforts should be devoted to engaging national entities, as this would have the double effect of building local institutional capacity and reducing costs. The utmost attention should be given to ensuring that there is no conflict of interest between the prospective national entity and the IFAD operation under consideration. In a few and very specific circumstances, IFAD might consider undertaking the supervision of fiduciary aspects itself.

Implementation support. The evaluation recommends that the policy state explicitly that: (i) IFAD should be made responsible for providing direct implementation support to all its operations globally. In this regard, it is important to specify the role of the CPMs, who could either be intensively involved themselves as "implementation backstoppers" or act as implementation-support task managers with more attention to process management, which would also require a degree of direct involvement in activities. In both cases, CPMs would ultimately be responsible for the process, content and outputs of implementation support activities; (ii) such support would cover all aspects of IFAD country programmes, both at the project level and beyond, such as policy dialogue and partnership strengthening; and (iii) the role of partner governments would be specified and given due emphasis, which would contribute to building greater ownership and local capacities, as well as reducing costs.

The above would result in new responsibilities for the Programme Management Department (PMD), which would require the allocation of additional staff and financial resources as well as new competencies and skills. It is fundamental that the additional resource requirements for the implementation of the new policy be clearly articulated in a comprehensive and explicit manner by IFAD. This would require a detailed cost analysis, particularly of the elements in paragraphs 65-66, as well as an assessment of the skills and competency of current CPMs. Based on the aforementioned analysis, IFAD would need to develop a proposal to meet the cost deficits in implementing the new policy. It would also need to develop a plan for enhancing the ability of CPMs to meet the specific requirements of the new policy, recognizing that it may not be possible to enhance these skills and competencies in all cases. Until the required level of financial resources would be made available to the Fund and additional PMD staff recruited and their competencies and skills developed and upgraded, IFAD might consider a phased approach to expanding implementation support in all operations.

The success of the new policy would also be determined by the support provided by IFAD management and the conducive environment it creates for the purpose. For example, management would need to: (i) ensure that appropriate opportunities are introduced for periodic staff training; (ii) establish an incentive framework and platforms for the sharing of knowledge acquired by CPMs; and (iii) allocate the necessary time for reviewing implementation experience under the policy. The Board should also play a proactive role in exercising oversight of implementation of the new policy and in approving administrative budgets for the purpose.

The policy should be evaluable, and in particular include a roll-out and implementation plan, with performance indicators that can be monitored periodically.

A chapter on supervision and implementation support should be included on a standing basis in IFAD's annual Portfolio Performance Report. The chapter would provide an analytical account of the opportunities and challenges in the area, and identify key lessons learned. Moreover, it would provide an indication of the ongoing operational measures introduced by IFAD to address emerging issues.

The undertaking of supervision of fiduciary dimensions and implementation support would require revisiting Article 7, Section 2(g) of the Agreement Establishing IFAD.

Other integral aspects that the policy should consider are contained in recommendations three to five.

Recommendation three: supervision and implementation support in the framework of the COSOP

The evaluation recommends development of an overall approach to supervision and implementation support during the preparation of COSOPs. This would take into account the need to supervise the fiduciary dimensions of all operations, and the provision of implementation support to the country programme, including areas such as policy dialogue, partnership-building and knowledge management, in addition to the support traditionally provided to projects. The COSOP would lay the provisions for the need to develop an annual supervision and implementation support plan for each country, indicating the specific objectives, human and financial resource allocations and expected results. Each PMD regional divisional would set up an electronic monitoring, assessment and reporting system, which would serve as a management tool to track the implementation progress of the plans and to flag issues requiring more immediate follow-up. A template should be developed by PMD for the section on supervision and implementation support that would be included in the COSOPs.

Recommendation four: quality assurance system

Within the framework of an overall enhanced quality assurance system at IFAD, there is a need to introduce quality assurance mechanisms for the supervision of fiduciary dimensions and implementation support activities. The evaluation recommends that IFAD establish a management review committee within PMD, which would review supervision and implementation support activities, results and related operational issues. Quality assurance, as well, needs to be strengthened in the PMD divisions. In this regard, semi-annual reviews of supervision and implementation support activities should be undertaken at the divisional level. Summaries of the discussions at these meetings should be circulated to all PMD divisions. Moreover, IFAD should build on the experience of other IFIs (in particular of the Quality Assurance Group at the World Bank) to establish an IFAD-specific quality assurance group, which would review aspects of supervision and implementation support, in addition to any other aspects related to implementation of the COSOP and its components. The Fund would need to thoroughly reflect upon the most appropriate location within IFAD's organizational structure for such a group, which would ensure the most objective and independent review possible of its supervision and implementation support efforts. The introduction of such a quality assurance group should take into consideration the mandates and performance of existing quality control systems within IFAD, such as the project development team, Technical Review Committee and Operational Strategy and Policy Guidance Committee.

Recommendation five: learning and knowledge management

Incentives should be provided to staff to encourage sharing of the knowledge they acquire. For example, time needs to be carved out in CPMs' workload for sharing the knowledge they generate through supervision and implementation support activities. The documenting and sharing of knowledge should be included as an indicator in assessing the annual performance of CPMs. Specific instruments need to be established to facilitate learning and knowledge-sharing. In particular, time should be reserved on a standing basis in the CPM forum for discussing issues and sharing knowledge generated through supervision-cum-implementation support activities. Each project mid-term review and project completion report should include a specific treatment of supervision and implementation support issues, as should all evaluations undertaken by OE. The project and country status reports should be reformatted to include a narrative section on supervision and implementation support, and ratings must be included in all cases. Other instruments should be introduced, such as peer reviews at the PMD divisional level in relation to implementation support activities.

The monitoring and evaluation systems at the project level need significant strengthening if they are to contribute effectively to learning. Moreover, in line with the new operating model, it is necessary to assist in the development of integrated monitoring and evaluation systems at the country level. This would not only facilitate monitoring and sharing of experience across the entire project/programme portfolio, but would also allow for tracking the implementation of the broader objectives of IFAD country programmes, such as policy dialogue and partnership-building.

IFAD should build on the experiences of other international financial institutions, and make more comprehensive use of information technology for knowledge management purposes in relation to supervision and implementation support. In this regard, it is recommended that the existing PPMS be expanded so that it can carry updated summaries of supervision and implementation support activities at all times. An enhanced PPMS should accordingly be made accessible to external partners through the IFAD internet with immediate effect. Last but not least, an integrated, analytical accounting system should be developed through close cooperation between PMD and the Office of the Controller. This would allow the monitoring and analysis of all costs related to supervision-cum-implementation support, including staff time.


1/ The resolution will cease to be effective in June 2006, as the last project (India) became effective in June 2001.

2/ For the purpose of this evaluation, the term ‘development effectiveness' encompasses the extent to which the DSPP's overall objectives have been met, the efficiency in implementing the pilot programme and the contribution of direct supervision in improving project implementation and potential project impact.

3/ For definitions of the criteria, see Methodological Framework for Project Evaluation (document EC 2003/34/W.P.3).

4/ These impact indicators were developed and presented by IFAD management to the Executive Board in April 1999 (see Box 1 of the present document and document EB 99/66/R.10/Rev.1).

5/ In Armenia, the Gambia and Uganda.

6/ In Bangladesh, Benin and Indonesia.

7/ In Armenia and India.

8/ In Bangladesh, Guinea, Indonesia and Peru.

9/ The role of the CLP was to provide comments and input at several key stages in the evaluation process (for a definition of the CLP, see paragraph 33 in the IFAD Evaluation Policy, document EB 2003/78/R.17/Rev.1). The members of the CLP for the DSPP evaluation were: Mr Nigel Brett, Mr Jim Carruthers, Mr Pablo Glikman, Mr Shyam Khadka, Mr Luciano Lavizzari, Mr John McGhie, Mr Ashwani Muthoo, Ms Rasha Omar, Mr Mohamed Tounessi and Mr Joseph Yayock.

10/ Mr Hans Wyss, former Director of Operations at the World Bank.

11/ Hence field work was undertaken in 13 countries of the DSPP (Gaza and the West Bank and Zimbabwe were excluded).

12/ The action plan can be seen in its original format in Appendix II.

13/ In December 2003, the Executive Board approved the field presence pilot programme in 15 countries globally.

14/ The rating scale used is from 1 to 4, where 1=negligible, 2=modest, 3=substantial and 4=high.

15/ The list of impact indicators may be seen in Chapter II, Box 1.

16/ The composite rating is the average of the six impact domains and the two overarching factors.

17/ In 1997 the GC had, indeed, decided that "progress, lessons learned and results of the test would be reported annually to the Executive Board" – see document GC 20/L.10/Add.1.

18/ Items 6, 7 and 8 under Recommendatio 5 in the action plan – see Appendix II.

19/ A study by the World Bank's Quality Assurance Group in 2003 concluded that, on average, from USD 100 000 to USD 125 000 should be allocated per year for supervision of community-driven development projects.

20/ The IEE report highlighted the same issue – see paragraph 18 on page 16 (document EB 2005/84/R.2).

21/ Although the actual cost per project per year for supervision by UNOPS, the main IFAD CI, is around USD 79 000.

22/ It could also be called ‘implementation support'.

23/ The term ‘supervision', from this point onwards in the document, means ‘supervision of the fiduciary aspects' related to IFAD financing.

LANGUAGES: English

Direct supervision pilot programme

diciembre 2004

Introduction

Background. In February 1997, the IFAD Governing Council adopted resolution 102/XX on Loan Administration and Supervision of Project Implementation, together with a Five-Year Plan of Action. The resolution stated that IFAD "may supervise specific projects and programmes financed by it". According to the action plan, no more than 15 IFAD-initiated projects were to be directly supervised and administered during the five-year period. This initiative, including the 15 projects, has since been referred to as the Direct Supervision Pilot Programme (DSPP). The Governing Council resolution entered into effect on 21 February 1997 and will cease to be operational five years after the date of effectiveness of the last approved project to be directly supervised by IFAD1. The Governing Council also decided that the DSPP should not entail any cost increases and that IFAD should use the same funds that would have been spent on supervision by cooperating institutions (CIs) to cover its direct supervision activities.

In order to assess the results of the action plan, in 2002/2003 the Office of Evaluation (OE) undertook an Evaluation of the Supervision Modalities in IFAD-Supported Projects (ESM). The objectives of the ESM were to "evaluate the effectiveness of current supervision modalities against the minimum supervision requirements [MSRs] … and other indicators of quality, and review the achievements under the Five-Year Plan of Action". It is important to note that most of the directly supervised projects were in their early stages of implementation at the time of the ESM. Hence, although the ESM provided concrete findings on supervision through CI, it provided only an overview of the emerging characteristics of IFAD's direct supervision efforts.

DSPP objectives. The overarching objective for embarking on the DSPP was to enable the Fund to acquire first-hand knowledge from supervision activities and to incorporate lessons learned from ongoing operations more effectively into its project design work. It was also to provide IFAD with "knowledge of the supervision function, of what are the costs of an adequate project supervision and of the development impact and human dimension of the projects in its portfolio. The Fund's involvement in direct supervision would also complement and improve cooperating institutions' own activities (mainly the human dimension of projects/programmes)". Although enhancing development effectiveness2 was not an explicit objective, direct supervision was expected to contribute to improving implementation performance and project impact.

Evaluation objectives. The main objective of the corporate-level evaluation (CLE) was to make an overall assessment of the DSPP's achievement in enhancing the implementation and impact of IFAD-funded operations. More specifically, the evaluation had the following key objectives: (a) compare and contrast direct supervision by IFAD with selected, relevant examples of supervision undertaken by CIs; (b) examine the processes established, alternative approaches and the experiences of country programme managers (CPMs) in undertaking direct supervision. This would include comparison with the approaches, systems and experiences of other international financial institutions (IFIs); (c) assess the efficiency of the direct supervision modality; and (d) examine the systems established to capture the experience and insights from direct supervision and the ways in which this has been of benefit to IFAD's project design processes and implementation support activities.

Evaluation methodology. The approach for the direct supervision evaluation has been to build on the methodology and results of the ESM. The criteria3 used in the CLE to assess the DSPP include: (a) the relevance of direct supervision in relation to the DSPP's objectives at the time of approval but also in today's context; (b) the effectiveness of the DSPP, measured against the achievement of the stated objectives of the programme and using the indicators specified for measurement of the impact of direct supervision;4 (c) the efficiency of direct supervision; and (d) an analysis and comparison of the actual and potential impact of the projects included in the direct supervision pilot programme with those supervised by CIs.

The evaluation process was planned to allow for triangulation of evidence and the views obtained from the main actors in the DSPP (the key government focal point at the national level, together with implementing agencies, beneficiaries, and the IFAD staff concerned). Moreover, OE had completed evaluations of three directly supervised projects5 in the past few years, which provided valuable additional sources of information and assessments. Since 2003, OE has also undertaken three country programme evaluations6 that included assessment of one directly supervised project in each of these countries. Relevant information and reports from the Independent External Evaluation (IEE) of IFAD, which analysed in detail two directly supervised projects7, have also been utilized. Finally, four projects8 in the evaluation control group have either been evaluated by OE or included in the IEE.

Evaluation scope. The evaluation involved a comparison of the directly supervised projects with 15 projects supervised by CIs (the latter therefore represented the evaluation's control group). That is, three CI-supervised projects per region were included in the control group based on a set of selection criteria agreed with the evaluation's core learning partnership (CLP)9. The list of projects included in the DSPP and the control group for the purposes of the evaluation can be seen in Appendix I.

Evaluation process. The CLE process began with the preparation of an approach paper, which provided an opportunity for developing a solid understanding of the evaluation's objectives, scope, methodology, time frames and expected outputs. The evaluation benefited from the views of the CLP. An external reviewer10 with wide experience in project supervision issues was contracted to advise OE at critical phases during the evaluation and to review key evaluation outputs. Moreover, OE undertook a thorough peer-review process within the division to improve the evaluation's overall quality.

The evaluation included the following activities: desk reviews of the 15 directly supervised projects and 15 projects in the control group; interviews with IFAD management and staff; field work in 13 of the 15 directly supervised projects and in eight control group projects11; discussion with the United Nations Office for Project Services (UNOPS) and selected IFIs at their headquarters and at the country level; direct supervision cost analysis; preparation of an early feedback note on an experimental basis, with the objective of sharing the emerging evaluation results and sensitizing IFAD management before the draft evaluation report was ready for discussion; organization of a round-table workshop to discuss the evaluation's results and lay the foundations for the agreement at completion point of the evaluation; and discussion at the September 2005 sessions of the IFAD Evaluation Committee and Executive Board.

The draft evaluation report was shared for review and comments with: staff in all the projects included in the DSPP and the control group, IFAD management and other staff, selected IFIs and CIs, and the government officials concerned at the national level in countries in which IFAD has funded a directly supervised project. Their comments have been included in accordance with the provisions in paragraph 42 of the IFAD Evaluation Policy.

The evolution of direct supervision of IFAD

Origin of direct supervision. The Executive Board reviewed the Joint Review on Supervision Issues for IFAD-Financed Projects in December 1996. The report was submitted to the Governing Council in February 1997, together with the Five-Year Plan of Action for the DSPP. The plan12 included the following actions, together with time frames for their implementation by IFAD:

  • "agreement by Governing Council for IFAD to supervise test projects;
  • criteria to be used for selection of test projects to be submitted to Executive Board;
  • 15 test projects to be determined (3 projects per region);
  • identify and negotiate with reputable private national or international organizations to undertake procurement and financial administration;
  • progress report of test projects to be reported to Executive Board;
  • analytical accounting system to be established to track the actual cost of direct supervision as well as supervision by CIs;
  • mid-term review of supervision of test projects to be submitted to Executive Board; and
  • establish monitoring system to evaluate the test projects."

DSPP modality. One important feature of the DSPP has been the cooperation between IFAD and UNOPS. The latter was contracted in July 1998 by IFAD to undertake the supervision of fiduciary aspects (such as procurement of good and services, disbursements, ensuring compliance with auditing and financial requirements) in the context of the DSPP. UNOPS was paid a standard amount equal to USD 12 000 per year per project for their services in the specified areas. IFAD's specific role in the DSPP was thus to arrange and conduct supervision missions, organize the necessary follow-up and provide the overall implementation support required by borrowers and their projects.

Monitoring and progress reporting. IFAD management presented indicators for the measurement of the impact of direct supervision to the Executive Board in April 1999 (see Chapter II, Box 1). IFAD further committed to reviewing the indicators at least one year after most directly supervised projects had become fully operational. For comparative analysis, the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects explicitly requested that a control group of CI-supervised projects, similar in other respects to the directly supervised projects, be identified and monitored.

In terms of reporting, the Governing Council decided that the progress, lessons learned and results of the pilot programme would be reported annually to the Executive Board. Moreover, IFAD would conduct a mid-term review (MTR) of the supervision of test projects, also to be submitted to the Board. In approving OE's work programme for 2004 during its session in December 2003, the Executive Board requested OE to undertake a CLE on the DSPP. Finally, the Governing Council decided that the President shall submit the results of IFAD's experience and conclusions on the DSPP to the Executive Board for its review before the corresponding resolution would cease to be operational (i.e. in June 2006).

Implications of selected new IFAD initiatives. The growing emphasis on ‘country presence'13 will have important consequences on the modus operandi and costs of both direct supervision and supervision by CIs. For instance, IFAD's country presence will have a role in supervision activities, ranging from simply facilitating the organization of supervision missions to the more substantive role of providing backstopping and follow-up on implementation issues. Furthermore, management is developing a proposal for the Fund's new operating model. There are two elements in this model that are likely to have important implications for IFAD's supervision activities. These are: (a) the shift in the unit of account from the project to the country programme level; and (b) the utilization of the Results and Impact Management System (RIMS) to capture and analyse data systematically according to a core set of indicators. IFAD will need to pay attention to these aspects in its future supervision activities.

Direct supervision analysis

Selection of the pilot projects. Most of the 15 projects were selected according to the criteria presented by IFAD to the Executive Board in September 1997. Two projects (in Gaza and the West Bank and Zimbabwe) have been adversely affected by political instability. While the inclusion of the Zimbabwe project in the DSPP was justified, because the unexpected events in the country could not have been foreseen at the time of Board approval, the inclusion of the Gaza/West Bank programme in an unstable country situation did not reflect an appropriate choice for inclusion in the pilot programme. The evaluation also notes that the Bangladesh project was not an appropriate choice in that it was not innovative, replicating broadly the design of a previous IFAD-financed project in the country.

Approaches to direct supervision. There have been various approaches to direct supervision in the 15 pilot projects. These include: (a) intensive CPM involvement in all steps of direct supervision, including fiduciary aspects. In these cases, the role of UNOPS was largely limited to disbursement processing; (b) the CPM completely delegates fiduciary aspects to UNOPS and focuses on implementation support matters, although s/he may not write mission reports and may not be as closely involved in the annual work programme and budget processes; (c) fiduciary aspects are dealt with by UNOPS, and the CPM appoints consultants who would be largely responsible for leading supervision missions and producing the corresponding documentation. The CPM may participate in key stages of the missions. In this approach, the CPM takes a management role and focuses on mission deployment and ensuring that all outputs are produced as required; and (d) this approach is specific to those countries in which IFAD has some form of local representation that plays a part in direct supervision by participating in supervision missions and following up between missions. The distribution of the pilot projects across the four approaches has been fairly evenly spread, with three to four projects each following approaches (a)-(c) and two projects using approach (d).

Implementation of direct supervision. The evaluation reviewed the implementation experience of direct supervision activities as compared with those under supervision by CIs. Its report observes that supervision planning during preparation of country strategic opportunities papers (COSOPs) and the project design phase was given limited attention in both forms of supervision. On another issue, the evaluation notes that the continuity of CPMs in direct supervision activities is important, and that in some pilot projects, there have been more than three CPMs allocated to the project in the last five years. In terms of supervision mission frequency and duration, the directly supervised projects have received close to two supervision missions per year, against one supervision mission for the control group projects. The average mission days per project per year were 15.2 days for direct supervision as compared with 11.2 days for the control group.

The composition of supervision mission teams in the directly supervised projects and in the control group has not been significantly different, with both groups making use of local consultants. There has also been more consultation between IFAD, the country and the project authorities before deciding what expertise to include in each supervision mission, especially in the context of the DSPP. Direct supervision missions on average comprised 2.6 personnel, compared with 2.5 for the control group.

So far, eight of the projects directly supervised have benefited from MTRs, whereas 11 projects in the control group have had an MTR. This is understandable, given that the average implementation period of the directly supervised projects (4.8 years) is less than that of the control group (6.5). In the case of the DSPP, the MTR has offered an opportunity to formalize a number of decisions made during the regular supervision missions that preceded the MTR. That is, given the more constant and intensive interaction between IFAD and the project authorities in directly supervised projects, CPMs have been able to stay on top of implementation issues and make key decisions within the framework of supervision missions themselves, rather than wait until the MTR stage. Hence, the MTR does not appear to have been critical in the context of the DSPP. It has been a critical instrument in CI-supervised projects, as it has provided CPMs the possibility of conducting a comprehensive review of what has been done and of introducing the required corrective measures to project design and implementation arrangements.

The quality of supervision reports varied across the directly supervised projects. In the terms of reference of and reports on direct supervision, attention largely focused on implementation issues and on physical and financial achievements, and less on lessons learned and impact. The same may be said of supervision reports prepared by CIs. However, supervision reports of the DSPP projects provided better coverage of issues of concern to IFAD, such as gender mainstreaming, monitoring and evaluation, beneficiary participation, building of grass-roots institutions and so on. Moreover, the documentation of the directly supervised projects was more complete and more easily accessible at IFAD than that of the control group.

In terms of feedback, the evaluation concludes that the DSPP did not systematically follow the set procedures established for feedback. Often an informal mode of communication was chosen rather than following official channels. In spite of the above, the evaluation notes that there is generally a faster response to project queries and follow-up on supervision recommendations under direct supervision than under CI supervision. The feedback in supervision undertaken by CIs is also erratic. Few CIs produce all the required documentation. The inadequate type and quality of reports and feedback bring up the critical issue of the current lack of quality assurance systems in the overall supervision activities of IFAD, which will be discussed later.

No notable difference in participatory processes was observed between IFAD direct supervision and supervision by CIs. However, there appears to be a clear preference for the increasing trend of conducting joint review missions (as organized by some IFIs) with the governments concerned, as opposed to traditional supervision missions. The term ‘supervision', itself, was described by many partners at the country level as being top-down, one that did not reflect the partnership ethos between the country concerned and IFAD.

Project status reports (PSRs) are an important instrument in IFAD's overall monitoring and reporting system. In PSRs of directly supervised projects, there has been no analysis or lessons on direct supervision processes or information on costs. In addition, PSRs do not include ratings on the IFAD impact indicators for direct supervision, which were developed at the beginning of the DSPP. For CI-supervised projects, CPMs include an assessment of the CI's performance and assign a corresponding rating. For directly supervised projects, PSRs do not include a rating of IFAD's performance in direct supervision. With regard to the process in the preparation of PSRs, it must be noted that the PSRs of CI-supervised projects are done by the CPM concerned, whereas the same CPM responsible for direct supervision prepares the PSR for the project under his/her direct supervision.

One important message emerging from the evaluation is the very wide support by IFAD partners of direct supervision. Government authorities, development organizations and project authorities were all of the unanimous opinion that there were advantages in having a direct contact with IFAD through CPMs. For example, partners conveyed the view that direct communication and interaction with IFAD contribute to better implementation and a stronger partnership, be it in terms of policy dialogue, identification of future pipeline and cofinancing opportunities or knowledge-sharing. There was one case in which a criticism was not of direct supervision, as an approach, but of the frequent staff changes at the CPM level that the project had experienced. The perceived lack of seniority and qualifications of some staff assigned by IFAD for this task were also raised for some projects.

Fiduciary aspects (under the formal responsibility of UNOPS) of directly supervised projects perform better than those of the control group. This finding is consistent with the ESM finding that UNOPS tends to perform better than other CIs. As UNOPS is the only CI involved in the DSPP, it is not unexpected that there would be a higher level of performance than for the control group, which incorporates a variety of CIs.

Project implementation performance. The evaluation first compared project implementation performance using indicators and ratings contained in the PSRs for both the directly supervised projects and the control group. Thereafter, the evaluation used the same set of indicators included in the PSRs and, based on its own independent ratings, compared the implementation performance of the directly supervised projects with those supervised by CIs.

From the evaluation's analysis, it is evident that the directly supervised projects perform better than CI-supervised projects across the PSR indicators in terms of, for example: compliance with loan covenants (directly supervised projects were rated on average 3.4 as against 3.1 for CI-supervised projects);14 performance of M&E systems (3.2 for directly supervised projects against 2.6); availability of counterpart funds (3.5 against 3.2) and so on. One explanation might be more optimistic reporting by CPMs, who, as mentioned previously, are themselves responsible for the preparation of the PSRs. However, the evaluation's independent assessment also demonstrates that, on the whole, directly supervised projects have a better implementation performance as compared with CI-supervised projects across the PSR indicators. All rating scores may be seen in Chapter III, Tables 4 and 5.

The comparison of the directly supervised projects with the 15 projects in the control group was also developed by assessing the average time from approval to effectiveness. The analysis reveals more favourable performance by directly supervised projects. That is, the average time lag is 15.36 months for directly supervised projects as compared with 17.21 months for all other projects in the same country. Moreover, in terms of disbursement performance, the average rating by the evaluation team for directly supervised projects is 2.6 as compared with 1.9 for CI-supervised projects in the control group. For example, the average cumulative disbursement rate for all directly supervised projects in the fifth year of project implementation was around 62% as compared with 43% in the control group. Chapter III, Figure 1 provides more information on disbursement performance.

Finally, the evaluation reviews one indicator that is not included in the PSRs: the time overrun factor in project implementation (number of years/months a project is extended beyond the original completion date). Based on the calculations made by the evaluation, the overall time overrun for directly supervised projects is on average 0.54 years, as compared to 1.4 years for projects in the control group. Time overrun is an important indicator, as it reflects the soundness of design, but also the ability of the supervision process to recommend timely corrections and improvements during implementation, as and when required. Time overrun is also significant because it has an administrative cost implication for the Fund: additional supervision costs and related staff time need to be allocated for the time that projects run beyond their original completion date.

The human (social) dimension in supervision and attention to key areas of IFAD's catalytic role. There is evidence based on the evaluation's analysis that through direct supervision, as compared with supervision by CIs, IFAD has paid more attention to issues such as the targeting of women (rated 3.1 for directly supervised projects, compared with 2.5 for projects supervised by CIs); targeting the poor (rated 3.0 for directly supervised projects against 2.4); beneficiary participation; gender mainstreaming; participatory monitoring and evaluation and so on. These trends are by and large consistent with the ratings included in the PSRs of the directly supervised and control group projects. Direct supervision has also provided IFAD an opportunity to focus more on issues such as innovation (rated 3.2 against 2.5 for CI-supervised projects), partnership, knowledge management and policy dialogue, which according to the IFAD strategic framework contribute to improving project performance and impact. In each country exposed to direct supervision, according to the evaluation and partners at the country level, the presence of the CPM during supervision is seen as an opportunity to advance IFAD's broader objectives, such as those listed above. Chapter III, Figures 2 and 3 provide all comparative rating scores on the aforementioned indicators for the DSPP and projects supervised by CIs.

The evaluation's rating for knowledge management of directly supervised projects (rating 2.7) is only marginally better than that of projects in the control group (2.5). This is partly explained by the fact that, while the knowledge acquired through direct supervision at the individual CPM level was high, there was no systematic effort to document, analyse or share such learning from direct supervision. Moreover, no specific activities were conducted or resources allocated to knowledge management in the directly supervised projects, and their supervision reports did not emphasize lessons learned. The same is true for the projects in the control group. However, this is considered a particular shortcoming in the case of direct supervision, as knowledge management in the broader sense was a specific objective of the DSPP (which is not the case for CI supervision).

Development effectiveness. The first thing to note is that, contrary to the requirement of the action plan adopted by the Governing Council, no specific, ongoing monitoring and reporting mechanisms to trace project impact were established, neither at the individual project level nor for the DSPP as a whole. In addition, contrary to the recommendation in the Report of the Joint Review on Supervision Issues for IFAD-Financed Projects, "for comparative analysis a control group of CI-supervised projects similar in other respects to the directly supervised projects" was not identified and monitored. Nevertheless, the evaluation made efforts to analyse the development effectiveness of the directly supervised projects and compare them with projects in the control group.

The evaluation has undertaken three specific types of comparison that, taken together, provide an overview of the development effectiveness of the directly supervised projects in relation to the CI-supervised projects. Tables 6, 7 and 8 in Chapter III contain the specific rating scores that provide the basis for the analysis.

First, based on three indicators (quality of supervision recommendations, follow-up actions and support provided to the project), the overall quality of supervision was assessed (see Chapter III, Table 6). These indicators are also used by the World Bank's Quality Assurance Group in determining the quality of the Bank's supervision, which is considered an important ingredient in achieving development effectiveness. In fact, in one of their recent evaluations, the Operations Evaluation Department of the World Bank concluded that well-supervised projects are twice as likely to succeed as are poorly supervised ones. The DSPP evaluation's assessment according to the three indicators reveals a positive trend in favour of directly supervised projects. For example, quality of supervision recommendations is rated 3.3 for DSPP, compared with 2.9 for CI-supervised projects.

Second, the evaluation compared directly supervised and control group projects using the 11 IFAD impact indicators for the DSPP.15 By and large, the directly supervised projects do better than those supervised by CIs across all indicators (see Chapter III, Figure 4). For example, "identification of new project concepts for inclusion in the pipeline" is rated 2.9 for DSPP as compared with 2.1 for CI-supervised projects. However, the control group projects perform marginally better in terms of timeliness of reporting (rated 2.9 for projects supervised by CIs, compared with 2.8 for DSPP). The reason is partly that, as subcontractors, CIs pay specific attention to project reporting. This issue has not been given the same level of attention in direct supervision, given the constant contact between IFAD and the project authorities. It can also be seen that the ratings for costs appear to favour CI supervision, as supervision by CIs costs less on average (this will be discussed in more detail later).

Third, the evaluation rated the directly supervised and CI-supervised projects using the six impact domains in the OE methodological framework for project evaluation (MFE). Moreover, the two overarching factors in the MFE (sustainability and innovation/replication) have been included in the analysis (see Chapter III, Table 7). In short, the directly supervised projects are rated better in most impact domains (e.g. food security is rated 2.9 for DSPP, compared with 2.5 for CI-supervised projects), although in two impact domains (environment and sustainability), the directly supervised projects performed less well as compared with the CI-supervised projects.

With regard to the latter issue, it should be noted that 12 of the 15 projects in the control group included a specific objective or component related to environmental matters. Consequently, their supervision missions have normally adequately reviewed project progress in this area. On the other hand, only five directly supervised projects have a specific environmental component and their supervision missions have not always included the required skills to undertake a thorough assessment of environmental issues and to provide the necessary backstopping to project staff. Furthermore, environmental matters have not received the same level of focus within IFAD at large as compared with areas such as gender mainstreaming and social capital formation.

No conclusive statement can be made on project sustainability at this stage. On the one hand, the slightly less positive ratings of the directly supervised projects as compared with the control group may be explained partly by the relatively fewer years of implementation of the directly supervised projects. That is, many of the projects in the DSPP have devoted greater emphasis, at least in the initial years, towards developing grass-roots organizations and promoting participatory processes and less towards productive activities. This may be a cause of the lower sustainability rating at this stage. Moreover, some of the directly supervised projects (in Brazil, India and the Sudan, for example) suffered initial delays in implementation due to compelling political and administrative circumstances, which could be another reason. On the other hand, the IEE concluded that during the early stages of a project there are relatively high expectations that project benefits will endure, but these expectations are modified in the later stages of implementation.

Finally, the overall composite rating16 of the six impact domains is compared with the rating available in the PSR on ‘meeting development objectives'. In both cases, directly supervised projects do better than CI-supervised ones (see Chapter III, Table 8). The MFE composite index is rated 2.9 for the DSPP as compared with 2.5 for CI-supervised projects.

Some general considerations must be made while interpreting the above results. First, it must be acknowledged that the differences in rating scores are relatively small. Nevertheless, the analysis shows that directly supervised projects have performed better, compared with the CI-supervised ones, according to most indicators considered in this evaluation. Additionally, direct supervision has contributed to enhanced results in furthering IFAD's broader objectives, such as policy dialogue, partnership-building and knowledge management. The evaluation also notes that the results of direct supervision could have been even greater had the pilot programme been implemented under more favourable conditions, for example if the CPM workload had been appropriately prioritized or if adequate monitoring and assessment systems had been put in place. However, in analysing the results, one must also take into account the time allocated to direct supervision. More than half the CPMs involved in DSPP said they spend up to double the time on direct supervision than on other projects in the portfolio. There are also issues of cost, which will be discussed later.

Quality assurance. IFAD lacks a continuous quality assurance system for supervision, which would have allowed the Fund to meet more fully the objectives of the pilot programme. As a result, there was limited quality assurance in direct supervision inputs and processes (e.g. mission terms of reference, composition and duration in the field) and in deliverables such as supervision reports. Other IFIs, in particular the African Development Bank (AfDB) and the World Bank, also have quality assurance mechanisms that allow for periodic assessment of supervision activities (e.g. once every two years). Through these quality assurance mechanisms, IFIs are able to take a holistic view of the supervision function and suggest corresponding systemic improvements across the organization. Such quality assurance mechanisms in the aforementioned IFIs are located outside the operations departments. The Inter-American Development Bank (IDB) is in the process of setting up a similar quality assurance system. In addition, IFIs generally have quality assurance mechanisms for supervision built into the operations divisions (e.g. at the African Development Bank, peer reviews are used for quality assurance).

Reporting to the Executive Board. Management has not complied with two key commitments in terms of reporting. First, no MTR of the pilot programme was undertaken in the last quarter of 2000/first quarter of 2001, as had been decided by the Governing Council. The reasons for this are neither evident nor documented. Second, although the Fund has provided progress reports on direct supervision to the Board on an annual basis, these have focused mainly on the deployment of inputs and processes in implementing direct supervision activities. The evaluation notes that, at the same time, the Executive Board could have exercised better oversight of implementation of the action plan and could have demanded more analytical information from management on the development effectiveness of the DSPP, in particular on the costs, results and lessons learned of the pilot programme.17

Operating environment for direct supervision. CPMs did not benefit from a favourable operating environment in which to manage the increasing workload caused by the introduction of direct supervision. More specifically, training was not provided at the outset of the pilot programme, nor were CPMs given any particular recognition or incentives. They accepted the additional responsibilities for direct supervision, which they undertook without managerial support for any reprioritization of their existing workload. Last, but equally important, management engagement appeared to diminish gradually after approval by the Governing Council of the DSPP, as demonstrated, for example, by the non-implementation of three key activities18 included in the action plan.

Learning and knowledge management

By participating in direct supervision activities, CPMs have acquired first-hand experience of the task of supervision and a better appreciation of the issues related to rural poverty reduction. The benefits of their experience are evident, as some CPMs have put their newly acquired knowledge to use in designing new projects and implementing ongoing ones. Still, supervision reports (of both direct supervision and CIs) focus more on implementation issues and less on lessons learned from supervision processes or broad rural development issues. This has led to missed opportunities to incorporate learning into project design across the institution and to improve the supervision activities of cooperating institutions in general. Nonetheless, in spite of the benefits to individual CPMs, there is a need to institutionalize this knowledge and establish systems/platforms through which CPMs involved in direct supervision could share their overall experiences among themselves and within the Fund. Some exchanges have occurred at the CPM level in an informal, unstructured manner. But, overall, institutional support to capture and channel learning from direct supervision to IFAD staff has been inadequate.

IFAD also lacks an adequate reporting and feedback mechanism at the country level: its current system for learning does not allow governments and other development organizations to become familiar with its successful innovative approaches and to learn from its experiences. This needs to be addressed in the light of the Fund's strategic objective of seeking partnerships for replicating and scaling up the activities it finances, as well as of the proposed new operating model. OE recognizes that knowledge management is an institution-wide concern and that direct supervision is only one of the components of the Fund's learning system. However, the unique opportunity for the institution as a whole to learn from the direct supervision pilot programme to enhance IFAD operations has been largely missed.

Efficiency of direct supervision

Based on the calculations of the evaluation, the average cost per year per project of direct supervision is around USD 93 000 as compared with USD 61 000 for CI-supervised projects in the control group. In this regard, it should be noted that the costs incurred go beyond the expectation of the Governing Council, which had decided that in implementing direct supervision activities "there would be no cost increase" to IFAD.

However, there are some points related to the average CI cost that merit being highlighted. First, there is quite a variation in costs across the different CIs. Hence, the average cost for CI supervision calculated above does not reflect the costs that all CIs charge. For instance, supervision through some CIs (e.g. the World Bank) costs over USD 100 000.19 The current overall cost of supervision through UNOPS (the CI with the greatest number of projects under supervision) is around USD 79 000 – see Chapter V for more data on costs. Moreover, according to the evaluation, the current cost attributed to UNOPS needs to be increased if the Fund is to expect them to provide enhanced quality services.20 Next, there are additional costs of CI supervision that are difficult to identify and have thus not been included in calculating the total cost of CI supervision. For instance, the Belgian Survival Fund has an annual allocation for CI supervision in its administrative budget, recorded outside IFAD's accounting system. Along similar lines, more recently, IFAD's Asia and the Pacific Division provided additional funds to the UNOPS Asia Office to augment its supervision-related activities. These and other such costs are not included in the average of USD 61 000 calculated for CI supervision. Finally, it should also be noted that the cost to IFAD for direct supervision includes the fielding of around two supervision missions per year to each project for longer durations than those of the average CI supervision activities.

Furthermore, the evaluation underlines that the longer implementation period of projects in the control group as compared with the directly-supervised projects has administrative cost implications for IFAD, which would raise the overall costs of supervision by CIs and which need to be considered.

The evaluation also argues that there are possibilities of reducing the costs of direct supervision to some extent. For instance, this could be achieved by making use of competent national entities to discharge the fiduciary responsibilities involved in supervision and thus enhancing the role of government in implementation support activities. Greater use of local consultants could also contribute to cost savings. Finally, as CPMs and the other IFAD staff involved acquire the necessary competencies and gain more experience in direct supervision, it is fair to assume that the overall time invested in the associated tasks is likely to reduce to some extent. This will have a corresponding effect on the staff costs component, leading to a reduction in the overall costs of direct supervision.

While the cost of direct supervision may still be higher than that of CI supervision, according to the evaluation it is paramount to assess the cost together with the corresponding benefits to the Fund. The analysis of the evaluation illustrates that direct supervision has contributed to better development effectiveness and has allowed the Fund to further its catalytic objectives of innovation, policy dialogue and partnership development. With regard to the latter point, partners at the country level expressed their preference for building partnerships directly with IFAD rather then managing such processes through proxy institutions such as CIs. Moreover, in the context of knowledge management, although the CPMs involved in direct supervision have acquired better understanding of implementation matters and despite the fact that knowledge from the DSPP has not been properly institutionalized, the evaluation observes that using a CI for supervision introduces an extra layer into the already feeble learning loop of the Fund. On a similar note, the knowledge that the staff or consultants of CIs have acquired by undertaking supervision on behalf of IFAD is largely lost to the Fund. Finally, the common opinion of governments and all other partners at the country level, who clearly favour IFAD's direct involvement in supervision activities, must also be given due consideration.

Supervision systems and experiences of other International Financial Institutions

By and large, the AfDB, the Asian Development Bank (AsDB), the IDB and the World Bank have similar supervision systems, with processes and details applied variously to fit their institutional structures. In the IFIs, generally, supervision is not limited to official missions and formal reports, rather it is a continuing and flexible process, specific to the needs of the particular operation and intended to foster a close partnership among an IFI, its borrower, and the implementing agencies. The planning of supervision is done carefully during the design phase of the project cycle, when appropriate resources and arrangements are put in place to facilitate supervision. The supervision process normally starts after the project is approved and ends when the last disbursement is completed and the project completion report is prepared.

At the Inter-American Development Bank, supervision is largely delegated to the country offices, with participation of staff from headquarters as required. At the Asian Development Bank and the World Bank, headquarters and country offices share responsibility for supervision. That is, in some cases, supervision responsibilities are entirely delegated to the country offices, especially in those countries in which offices have staff with the required sectoral know-how. Under such arrangements, selected staff at headquarters are sometimes asked to join supervision missions. In other cases, the "task managers", if based at headquarters, retain responsibility for supervision and, in turn, involve staff at their country offices in supervision missions. At the African Development Bank, so far, task managers based at headquarters have full responsibility for project supervision. However, with the establishment of 26 country offices by the end of 2006, AfDB expects in-country staff to be involved in one way or another in project supervision and related follow-up, as well as in portfolio management issues. One important aspect of supervision by these IFIs is that certain supervision functions are being increasingly located in their country offices. For example, the review of documents related to fiduciary aspects (such as bidding proposals and accounting matters) is largely handled by the country offices. It is important to realize, however, that the degree to which such field office staff can make decisions varies according to the delegation provided by the responsible task managers. Finally, some IFIs have well-established quality assurance mechanisms for supervision and others are rapidly moving in the same direction.

Conclusions

A consistent trend in the overall evaluation analysis demonstrates that, compared with CI supervision, direct supervision has greater potential to contribute to better development effectiveness at the project level and, at the same time, allows greater attention to IFAD's broader objectives at the country programme level. That is, direct supervision by IFAD can contribute to better and timelier project implementation, which in turn enhances overall results and impact. Moreover, through direct supervision, IFAD has been able to emphasize issues of prime concern such as gender mainstreaming, targeting and the building of grass-roots institutions, which taken together are important elements in ensuring sustainability.

In view of their more frequent and longer presence in countries with direct supervision, CPMs have wider opportunities to advance IFAD's objectives at the country programme level, including policy dialogue and partnership development. CIs do not consider these to be a priority, nor can such activities be effectively advanced through third parties. Although there is no conclusive evidence that new partnerships have resulted from the DSPP, governments and other development partners at the country level have unanimously expressed their deep appreciation of the more frequent contact with CPMs, which has been facilitated by IFAD direct supervision activities. The same partners communicated that they find it more useful to deal directly with IFAD staff rather than with CI representatives. In this regard, for example, partners affirmed that the response rate and follow-up on implementation issues are faster under direct supervision than under supervision by CIs.

Direct supervision has contributed to developing IFAD's knowledgebase. In this regard, in particular, the CPMs responsible for direct supervision have acquired knowledge of supervision processes, project implementation and rural development issues in the countries concerned. This knowledge has enabled them to better design and implement new operations. However, the knowledge gained at the CPM level has not been systematically shared with others, nor has it been institutionalized, which is one of the key shortcomings of the DSPP. The evaluation notes that supervision by CIs also offers possibilities for knowledge generation. However, CI involvement in supervision makes the transmission line of knowledge from the CI to IFAD and the country more cumbersome in the already feeble knowledge systems of the Fund.

The evaluation concludes that direct supervision allows CPMs to strengthen country-level coordination both within the context of IFAD operations and with the development community at large. It also facilitates the strengthening of existing IFAD-funded programmes and the identification of new programmes and cofinancing opportunities, which are mostly available at the country level, given that the majority of our international and bilateral partners have delegated an increasing amount of authority to their country representatives.

Unlike most other IFIs, IFAD lacks a quality assurance system for direct supervision. As a result, the DSPP was approached and implemented in a variety of ways, based on the perception and understanding of individual CPMs. Compliance with the minimum supervision requirements and the direct supervision guidelines was also not monitored. In conclusion, both continuous and periodic quality assurance systems are fundamental if direct supervision activities are to be expanded.

At face value, the average cost of direct supervision (USD 93 300) is higher than the average cost of supervision by CIs (USD 61 461)21. However, the evaluation argues that cost should not be seen in isolation from the benefits the DSPP has evidenced. Moreover, in discussions with UNOPS (the main IFAD CI), it is clear that the costs to IFAD for supervision by UNOPS need to be increased if they are to deliver the type and quality of service IFAD requires in the future. In parallel, the evaluation notes that there is the potential for efficiency gains in direct supervision if, for example, the fiduciary responsibilities related to supervision are entrusted to competent national entities and greater use is made of local consultants for implementation support activities.

The evaluation notes that management's interest has appeared to gradually diminish following approval of the DSPP by the Governing Council. This is illustrated by the fact that management did not fully implement all decisions of the Governing Council. For instance, it did not undertake a mid-term review of the DSPP as required. Neither did it establish an integrated, analytical accounting system to track costs of the DSPP. Nor did it set up a monitoring and assessment system to measure the performance and impact of direct supervision. According to the evaluation, however, neither did the Executive Board exercise adequate oversight to ensure that IFAD management would fulfil all its commitments under the DSPP. The evaluation believes that the outcome of the DSPP would have been even more significant had all the requirements laid down by the Governing Council been implemented.

The evaluation reveals that not all concerned have the same understanding of the notion of supervision. In fact, there is often confusion as to what constitutes supervision missions, implementation support, follow-up activities, fiduciary responsibilities and so on. There is also a lack of clarity on the roles and responsibilities of IFAD, CIs, project staff and government authorities. Additionally, it is worth noting that many partners at the country level felt that the term ‘supervision' – when applied to the implementation aspects of projects rather than to fiduciary aspects – has a paternalistic undertone and they felt uneasy with its continued application. Supervision in that sense reflects a top-down, non-participatory approach to the function, which is inconsistent with the Fund's objectives of promoting ownership and partnership with governments and other institutions.

Recommendations

It is important to state that the evaluation's recommendations have taken into consideration the relevant recommendations contained in the ESM report. The evaluation's five key recommendations, given below, are mutually reinforcing. The evaluation recommends that they all be implemented fully in order to ensure that the desired impact is achieved in IFAD's future efforts in the area of supervision and implementation support.

Recommendation one: definition of supervision

The evaluation recommends that the concept of ‘supervision' as used by IFAD be divided into two distinct operational parts: (i) supervision of fiduciary aspects, including aspects related to procurement review, disbursement processing and compliance with financial and auditing requirements; and (ii) support to programme and project implementation.22 This would include, for example, the organization of periodic ‘implementation support' missions and related follow-up; an assessment of the achievement of programme/project objectives and assistance in identifying remedial solutions for implementation challenges, based on interaction and dialogue with project authorities and other partners at the country and project level; and the provision of guidance in preparing the annual work plans and budgets. It would also include oversight of project and programme implementation, for example, in terms of monitoring the achievement of physical targets.

Recommendation two: develop a comprehensive supervision23 and implementation support policy for IFAD

The Fund should develop a specific overall supervision and implementation support policy for its operations. The policy should reflect the following elements.

Supervision of fiduciary aspects. IFAD should be allowed to decide, on a case by case basis, whether to subcontract a competent national, regional or international entity to perform such functions. Special efforts should be devoted to engaging national entities, as this would have the double effect of building local institutional capacity and reducing costs. The utmost attention should be given to ensuring that there is no conflict of interest between the prospective national entity and the IFAD operation under consideration. In a few and very specific circumstances, IFAD might consider undertaking the supervision of fiduciary aspects itself.

Implementation support. The evaluation recommends that the policy state explicitly that: (i) IFAD should be made responsible for providing direct implementation support to all its operations globally. In this regard, it is important to specify the role of the CPMs, who could either be intensively involved themselves as "implementation backstoppers" or act as implementation-support task managers with more attention to process management, which would also require a degree of direct involvement in activities. In both cases, CPMs would ultimately be responsible for the process, content and outputs of implementation support activities; (ii) such support would cover all aspects of IFAD country programmes, both at the project level and beyond, such as policy dialogue and partnership strengthening; and (iii) the role of partner governments would be specified and given due emphasis, which would contribute to building greater ownership and local capacities, as well as reducing costs.

The above would result in new responsibilities for the Programme Management Department (PMD), which would require the allocation of additional staff and financial resources as well as new competencies and skills. It is fundamental that the additional resource requirements for the implementation of the new policy be clearly articulated in a comprehensive and explicit manner by IFAD. This would require a detailed cost analysis, particularly of the elements in paragraphs 65-66, as well as an assessment of the skills and competency of current CPMs. Based on the aforementioned analysis, IFAD would need to develop a proposal to meet the cost deficits in implementing the new policy. It would also need to develop a plan for enhancing the ability of CPMs to meet the specific requirements of the new policy, recognizing that it may not be possible to enhance these skills and competencies in all cases. Until the required level of financial resources would be made available to the Fund and additional PMD staff recruited and their competencies and skills developed and upgraded, IFAD might consider a phased approach to expanding implementation support in all operations.

The success of the new policy would also be determined by the support provided by IFAD management and the conducive environment it creates for the purpose. For example, management would need to: (i) ensure that appropriate opportunities are introduced for periodic staff training; (ii) establish an incentive framework and platforms for the sharing of knowledge acquired by CPMs; and (iii) allocate the necessary time for reviewing implementation experience under the policy. The Board should also play a proactive role in exercising oversight of implementation of the new policy and in approving administrative budgets for the purpose.

The policy should be evaluable, and in particular include a roll-out and implementation plan, with performance indicators that can be monitored periodically.

A chapter on supervision and implementation support should be included on a standing basis in IFAD's annual Portfolio Performance Report. The chapter would provide an analytical account of the opportunities and challenges in the area, and identify key lessons learned. Moreover, it would provide an indication of the ongoing operational measures introduced by IFAD to address emerging issues.

The undertaking of supervision of fiduciary dimensions and implementation support would require revisiting Article 7, Section 2(g) of the Agreement Establishing IFAD.

Other integral aspects that the policy should consider are contained in recommendations three to five.

Recommendation three: supervision and implementation support in the framework of the COSOP

The evaluation recommends development of an overall approach to supervision and implementation support during the preparation of COSOPs. This would take into account the need to supervise the fiduciary dimensions of all operations, and the provision of implementation support to the country programme, including areas such as policy dialogue, partnership-building and knowledge management, in addition to the support traditionally provided to projects. The COSOP would lay the provisions for the need to develop an annual supervision and implementation support plan for each country, indicating the specific objectives, human and financial resource allocations and expected results. Each PMD regional divisional would set up an electronic monitoring, assessment and reporting system, which would serve as a management tool to track the implementation progress of the plans and to flag issues requiring more immediate follow-up. A template should be developed by PMD for the section on supervision and implementation support that would be included in the COSOPs.

Recommendation four: quality assurance system

Within the framework of an overall enhanced quality assurance system at IFAD, there is a need to introduce quality assurance mechanisms for the supervision of fiduciary dimensions and implementation support activities. The evaluation recommends that IFAD establish a management review committee within PMD, which would review supervision and implementation support activities, results and related operational issues. Quality assurance, as well, needs to be strengthened in the PMD divisions. In this regard, semi-annual reviews of supervision and implementation support activities should be undertaken at the divisional level. Summaries of the discussions at these meetings should be circulated to all PMD divisions. Moreover, IFAD should build on the experience of other IFIs (in particular of the Quality Assurance Group at the World Bank) to establish an IFAD-specific quality assurance group, which would review aspects of supervision and implementation support, in addition to any other aspects related to implementation of the COSOP and its components. The Fund would need to thoroughly reflect upon the most appropriate location within IFAD's organizational structure for such a group, which would ensure the most objective and independent review possible of its supervision and implementation support efforts. The introduction of such a quality assurance group should take into consideration the mandates and performance of existing quality control systems within IFAD, such as the project development team, Technical Review Committee and Operational Strategy and Policy Guidance Committee.

Recommendation five: learning and knowledge management

Incentives should be provided to staff to encourage sharing of the knowledge they acquire. For example, time needs to be carved out in CPMs' workload for sharing the knowledge they generate through supervision and implementation support activities. The documenting and sharing of knowledge should be included as an indicator in assessing the annual performance of CPMs. Specific instruments need to be established to facilitate learning and knowledge-sharing. In particular, time should be reserved on a standing basis in the CPM forum for discussing issues and sharing knowledge generated through supervision-cum-implementation support activities. Each project mid-term review and project completion report should include a specific treatment of supervision and implementation support issues, as should all evaluations undertaken by OE. The project and country status reports should be reformatted to include a narrative section on supervision and implementation support, and ratings must be included in all cases. Other instruments should be introduced, such as peer reviews at the PMD divisional level in relation to implementation support activities.

The monitoring and evaluation systems at the project level need significant strengthening if they are to contribute effectively to learning. Moreover, in line with the new operating model, it is necessary to assist in the development of integrated monitoring and evaluation systems at the country level. This would not only facilitate monitoring and sharing of experience across the entire project/programme portfolio, but would also allow for tracking the implementation of the broader objectives of IFAD country programmes, such as policy dialogue and partnership-building.

IFAD should build on the experiences of other international financial institutions, and make more comprehensive use of information technology for knowledge management purposes in relation to supervision and implementation support. In this regard, it is recommended that the existing PPMS be expanded so that it can carry updated summaries of supervision and implementation support activities at all times. An enhanced PPMS should accordingly be made accessible to external partners through the IFAD internet with immediate effect. Last but not least, an integrated, analytical accounting system should be developed through close cooperation between PMD and the Office of the Controller. This would allow the monitoring and analysis of all costs related to supervision-cum-implementation support, including staff time.


1/ The resolution will cease to be effective in June 2006, as the last project (India) became effective in June 2001.

2/ For the purpose of this evaluation, the term ‘development effectiveness' encompasses the extent to which the DSPP's overall objectives have been met, the efficiency in implementing the pilot programme and the contribution of direct supervision in improving project implementation and potential project impact.

3/ For definitions of the criteria, see Methodological Framework for Project Evaluation (document EC 2003/34/W.P.3).

4/ These impact indicators were developed and presented by IFAD management to the Executive Board in April 1999 (see Box 1 of the present document and document EB 99/66/R.10/Rev.1).

5/ In Armenia, the Gambia and Uganda.

6/ In Bangladesh, Benin and Indonesia.

7/ In Armenia and India.

8/ In Bangladesh, Guinea, Indonesia and Peru.

9/ The role of the CLP was to provide comments and input at several key stages in the evaluation process (for a definition of the CLP, see paragraph 33 in the IFAD Evaluation Policy, document EB 2003/78/R.17/Rev.1). The members of the CLP for the DSPP evaluation were: Mr Nigel Brett, Mr Jim Carruthers, Mr Pablo Glikman, Mr Shyam Khadka, Mr Luciano Lavizzari, Mr John McGhie, Mr Ashwani Muthoo, Ms Rasha Omar, Mr Mohamed Tounessi and Mr Joseph Yayock.

10/ Mr Hans Wyss, former Director of Operations at the World Bank.

11/ Hence field work was undertaken in 13 countries of the DSPP (Gaza and the West Bank and Zimbabwe were excluded).

12/ The action plan can be seen in its original format in Appendix II.

13/ In December 2003, the Executive Board approved the field presence pilot programme in 15 countries globally.

14/ The rating scale used is from 1 to 4, where 1=negligible, 2=modest, 3=substantial and 4=high.

15/ The list of impact indicators may be seen in Chapter II, Box 1.

16/ The composite rating is the average of the six impact domains and the two overarching factors.

17/ In 1997 the GC had, indeed, decided that "progress, lessons learned and results of the test would be reported annually to the Executive Board" – see document GC 20/L.10/Add.1.

18/ Items 6, 7 and 8 under Recommendatio 5 in the action plan – see Appendix II.

19/ A study by the World Bank's Quality Assurance Group in 2003 concluded that, on average, from USD 100 000 to USD 125 000 should be allocated per year for supervision of community-driven development projects.

20/ The IEE report highlighted the same issue – see paragraph 18 on page 16 (document EB 2005/84/R.2).

21/ Although the actual cost per project per year for supervision by UNOPS, the main IFAD CI, is around USD 79 000.

22/ It could also be called ‘implementation support'.

23/ The term ‘supervision', from this point onwards in the document, means ‘supervision of the fiduciary aspects' related to IFAD financing.

LANGUAGES: English

Supervision modalities in IFAD supported projects

enero 2004

Exexutive summary

Introduction1

Improving the quality of project implementation and achieving better results on the ground has been a priority for the International Fund for Agricultural Development (IFAD) since the early 1990s. This gave rise to a series of studies to examine and clarify the role of project supervision in the IFAD project cycle2. In 1996 the Programme Management Department (PMD) commissioned a Joint Review of Supervision Issues in IFAD Financed Projects in collaboration with four major Cooperating Institutions (CIs). The review presented five recommendations: (i) introduce and enforce Minimum Supervision Requirements (MSRs) for the CIs; (ii) improve coordination procedures between IFAD and the CIs; (iii) establish an efficient portfolio management system; (iv) strengthen IFAD priority areas of getting results on the ground and reinforce the learning loop; and, (v) undertake an experimental direct project supervision programme by IFAD for 15 of its initiated projects (three for each regional division).

The Governing Council (GC) approved the recommendations of the Report of the Joint Review on Supervision Issues and an associated five year Plan of Action (GC 20/Resolutions, 21 February 1997, 102XX). The five year Plan of Action (1997-2001) provided detailed steps and timeframes to guide implementation of these recommendations by PMD. As the Plan of Action was to be completed by December 2001, the Office of Evaluation (OE), in agreement with PMD, decided to undertake an Evaluation of the Supervision Modalities in IFAD projects.

In addition to the implementation of the Plan of Action, supervision of IFAD financed projects since 1997 has been affected by the emergence of new areas of IFAD priorities. These have been fully taken into consideration by the evaluation, they include: (i) more emphasis on IFAD specific aspects, i.e. participation, targeting, gender mainstreaming and gender sensitive implementation, and stressing impact achievement and assessment during implementation; (ii) the emergence IFAD's new strategic imperatives as articulated in its Strategic Framework 2002-2006 and the efforts to strengthen IFAD's field presence and accountability; and (iii) changes in the nature of IFAD projects from input/output orientation to process orientation and community-driven approaches.

The evaluation objectives were to assess: (i) the effectiveness of current supervision modalities against the MSRs and others indicators of quality and review the achievements under the Five-Year Plan of Action; (ii) the adequacy of current supervision modalities seen from different partners perspectives; (iii) the efficiency of current supervision modalities; and, (iv) the current relevance of the 1998 MSRs. As most of the directly supervised projects are in the early stages of implementation, the evaluation only reviewed emerging characteristics of the direct supervision programme, including an initial analysis of the cost structure.

The evaluation methodology consisted of: (i) a desk review (DR) and assessment of supervision reports in 57 representative projects by regions and CIs; (ii) field partners' perspectives from a survey of 112 project management units (PMUs) (response rate 60%) cross-referenced through field visits to selected ongoing projects; (iii) IFAD staff perspectives from questionnaires sent to 65 Country Portfolio Managers (CPMs), regional directors and other operational staff (63% response rate), direct discussion with IFAD staff and PMD's assessment of project supervision in the project status reports (PSRs); (iv) perspectives of managers and staff of the two main CIs – the United Nations Office for Project Services (UNOPS) and the World Bank; and, (v) analysis of supervision costs collated by the IFAD Office of Internal Audit (OA).

Supervision of IFAD projects

Four supervision functions have been defined for IFAD (as per International Financial Institutions (IFIs) practice) and elaborated in the 1996 Report of the Joint Review (Box 1). Because of their open-endedness, the third and fourth functions (referred to jointly as implementation support by the supervising entity) have a large resource absorption potential. In addition as part of the 1997 Action Plan IFAD developed in 1998 the MSRs for its CIs (Table 1 of the Annex).

 

Supervision Functions

  1. Supervising the procurement, disbursement and end use of funds (mandatory, fiduciary or core)

  2. Monitoring compliance with loans/grant contracts (mandatory, fiduciary or core)

  3. Facilitating implementation by helping borrower's interpret and respond to the lender's requirements (discretionary)

  4. Providing substantive implementation assistance to borrowers (discretionary)

Source: Report of the Joint Review, 1996

Supervision during project implementation, despite its importance, is only one of many factors that determine project performance. These factors include: (i) quality of the project design and preparation; (ii) quality of project management; (iii) the political and economic environment; (iv) commitment and ownership by government, co-financiers and implementation agencies; and, (v) quality and timing of supervision inputs.

Since the GC decision of 1997 IFAD has reduced the number of CIs used by one (Inter American Development Bank) while the share of the African Development Bank (AfDB) and the Asian Development Bank (AsDB) in supervising IFAD projects declined markedly. The five main CIs (UNOPS, World Bank, Corporation Andina de Fomento (CAF), the Arab Fund for Economic and Social Development (AFESD) and Banque Ouest Africaine de Developpement (BOAD) have increased their share of supervising IFAD projects (from 80% to 85%) between 1996 and 2002. These CIs, with IFAD direct supervision, were responsible for supervising 92% of the portfolio in December 2002. (Table 2 of the Annex). The evaluation concentrated its analysis on these five CIs and to the extent possible IFAD direct supervision.

In addition to the implementation support function embodied in project supervision by CIs (paragraph 6), IFAD uses the term "implementation support" (sometimes also referred to as implementation follow up by IFAD) to pursue its policy stated in the Lending Policies and Criteria3 (1994 version, paragraph 43): "The Fund... will itself actively participate in [the supervision of project implementation], in order to ensure observance of its lending policies and criteria". This activity/ies consists mainly of missions by IFAD staff and specialized consultants to visit projects for safeguarding IFAD's specificity and to supplement and strengthen the supervision by CIs. IFAD staff have access to various sources of funds to supplement CIs work in the implementation support area. The evaluation briefly reviewed this function as exercised currently by IFAD.

Effectiveness of current superivison modalities

Desk teview of supervision quality

Quality of supervision. The evaluation considered four elements as indications of the quality of supervision (i.e., its effectiveness) based on expectations of the 1997 GC decision as well as IFAD's priorities: (i) adequacy of supervision inputs and processes; (ii) how well CIs are meeting the MSRs; (iii) the degree to which IFAD specific aspects are being addressed through the supervision process; and (iv) the degree to which IFAD's strategic imperatives are being incorporated into the supervision process.

Supervision inputs. These were defined as frequency and length of supervision missions (SMs), size and composition of missions and mission leaders turnover. The evaluation found that:

  • most CIs carry out one mission per year, the World Bank and IFAD has closer to two missions per year;
  • IFAD direct supervision and the World Bank have the longest average length of SM (18 days) and AFESD the shortest (five days). AFESD and CAF mission durations did not allow adequate time for field visits.
  • CIs have had various levels of variations of SM leadership with UNOPS having the highest turnover. Evaluation findings indicate that the lack of continuity in supervision mission leadership is reducing the effectiveness of supervision and project performance.
  • the World Bank and IFAD direct supervision, on average, have the largest SMs followed by UNOPS then the other CIs. Supervision findings indicate that the larger SM team, allowing various disciplines to be included, led to overall better supervision output.

Weaknesses in definition of MSRs criteria. The evaluation found that the standards applied for fiduciary issues are not defined in sufficient detail in the MSRs. The issue of irregularity in project auditing is a case in point. The definitions of implementation support and human dimension in the MSRs are equally vague relative to IFAD's long and extensive experience in rural poverty reduction.

Supervision quality under MSRs criteria. Overall, the DR found the MSRs have been met with some variations among its components. For core supervision functions, CI performance was overall good with little variance among the CIs. For implementation support greater differences were found, with World Bank, IFAD direct supervision and UNOPS showing the best performance. All CIs and IFAD direct supervision found the human dimension more difficult to handle though IFAD and UNOPS have a certain edge. To a large extent this is due to the lack of clarity of the concept and various interpretations by various CIs. CI performance with respect to supervision administration was rather low. This covers supervision planning and strategy for the whole project. In sum, the DR concluded that the MSRs introduced in 1997 were generally met though at a lower level than had been anticipated given the expectations of the GC decision. The evaluation also found that the supervision process, its expected standards and emphasis were not sufficiently documented or described in the Appraisal Reports (ARs), the loan documents and the letters of appointment.

IFAD specific aspects and strategic imperatives. Supervision fared lower in addressing IFAD specific aspects. UNOPS and IFAD direct supervision performed relatively better, due to their closeness to the evolving IFAD priorities (UNOPS not being an IFI with a specific mandate). On the strategic imperatives, no clear pattern could be discerned. The generally poor ratings seem largely the result of unclear definition and a lack of institutionalised and systematic communication on these imperatives by IFAD to the CIs and their staff (neither the CIs nor IFAD were held accountable on this particular aspect by the evaluation analysis). The World Bank scored better in linking supervision with policy dialogue whereas UNOPS was best in emphasising impact achievement. This analysis also indicated that increasing the size of SMs, especially by including a sociologist and/or gender specialist, improved the prospect for better addressing the IFAD specific aspects. The time SMs spent in the field was a positive factor in addressing the specific aspects.

IFAD-specific aspects were understood differently among IFAD regions. For example participatory approaches were better appreciated and implemented in Asia and Latin America. Similarly, gender issues were not only understood differently in the different regions but mechanisms of support and guidance provided for them varied by CIs in the various regions

Relation between supervision quality and project performance. Quantitative analysis revealed an apparent weak influence of supervision (core as well as implementation support) on project performance. Regression of average project performance rating for each CI in each region (as per CPMs assessment) against the assessed supervision rating showed only a slight positive correlation between supervision ‘quality' and project performance.

The Focus of ARs and the loan agreements in outlining IFAD specific aspects and its implementation mechanisms was found to be an important factor in the attention eventually attached to these aspects during supervision. However, detailed description in ARs is not sufficient to ensure that SMs will highlight these issues, as this depends on the quality of the SM team.

The cooperation agreements and letters of appointment for the CIs are vaguely worded, do not refer clearly to the MSRs nor IFAD's specific requirements in supervision, hence do not provide incentives for good supervision. This is particularly significant for UNOPS and the World Bank in supervising IFAD-initiated projects. Many of these agreements have not been updated since the signature of the original version.

Perspectives from the Field

Overall Views from the Field. Project managers' views on the supervision process indicate that, overall, the process is working quite well and is highly appreciated with high ratings given to reporting, meeting core supervision requirements and assisting to achieve project goals. Project managers feel that supervision is a positive experience and helps them in project implementation. Project management assessment of supervision was systematically above average and higher than the DR estimate. They scored IFAD direct supervision best in terms of achieving the MSRs, as well as the IFAD-specific aspects. About three quarters of project management expectations from supervision are in the area of implementation support, not the core/mandatory aspect of supervision. The most frequent expectations relate to advice and guidance on achieving and assessing impact, targeting women and participatory monitoring.

Concerns of project Managers. There remained concerns by project managers in specific areas of supervision: they expressed the need for:

  • more participatory supervision relating to more involvement in the preparation of SM TORs and the selection of SM specialists and more intensive interaction with the target group and project staff during SM;
  • inclusion of more appropriate skills mix on the missions particularly to cover IFAD's specific aspects;
  • better, more frequent access to implementation support sources at local level;
  • more reliance on in-country or regional resources in providing implementation support and invest in their capacity enhancement;
  • greater support in developing their Monitoring and Evaluation (M&E) systems particularly for participatory monitoring and impact assessment;
  • flexibility in supervision frequency as per project needs with two supervision missions per year particularly in early project implementation and for projects covering a large geographic area.

Preference for IFAD direct supervision. While project managers assessment for the main service providers was not significantly different and was above average for all five CIs, IFAD direct supervision appears most appreciated by IFAD supported projects because of:

  • the direct involvement of the CPMs and thus the elimination of the ambiguity in the respective roles of IFAD and the CI;
  • more effective consultations with government and better influence on decision makers;
  • a generally faster response to project queries;
  • the larger teams and more frequent supervision missions used under IFAD direct supervision which appear to have given better services to the projects concerned.

Achieving MSRs and IFAD specific aspects. Project manager response indicate that MSRs are overall achieved by all main CIs. The perception of project managers with regard to the supervision of IFAD specific aspects is that SMs on the whole address these aspects satisfactorily, but participation by the target group and gender sensitive implementation are the least satisfactory. Evaluation field visits, in particular, indicate that the expectations of project management regarding IFAD-specific areas in general are driven by a realism with which they have to address these subject matters every day and a realization of the extent of difficulties involved.

The field visits also suggested that the role and responsibilities of the field partners (governments and implementing agencies) in project supervision should be outlined more specifically in the ARs and other loan documentation. In particular project managers felt that SMs could be used for helping to resolve pending problems with higher level officials. IFAD's advantage in this area was the clearest.

IFAD Staff views

CPM assessment of CI performance. Within PMD's internal project portfolio review, IFAD's CPMs are requested to rate the CI's supervision performance for each project. No rating is provided for direct supervision. The CPMs rated the performance of the two CIs with the largest portfolios, UNOPS and the World Bank, as well as BOAD, most favourably followed by AFESD and CAF. In the view of the CPMs there was, on average, a reasonable supervision performance for the main CIs though many areas were pointed out for improvements. For the weaker ones IFAD stepped in to overcome identified weaknesses through the use of IFAD implementation support instrument.

Relations with CIs. All CPMs view supervision as a process to provide feedback on whether a project is in line with its objectives, to recommend corrective actions and to ensure adherence to the fiduciary aspects. But significant ambiguities remain among CPMs about the extent of their involvement in CI supervised projects. This highlights the "grey" areas, which exist in the respective responsibilities of the CIs and the CPMs. A large majority of IFAD staff believes that: (i) CI responsibilities should be more clearly defined in legal documents; (ii) a more specific and detailed letter of appointment tailored to each project is needed; (iii) clear performance indicators for CIs should be developed and the CIs held accountable to them; (iv) IFAD should develop and provide detailed guidelines for supervision of IFAD specific areas of concern and should enhance the capacity of CIs through training in these areas; and (v) the CIs need guidance in supporting the project to establish a better system to monitor and assess project impact.

Minimum supervision requirements. CPMs recognized that CIs on the whole meet the MSRs. However, a high majority of staff (75%) do not find the MSRs particularly useful and attest to their inadequacy regarding IFAD specific aspects. Many CPMs were not completely clear over the exact content of the MSRs.

Current supervision modalities. These were not found satisfactory for IFAD staff. A good majority (71%) believes that IFAD should not continue with the modality prevailing currently nor with the IFIs type supervision being used. Many argued that IFAD needs to develop different modalities based on building local capacities for implementation support and a form of IFAD's field presence. They stressed that supervision should be undertaken in close partnership with all stakeholders and use locally based resources to reflect the changing nature of IFAD supported projects (see also paragraph 30). CPMs believe that IFAD has become too dependant on one cooperating institution (UNOPS), and that the Fund should promote more diversification and competition amongst the CIs to enhance the quality of service. A large majority (71%) would like to see the projects supervised by UNOPS decrease. Some argued that as most CIs have no presence on the ground nor do they have real ownership for IFAD initiated projects, they cannot provide the desired supervision quality. Several experienced staff members observed that two (full) SMs per year gave a much better result, particularly during the first 2-3 years of a project.

Supervision quality. A strong interest (82% of staff interviewed) was expressed in strengthening PMD's internal portfolio review mechanism to improve supervision quality. ARs and Project Implementation Manuals (PIMs) need to address issues essential for good supervision more explicitly to help CIs carry out their task, facilitate monitoring and promote accountability. CPMs believe IFAD specific aspects are generally not well covered in supervision because these issues are not properly covered in the legal agreements between IFAD and the borrowing governments, and between IFAD and the CIs.

View of IFAD Staff regarding future supervision modalities. Two thirds (65%) of IFAD staff believe that direct supervision is the preferred option. The same figure for the group of CPMs that currently directly supervise the 15 pilot projects was as high as 80%. About one third of the total staff interviewed rated direct supervision as their least preferred option. This reflects different views as to what are, and should be, the CPM's priority work areas and their involvement in the supervision work of the CIs. CPMs not involved in direct supervision were generally less enthusiastic about direct supervision than those who are currently doing it, suggesting that the experience with direct supervision for the responsible CPM has had significant rewards. The CPMs undertaking direct supervision were very conscious of the additional workload arising from direct supervision as well as the difficulties encountered in mobilising the required resources, yet most of them are professionally satisfied and stimulated by direct supervision and believed in its worth for achieving better quality supervision and eventually better results on the ground.

IFAD's culture and the role of supervision. Many CPMs emphasized that IFAD still needs to shift its culture from design to implementation and impact achievement. To achieve this shift resources allocated to implementation phase should increase and changes should also be effected in the supervision modalities which would reflect the new trends in the way IFAD undertakes its business. Reference was made to the shift in IFAD projects from input/output orientation to process orientation and community-driven approaches, the emphasis on building local level institutional capacities, undertaking policy dialogue, enhancing local level partnership and empowering the rural poor. The quality of implementation support, embodied in supervision, should be consistent with this new orientation. IFAD should be more open to other more innovative options for supervision, and link it with the current IFAD efforts to develop its field presence.

Cost of supervision

The estimated cost of supervision is based on the work of IFAD's Office of Internal Audit for the two years 2001 and 2002. Cost of supervision of IFAD supported projects include: (i) fees paid directly to CIs; (ii) the IFAD staff costs incurred to oversee support and implement supervision by CIs and by IFAD; and (iii) the cost of implementation support to the supervision process through IFAD follow up budget and grant funds. The cost of IFAD staff time, in terms of salary, pension etc., has been imputed using estimated time inputs from CPMs and other staff. For 2001 and 2002 the average supervision cost ranged from USD 22 874 to USD 89 873 for directly supervised projects with an average of about USD 60 000 for CIs and IFAD. The average cost of the World Bank and of UNOPS for IFAD initiated projects was 74 254 and 68 682 USD respectively.

The analysis of supervision effectiveness, and supervision costs indicates that, on the whole the best performing supervisors (CIs or IFAD) are those with the higher cost. This is an approximate indicator that IFAD is getting proportional benefits from its resource allocation to supervision. There is, however, still quite a scope for enhancing supervision quality.

UNOPS, compared to the World Bank in particular, provides a satisfactory standard of full supervision services considering the lower level of fees that IFAD makes available for this CI. The extra costs of World Bank supervision on IFAD initiated projects are only partially offset by improved mandatory supervision services. Some of the regional co-financing CIs including AFESD (with below average supervision fees) perform well on the core supervision functions but need additional implementation support by IFAD to achieve an overall acceptable standard. CPMs have been active and imaginative in obtaining funds from grant resources and supplementary trust funds to provide additional implementation support to these institutions (see paragraph 47).

Cooperating institutions perspectives

Both UNOPS and World Bank staff highlighted the importance of informal relationships between CPMs and the supervision staff relative to (the generally irregular) formal interaction between the organizations. Both institutions suggested a desire for greater clarity in spelling out the respective responsibilities. In particular, UNOPS stressed the need to clarify IFAD expectations and welcomed the provision of more detailed guidelines and training on IFAD specific requirements. UNOPS maintains that its fees are set at a relatively low level in relation to IFAD's requests. This and the high work loads of its staff and their frequent field travel led some UNOPS staff to view IFAD as a "high maintenance client". The majority of UNOPS staff expressed the view that the relationship is increasingly dealt with as "client service provider" in the strictest sense not as a partnership for poverty alleviation.

UNOPS Portfolio Management Officers (PMOs) saw supervision as ‘facilitating an evolving process' to deliver successful projects4. Implementation support was seen as an important task but views varied on how well this could be achieved by UNOPS within the limited time and resources available for SMs. It was observed that supervision should identify where implementation support is needed and IFAD should mobilize resources on an ongoing basis to provide the inputs required. This is particularly relevant for building institutional capacity as it is a long term process. Supervision processes5 in UNOPS are not fully documented (though guidelines on loan administration exist) and new staff rely on more experienced staff for support. In offices with a low rate of staff turnover this is not a major problem, but other offices have, and are still, experiencing high staff turnover6. This places more pressure on the more experienced UNOPS/PMOs. A higher turnover of staff, resulting partly from the high work load and constant travelling, as well as lack of appropriate incentive framework, have reduced the quality of supervision. The major concern expressed, aside from the limited cost allowance, is that IFAD does not have a systematic process to communicate its priorities on IFAD specific aspects and IFAD strategic imperatives to the CIs.

The World Bank uses a similar but more comprehensive supervision process than UNOPS. The supervision guidelines are extensive and are supported by specific financial management, procurement and safe guard procedures and documentation. The World Bank has clear accountability and quality assurance management processes in place for supervision. World Bank staff are very clear that supervision processes on IFAD projects have to be consistent with World Bank projects. Supervision inputs must cover three main areas: (i) fiduciary issues; (ii) ten safeguard requirements; and, (iii) implementation support. The first two areas were the highest priority given available resources.

As IFAD developed its capacity and expertise in certain areas, pressure has been applied by some CPMs on the World Bank to address IFAD specific aspects as part of supervision implementation support. Often the relatively small IFAD funded component(s) is more complex, geographically remote and with emphasis on community based activities and local institutions building. This requires additional time and skills inputs that the World Bank argues is not paid directly for. The World Bank staff see advantages in working with IFAD on projects targeted at rural poor, but they argue that IFAD will need to provide additional resources to ensure IFAD's specific interests are covered in projects that are initiated and supervised by World Bank. World Bank staff also flagged an apparent lack of communications and feedback processes, including systematic apprising about the changing IFAD priorities. This aspect assumes greater importance with the IFI CIs if IFAD is to succeed in influencing these institutions toward the goals of rural poverty reduction.

UNOPS has and can adjust its approaches to meet IFAD requirements but the World Bank staff are somewhat more constrained by the requirements to meet World Bank mandatory supervision requirements and safeguards. Although UNOPS has staff dedicated to IFAD work these staff are not included in the other parts of the programme cycle, e.g. design efforts, IFAD's internal portfolio review, and IFAD fora concerned with developing future vision and strategy.

Implementation of the five-year Plan of Action

Implementation of the Action Plan has been substantial but not complete as summarized below:

  1. MSRs have not been incorporated into formal and legal documents with the CIs. For core supervision functions, MSRs are fulfilled, to varying degrees, by most CIs. However, IFAD specific concerns are not being adequately addressed by the MSRs and some are dealt with in a vague manner. CIs performance relating to these concerns is lower. There is no clear system to hold CIs accountable for MSRs.

  2. Coordination Procedures between IFAD and CIs have not been systematically addressed with informal cooperation between CPMs and their CI counterparts still the basis for coordination and effectiveness.

  3. Portfolio Management. Good progress has been made and a more efficient portfolio management system is in place. PSRs are prepared as part of the yearly regional portfolio reviews with a consolidated Project Portfolio Review (PPR) presented annually to the April Executive Board (EB). The "realism" of estimates of project and CI performance could be improved through an agreement on clearer more detailed indicators to facilitate monitoring. A separate accounting system to track the actual cost of direct supervision has been developed, but is not used systematically and effectively.

  4. Strengthening Learning Loop. The main learning linkage continues to be an informal and loose process. Consultants working directly with CPMs undertake implementation follow up and project design/review work. No formal mechanism was put in place to capture learning from CI and IFAD supervision experiences. Given staff workload it is not evident how effective is the informal process of exchanging knowledge.

  5. The Direct Supervision Pilot Programme has been fully implemented using modalities ranging from the UNOPS model to large consultant teams mobilized twice per year. Average annual direct supervision costs are higher than those for other supervision modalities. Systems to monitor and assess improvement in project performance through direct supervision have not been put in place nor have mechanisms to ensure that the value added by direct supervision experience is shared and disseminated in-house and with CIs.

Main conclusions

Supervision in IFAD's Project Cycle. Institutional attention to supervision is a reflection of IFAD's strategic emphasis and priorities with respect to various stages of the project cycle. The dominance of design (and pre implementation) cost in IFAD's project cycle is a manifestation of an underlying institutional priorities to project approval stage. IFAD efforts to shift its emphasis to implementation stage and impact achievement have to be enhanced. One way of doing this is through strengthening the supervision processes.

Supervision Effectiveness. A large cross regions and cross CIs assessment by the evaluation team attested to a reasonable level of supervision overall though with variations between the various tasks making up project supervision and between the CIs. Consistently CIs have performed better on fiduciary aspects whereas implementation support, particularly for IFAD's specific requirements (and strategic imperatives), lagged behind. A similar view as to the effectiveness of supervision was found among IFAD operational staff. They also expressed strong views regarding the need to modify existing supervision modalities.

The supervised clients (project managers), on the other hand, expressed remarkable satisfaction with supervision of their projects. They ranked the services they receive from supervision at a consistently higher level (between fully and highly satisfactory), but services received from IFAD's direct supervision ranked best. However, they strongly expressed the need for more frequent and better access to local level implementation support, more participatory supervision and for changing supervision frequency with the nature of the project and the implementation stage.

On the whole, UNOPS, the recently started IFAD's direct supervision and the World Bank showed a stronger supervision performance than the regional, smaller CIs. IFAD direct supervision followed by UNOPS ranked systematically better in implementation support. CAF and AFESD were found relatively less effective than the other CIs. Smaller regional IFI/CIs notably CAF and BOAD are receiving closer attention by the regional divisions in terms of building their capacity and there are recent emerging signs that these efforts may pay off. All perspectives indicated that while a number of areas require attention from IFAD to improve supervision, the IFI model in supervision, currently prevailing in IFAD, is not likely to deliver the quality of supervision needed under IFAD changing priorities and nature of field operations. An innovative break through is needed to move supervision to a higher plateau of performance.

Implementation of the five year Action Plan. While attention to supervision in IFAD has no doubt improved after 1997, the changes called for by the Five-Year Plan of Action have only partly been achieved. Progress was concentrated in the PPR/PSR processes and, most important, in the introduction of direct supervision. The MSRs were developed, and are currently met, but the standards they embody are below what an institution of IFAD's mandate and experience in poverty reduction should be expecting. There remains an important agenda to be pursued. MSRs – or any equivalent form of common guidance for supervision of IFAD supported projects – must be made the working instrument that ensures consistency between IFAD specific interests and what is pursued on the ground. Some other unfinished agenda also remains from the 1997 Plan of Action in the areas of coordination between IFAD and the CIs and reinforcing the learning loop between design and implementation.

Project portfolio management. Very good progress has been made in portfolio management. This has enabled IFAD's management and its EB to focus attention on critical portfolio concerns and to take necessary actions. It also has helped to bring an institutional perspective to what otherwise CPMs might have looked at from single project and/or country angles. Nevertheless there remains some short comings about the PPR process as a vehicle for improving the quality of supervision. Renewed and more intensive efforts are needed in these areas to enhance supervision quality and overall accountability in delivering supervision services. IFAD's Controller's Office remains concerned about the handling of audits.

Concentration of supervision and the IFIs model. Concentration of project supervision into fewer CIs has continued in recent years. UNOPS is currently responsible for supervising about 60% of ongoing projects and is the only non IFI CI. But the IFIs model of supervision7, also used by UNOPS, has gone largely unchanged. Because of the changing way of conceptualizing and designing projects in IFAD, the IFI model in supervision may not be the most effective modality to enhance project performance. The evaluation found some interesting pioneering work (notably in PL) done through the use of regional technical assistance grant (TAG) funded programmes for capacity building of local and regional institutions to provide significant inputs into the supervision process in support of project implementation. Some CPMs are also pioneering local level partnerships a proxy for field presence to promote the right type of project implementation support. There is an opportunity to extend and develop further innovative supervision modalities that are especially suited to the needs of IFAD assisted projects in different regions.

The role of implementation support by IFAD. Implementation support, to supplement CI supervision, has existed in IFAD from the very beginning and is firmly established as an input in the supervision process. Despite its importance and the large amount of resources it absorbs (more than one quarter of supervision costs), IFAD has not developed a clear operational policy and priorities on which to anchor this concept. The evaluation did not find any direct correlation between implementation support and CI performance nor with project performance. Most importantly the use of implementation support is not linked with a clear policy of local and national capacity building. This may not be consistent with sustainability requirements.

IFAD supported projects expressed the desire for more frequent interaction with locally-based implementation advisers, particularly for process oriented community driven projects (currently the majority of IFAD projects). There is a need for a re-examination of the concept of implementation support in IFAD (and for the CIs), re-position it within a medium-term strategy for strengthening local capacity towards the support of community-driven projects and adopt a more systematic approach in its execution.

Cost of supervision. The results of cost analysis for all CIs over two years (2001 and 2002) demonstrated that the best performing CIs are those with the higher supervision cost. As various CPMs observed "IFAD gets from CIs what it pays for". While this is an approximate indication that benefits are in line with cost incurred, no conclusive statements can yet be made (particularly regarding IFAD direct supervision) as this would require, inter alia, analysis over a longer time horizon. Difficulties in data availability and compilation prevented the undertaking of such exercise.

Direct supervision. The evaluation preliminary findings noted many positive features in the performance of direct supervision. This was confirmed by the DR and the project managers survey. The evaluation also noted some tradeoffs, in particular in terms of the high unit costs of direct supervision. However, the evaluation could not draw any firm conclusions regarding direct supervision as the average implementation period of the directly supervised projects and the disbursement rates were much below other projects reviewed.

Recommendations

The evaluation recommends that actions are taken to shift the boundaries of the supervision modalities for IFAD towards a new model that better reflects the nature and needs of IFAD supported projects. It is recommended that actions are taken at two levels simultaneously: (i) Policy Level and (ii) Operational Level.

Policy level

Recommendation 1: develop a new oolicy and regional programmes for supervision in IFAD. The need to introduce improved supervision modalities reflecting the changing priorities and nature of the Fund's supported projects has become evident through the evaluation.

A number of pioneering initiatives are now under way in IFAD at regions and/or CPMs' level. The evaluation recommends that IFAD builds on these initiatives and develop a new medium term IFAD policy for supervision. The policy should combine two crucial dimensions of IFAD's emerging priorities: building local/regional level capacities and promoting innovative partnership with stakeholders at local level for pro-poor implementation support. The policy should specify the principles involved in designing and implementing regional programmes that extend and develop improved supervision modalities taking into consideration the variance in institutional capacities in different regions, and linking explicitly with the IFAD ongoing efforts regarding its field presence. An implementation plan with measurable indicators would have to be associated with the development of such a policy, including at the level of the different regional divisions programmes. While the results of the policy are gradually materializing, IFAD should continue to address areas needing strengthening under the current modalities.

Operational level

Recommendation 2: tevise and update IFAD's minimum dupervision requirements for its CIs and hold CIs firmly accountable to the MSRs

The number one recommendation of the 1997 Five-Year Plan of Action, is in need of renewal and reinforcement with respect to (i) its transformation into a dynamic system that reassures IFAD that emerging new concerns and priorities are reflected in supervision standards; (ii) core supervision and fiduciary aspects and IFAD specific requirements to be spelled out clearly and their standards specified; (iii) cooperation agreements and letters of appointments with CIs to rigorously reflect the new MSRs (or new standards) with special emphasis on IFAD specific aspects and strategic imperatives; (iv) adherence of CIs to these principles to be closely monitored (see also recommendation 5).

Recommendation 3: build CI capacity in the supervision of IFAD's specific aspects and provide appropriate tools

It is IFAD's responsibility to strengthen the capacity of the supervisors of its supported projects (CIs and CPMs). Written guiding frameworks, periodic joint CI/IFAD training sessions and sharing of learning and best practices between CIs and between IFAD and CIs can lead to great improvement. Training of project staff themselves in these aspects is an important dimension of capacity building and should be considered a priority area for implementation support by IFAD.

Recommendation 4: assess and revise IFAD's implementation support patterns and practices

It is critical to realign the use of all IFAD instruments used in supervision towards capacity building and local level partnership for better project implementation. It is recommended that PMD undertakes an assessment of implementation support practices in IFAD and their effect on project performance and to establish on these basis clear principles and criteria for the use of this instrument and monitor it closely to ensure that resources are allocated in areas of maximum returns for project performance.

Recommendation 5: Improve supervision quality assurance in IFAD

It is recommended to strengthen the PRR process by adding a strong institutional focus on improving the quality of supervision and facilitating monitoring of CI performance. This would increase the confidence in the reliability of the PPR system as a portfolio monitoring mechanism and enable IFAD to give a better informed feedback to CIs. Under such a focus IFAD would introduce a vehicle to provide incentives for good supervision, enhance supervision accountability for meeting well defined standards and improve results on the ground. This should apply to CI as well IFAD direct supervision. The latter is particularly important as no system exists at the moment within PSR to assess the performance of supervision in directly supervised projects. CIs would have to be incorporated in such quality assurance process to ensure ownership of recommendations. Reviewing other IFIs practice in this respect would help identify good practices.

Recommendation 6 strengthen the learning loop from supervision

Much more needs to be done for a systematic strengthening of the learning loop between design and implementation across and within projects. A more systematized learning process has to be developed, involving the principal actors involved with supervision on the one hand and with project formulation/appraisal and the project portfolio review mechanism on the other. It seems especially appropriate to draw on UNOPS PMOs who have specialized experience with the implementation of IFAD supported project as well as supervision consultants.

Recommendation 7 improve coordination between IFAD and CIs and strengthen the partnership dimension

This continues to be a matter of concern for IFAD and the CIs, and to some extent project managers. Institutionally the agenda for improvements in this domain should eliminate areas of overlap, clarify responsibilities, improve communication, establish a more consistent reporting system across all CIs, and expand the view about CIs from mere service providers to partners for rural poverty reduction with joint responsibility for outcome and stronger ownership. Appropriate incentives for supervisors are of essence to develop such ownership.

Recommendation 8 exercise realism in setting up fees for the CIs and base it on project needs

If the scope of supervision is to be consistently maintained or even enhanced, including the practice of an average of two supervision missions at least at the initial phase of project implementation, of a longer stay in the project area, and adequate interaction with beneficiaries, realism in the setting of fees is essential. Implementation support, a subject recommended for review under recommendation 4, is a complementary resource in strengthening supervision services performed by CIs and would have to be considered in the context of setting realistic CI fees.


1/ The Corporate Level Evaluation Team comprised: Ms Mona Bishay, Deputy Director Office of Evaluation (Team Leader); Mr Ian Teese (Senior Evaluation Consultant/Agricultural Economist); Ms Maliha Hussein (Evaluation Consultant/Rural Sociologist); Ms Sandra Romboli (IFAD Consultant); and Mr Hans Wyss (Special Resource Person). The team was also supported by three consultant reviewers: Jane Blaxland, Hamdy Eisa and Octavio Damiani for the French, Arabic and Spanish reports respectively.

2/ The Agreement Establishing IFAD states that the Fund shall delegate this role to cooperating institutions: "The Fund shall entrust the administration of loans, for the purposes of the disbursement of the proceeds of the loan and the supervision of the implementation of the project or programme concerned, to competent international institutions". (Article 7, Section 2(g) of the Agreement Establishing IFAD).

3/ 1994 version, paragraph 43.

4/ The role of the CIs were modified in the new General Conditions produced in 1998 to add the facilitation role of supervision. This had apparently not been clearly communicated to some PMOs.

5/ In addition to loan and procurement administration, new PMOs expressed a need for further training in high level negotiation and change management skills.

6/ By mid 2003, UNOPS Kuala Lumpur office will possibly have three out of four IFAD PMOs with less than 12 months supervision experience.

7/ With supervision done through external missions once or twice a year concentrating mainly on fiduciary aspects.

 

LANGUAGES: English