2017 Annual Report on Results And Impact of IFAD operations
Every year, since 2003, the Independent Office of Evaluation of IFAD produces the Annual Report on Results and Impact of IFAD Operations (ARRI). This 15th edition of the ARRI highlights results, trends and issues based on independent evaluations conducted in 2016.
The ARRI finds that, overall, performance of IFAD operations shows improvement from 2009. Currently, 80 per cent or more projects assessed against the criteria of gender equality and women’s empowerment, innovation and scaling up, relevance and rural poverty impact were rated moderately satisfactory or better. Longstanding challenges are found in efficiency and sustainability of benefits.
Every year, the ARRI is focused around a learning theme so that evaluation can deepen the analysis on specific topics of interest and importance to IFAD operations. This year's learning theme focuses on financial management and fiduciary responsibilities in IFAD-supported operations. The report contains a chapter dedicated to this learning theme.
The 2017 ARRI recommends that IFAD rise above the current plateau in performance and achieve greater impact on the ground. IFAD needs to build on its strengths by raising its performance to satisfactory or better and address longstanding bottlenecks to maximize sustainable results. This requires a change in IFAD’s modus operandi that includes a new approach to programme design, which will enable IFAD operations to efficiently deliver more relevant and sustainable results.
2016 Annual Report on Results And Impact of IFAD operations
The 2016 Annual Report on Results and Impact of IFAD Operations (ARRI) draws from 40 independent, impartial and rigorous evaluations carried out in 2015. This year’s report assesses results against the Ninth Replenishment of IFAD’s Resources (IFAD9). It also identifies opportunities and challenges in light of the priorities for the Tenth Replenishment of IFAD’s Resources (IFAD10) and in the broader context of the Sustainable Development Goals and the 2030 Agenda for Sustainable Development.
Areas of strength
The broad picture of performance in the IFAD9 period emerging from the 2016 ARRI is positive. The 2015 evaluations found that overall 80 per cent of the projects are rated moderately satisfactory or better for most of the criteria in 2013-2015.
IFAD has made a positive contribution to rural poverty reduction, for which the percentage of moderately satisfactory or better projects increased from 87 per cent for operations completed in 2011-2013 to 92.3 per cent in 2012-2014.
This is mainly a result of the Fund’s attention to improving income and assets, supporting human and social capital empowerment, enhancing agricultural productivity and food security and boosting technological innovation and scaling up.
In particular, IFAD operations completed in 2012-2014 achieved the highest impact on household income and assets, as compared to other impact domains, with 92.3 per cent of the projects rated moderately satisfactory or better. The contribution of IFAD’s operations to food security and agricultural productivity, which is the keystone of the Fund’s mandate, has been substantive in terms of both improving the availability of and access to food, and enhancing agricultural productivity. Eighty-six per cent of projects are assessed as moderately satisfactory or better in 2012-2014, which is the highest percentage since 2007.
Moving beyond the project level, the assessment of non-lending activities (e.g. knowledge management, partnership-building and policy dialogue) in the 2015 Country Strategy and Programme Evaluations show an improvement of performance in knowledge management from 67 per cent moderately satisfactory country programmes since 2010-2012 to 78 per cent in 2013-2015. This confirms the increased attention of the Fund to sharing experiences and lessons learned.
Areas for improvement
Notwithstanding the mentioned positive findings, the 2016 ARRI highlights that IFAD’s performance is largely only moderately satisfactory and identifies systemic issues at both project and country levels that merit further attention moving forward to achieve satisfactory and highly satisfactory results:
Weak targeting strategies, which are often not flexible enough to adapt to changing contexts, and poverty analyses that do not sufficiently capture the differences among groups of rural poor, limit the reach of project activities to all targeted beneficiaries, in particular the poorest.
Poor nutrition mainstreaming in IFAD’s portfolio and the lack of evidence of the potential contributions of improved agricultural productivity to food security and nutrition hinder increased impact in rural poverty reduction.
Inadequate management of fiduciary responsibilities which constrains efficiency of operations and better performance of governments.
Non-lending activities have historically been the weakest area of IFAD support with 64 per cent of the country programmes rated only moderately satisfactory since 2006. The 2015 Country Programme Evaluations found that while performance in knowledge management is improving, policy dialogue and partnership-building at country level show a decline in performance. Partnerships remain bound to the project level, thus limiting dialogue, dissemination of results and scaling up of successful approaches.
Each year since 2007, the ARRI has focused on a learning theme. The topics for the learning themes are agreed with the IFAD Executive Board, with the aim of deepening the analysis on selected issues that merit additional reflection and debate in order to enhance the performance of IFAD operations. The learning theme presented in this year’s ARRI is knowledge management, with an emphasis on how operations can learn to improve performance.The report highlighted that notwithstanding the improvements in performance in knowledge management, the limited allocation of resources, time and incentives restrain the systematization of knowledge management in IFAD and the establishment of stronger learning loops among the project, country and institutional levels. By and large, the knowledge acquired during project implementation is often not systematically captured and widely shared.
2015 Annual Report on Results and Impact of IFAD operations
The 2015 Annual Report on Results and Impact of IFAD Operations (ARRI) shows that, overall, IFAD operations are satisfactory and making good contribution to inclusive rural transformation. Projects and programmes had positive results in reducing rural poverty and improving the living conditions of rural poor people, in particular in terms of increased income and assets, better human and social capital and empowerment, and gender equality and women’s empowerment.
Nevertheless, there are areas that can be further improved, such as operational efficiency, monitoring and evaluation and non-lending activities (e.g. policy dialogue, knowledge management and partnership-building).
This year’s learning theme is the sustainability of benefits of IFAD’s operations, which was found to be one of the weakest-performing areas.
2014 Annual Report on Results and Impact of IFAD operations
The Annual Report on Results and Impact of IFAD Operations (ARRI) is the flagship report of the Independent Office of Evaluation of IFAD (IOE). The ARRI was first published in 2003, and provides an independent assessment of the performance of IFAD operations and identifies issues that need to be addressed to further strengthen results on the ground. This year's ARRI draws on a robust sample of ratings from 224 project evaluations carried out by IOE since 2002 - including 35 project evaluations undertaken in 2013. One of these is the first impact evaluation by IOE, which was carried out in Sri Lanka.
IFAD is one of the very few development organizations that produces an annual report like the ARRI, reflecting its commitment to promoting accountability, learning and transparency in reporting on results and impact.
Performance and trends
This year's ARRI shows that IFAD's performance as a partner is the best since the ARRI was first produced in 2003. The numerous changes and reforms to IFAD's operating model – in particular direct supervision and implementation support, the establishment of IFAD country offices and better portfolio management – are contributing to better performance.
Other areas of good performance include the relevance of operations and rural poverty impact, including promotion of gender equality and women's empowerment, and innovation and scaling up. In fact, rural poverty impact is moderately satisfactory or better in 90 per cent of the projects that were evaluated during 2012-2014.
This year's ARRI also analysed independent evaluation ratings grouped by IFAD replenishment periods, starting with the Fifth Replenishment (2001- 2003). Generally speaking, with the exception of efficiency, IFAD operations have performed best during the Ninth Replenishment period (2013-2015).
However, there is scope for strengthening the performance of IFAD-supported operations in some critical areas in order to achieve greater development effectiveness. Firstly, efficiency remains the weakest area, with 42 per cent of operations rated as moderately unsatisfactory or worse. Secondly, further efforts will be needed to support governments in enhancing their capacity and performance, since they are ultimately responsible for executing IFAD-funded projects and programmes. Thirdly, although there have been recent improvements, sustainability of benefits is still weak and will require greater attention.
There is scope to further enhance performance in non-lending activities, i.e. knowledge management, policy dialogue and partnerships development, which are also essential for scaling up impact.
As in the past, this year's ARRI benchmarks the performance of IFAD operations externally and internally. Externally, it compares the performance of IFAD operations with that of the agriculture sector operations of selected multilateral development banks. Internally, it compares operational performance across the five geographic regions and against the agreed targets in the corporate results measurement framework.
The ARRI finds that the performance of IFAD operations is on par with or better than the performance of the Asian Development Bank (ADB), African Development Bank (AfDB) and World Bank agriculture operations. When interpreting these findings, the different nature and size of the organizations must be kept in mind – especially the fact that IFAD operations are normally implemented in more challenging contexts, such as remote rural areas, and give special attention to participatory approaches, gender mainstreaming, targeting and grass-roots institution-building.
Internally, the analysis revealed that the performance of IFAD operations is strongest in Asia and the Pacific Region, followed by East and Southern Africa. IFAD's operations performance is weakest in West and Central Africa, even though an important contributing factor is the relatively weaker policy and institutional context, as well as the large proportion of fragile and conflict-affected states in the region. Finally, the ARRI finds gaps in current performance based on independent evaluation ratings and IFAD's Ninth Replenishment targets for operations, to be accomplished by the end of 2015. While the gap is relatively small in some areas (relevance and impact), it is quite large in others (efficiency, sustainability, partnerships and government performance).
- Completion reviews of country strategic opportunities programmes (COSOPs). The ARRI found that the preparation of COSOPs is underfunded and would be enhanced if new COSOPs were informed by completion reviews by IFAD Management. The ARRI recommends that these reviews be conducted systematically in the future.
- Budgets for COSOPs and projects.
Consider a more differentiated approach towards budget allocations ( e.g depending on the country context) for COSOP development and management, project design, supervision and implementation support, and non-lending activities.
- IFAD's organizational decentralization.
For better on-the-ground results, opportunities for greater decentralization of IFAD and further out-posting of country programme managers, should be explored by building on ongoing efforts. Consideration for establishing other regional/subregional offices, based on the experience accumulated thus far should also be considered.
- Use of independent evaluation ratings.
The Independent evaluation ratings of IOE, where available, should be used in applied in the Report on IFAD's Development Effectiveness ( RIDE) against the criteria/indicators in the IFAD Results Measurement Framework and in other ad hoc reports relating to results. This is consistent with practice of other selected multilateral development organizations and will ensure consistency in the reporting of the ARRI and the RIDE in relation to country programme and project performance.
2014 ARRI learning theme: Project management
Project management is the learning theme for the 2014 ARRI. Overall, evaluations by IOE and others find that a strong project management team is a key determinant of successful project outcomes.
The ARRI identifies a number of features that enhance the performance of project management: the commitment and ownership of project management staff; regular supervision implementation support by IFAD; participation in annual country-level portfolio review and regional implementation workshops; clear guidelines and procedures for procurement and financial management; and participation in training on issues of priority to IFAD.
There are also several challenges affecting the performance of project management: delays in the appointment of, and frequent changes in, project staff; low priority given to monitoring and evaluation activities; challenges in applying the IFAD Results and Impact Management System; limited knowledge of IFAD policies and procedures; and interference by the designated executing agency.
Good project management is a key driver towards ensuring sustainability of benefits. For instance, evaluations found that sustainability prospects were higher in those cases where project management proactively prepared ‘exit strategies' together with IFAD and government early on the project life cycle.
2013 Annual Report on Results and Impact of IFAD Operations (ARRI)
This is the eleventh edition of the Annual Report on Results and Impact of IFAD Operations (ARRI), prepared by the Independent Office of Evaluation of IFAD (IOE). It consolidates and summarizes the results and impact of IFAD-funded operations on the basis of evaluations conducted during 2012 and in previous years.
The Fund is one of the very few multilateral and bilateral development organizations that produces an annual report similar to the ARRI. Its production is a reflection of IFAD's overall commitment to promoting accountability, learning and transparency in reporting on performance. While the primary audience for the ARRI is IFAD Management, staff and consultants, and the Fund's Evaluation Committee and Executive Board, the report is also of interest to recipient countries and the wider development community.
The objectives of the ARRI are to: (i) present a synthesis of the performance of IFAD-supported operations based on a common methodology for evaluation; (ii) highlight key learning issues and development challenges that IFAD and recipient countries need to address to enhance the development effectiveness of these operations.
IOE is able to produce the ARRI because all its evaluations follow a consistent methodology, captured in the Evaluation Manual, which builds on international good practice. In preparing the 2013 ARRI, IOE took serious note of feedback on last year's edition by IFAD Management and the Evaluation Committee and Executive Board, and, accordingly, adjusted the way independent evaluation data have been analysed and reported on in this edition.
Two changes have been made. First, the 2013 ARRI has analysed and reported evaluation results based on the year in which projects were completed, rather than the year in which evaluations were undertaken (the past practice). This is consistent with the practice in other international financial institutions and has ensured that the results for each time period are representative of a similar cohort of projects.
Second, a new data series has been introduced based only on evaluation data from project completion report validations (PCRVs) and project performance assessments (PPAs). This avoids a possible problem associated with combining results across a diverse mix of evaluation types. The one limitation of the new data series is that IOE only began conducting PCRVs and PPAs in 2010. Thus it is difficult to conduct trend analysis based on these data at this stage, a limitation that will be overcome once data from other years become available. Consequently, the series based on data from all types of evaluations will continue to be presented for the time being in order to report on trends.
The relevance of IFAD-supported projects remains generally high, illustrating the importance of the Fund as a global organization promoting sustainable small-scale agricultural development. The rural poverty impact of IFAD operations has shown improvement since 2005-2007. An upward trend is also visible in project performance and overall project achievement in projects that closed since 2009-2011.
There are three further areas of good performance that deserve to be highlighted. First, IFAD's own performance as a partner over 2011-2013 is the best since the ARRI was first issued in 2003. This is noteworthy, and the coherent Change and Reform Agenda introduced in 2009 is likely to have also been a key ingredient in improved performance in this area. Direct supervision and the consolidation of existing, and establishment of new, IFAD country offices are two fundamental changes to IFAD's operating model that are contributing to better results.
Second, IFAD operations score high in promoting innovative approaches and in scaling up, which are fundamental to achieving a wider impact on rural poverty. In fact, for a relatively small organization such as IFAD (compared with the World Bank or other regional development banks), the capacity to develop and test innovative solutions in agriculture and rural development is critical, because, if successful, these can eventually be scaled up by governments, donors and other partners. Third, IFAD operations are very good at promoting gender equality and women's empowerment, an area in which the Fund is developing a comparative advantage, a track record and specialization. This, too, is fundamental, given that a significant number of poor rural women in developing countries work in agriculture and derive their livelihoods also from related activities.
The performance of IFAD operations is generally on a par with the agriculture-sector operations of the World Bank globally, similar to that of the African Development Bank in Africa and better than the Asian Development Bank in Asia and the Pacific. This is reassuring, given that the contexts in which IFAD works are often more challenging (i.e. remote and marginalized areas with a focus on poor rural people) and the nature of its operations is generally more demanding (i.e. with a focus on participatory approaches, targeting and empowerment).
There are areas, however, that deserve added attention moving forward. Notwithstanding the positive picture in general, a large number of projects continue to manifest moderately satisfactory performance and hardly any are highly satisfactory in the evaluation criteria assessed. Thus, there are opportunities for further improvement overall.
Moreover, two areas flagged in the past remain problematic: efficiency of operations and sustainability of benefits. These are the two weakest performing evaluation criteria. Government performance as a partner has also not shown much improvement over time. This is particularly important, as projects financed by IFAD are ultimately executed by governments, and their performance is thus critical to successful outcomes.
With regard to the performance of country strategies, the ARRI underlines that country programme evaluation ratings for relevance, effectiveness and overall performance, as well as for non-lending activities (partnership-building, policy dialogue and knowledge management), have improved overall since 2006-2008. Lack of resources – human and financial – remains the main limiting factor for non-lending activities.
The ARRI also notes that the grants programme is a very important instrument at IFAD's disposal, yet there are opportunities for a more strategic use of grants, including tighter synergies at the country level with investment projects financed by the Fund. There is also room for improvement in grant monitoring, supervision and reporting on outcomes.
This edition covers one learning theme: understanding exceptional projects. The review revealed a strong association between factors in all types of countries. Poor designs, poorly managed in difficult contexts, lead to exceptionally unsuccessful project outcomes. Good designs, well managed in supportive contexts, lead to exceptionally successful project outcomes. The few exceptionally successful projects in difficult contexts generally had good designs, high-quality project management, and good support from IFAD and government. IOE will deepen this initial analysis in 2014, when it undertakes the corporate-level evaluation of IFAD's engagement in fragile states and prepares an evaluation synthesis report of the organization's opportunities and challenges in working in middle-income countries.
IFAD has a unique role to play in the international aid architecture in reducing rural poverty, and its projects and programmes are making important contributions to government efforts to promote sustainable small-scale agriculture development. It is an organization on the move, concerned with making adjustments to its operating model, with the aim of achieving better results. There are, however, opportunities for improvement to achieve excellence, which will require continued focus on improving internal business processes, supporting governments in improving their capacity, including in the area of monitoring and evaluation (M&E), and greater customization to country contexts.
Three specific conclusions are worth highlighting:
- There is good evidence of improvement in some aspects of IFAD's performance. However, there are a number of persistent challenges, such as in efficiency, sustainability and M&E. These require a step-change in approach. Business as usual, and incremental improvement efforts, are as unlikely to lead to significant change in the next few years as they have in the past few years.
- Considerable progress has been made in improving project design and project supervision. A similar focus and effort is now required for project management. This is emerging as a very important, but underemphasized, determinant of project performance in all country contexts. A more consistent, more effective approach to ensuring and supporting high-quality project management is required, particularly in the early stages of project implementation.
- Fragile and/or conflict-affected states are an important focus for IFAD in both low- and middle-income countries. Fragility, conflict and poverty increasingly coincide. Thus, IFAD's relatively poorer performance in these situations represents an important challenge. The forthcoming corporate evaluation of performance in fragile situations will provide a critical opportunity for significant rethinking and change.
The 2013 ARRI makes the following four recommendations:
- the corporate-level evaluation on fragile states planned for 2014 should be expanded to cover fragile and conflict-affected situations in both low- and middle-income countries;
- the 2014 learning theme should examine the role of government, with a particular emphasis on strengthening the performance of project management teams;
- IOE should work with IFAD Management to design and implement a corporate-level evaluation on project management, including project-level M&E; and
- IFAD Management should pay special attention in next year's Report on IFAD's Development Effectiveness and the Annual Project Portfolio Review to the persistent challenges identified in this and previous ARRIs.
Annual Report on the Results and Impact 2012
This tenth Annual Report on Results and Impact of IFAD Operations (ARRI) presents a review of evaluations completed in 2011, with particular emphasis on highlighting what has changed since the first ARRI in 2003. The overall picture that emerges is positive, but with room for improvement, especially in light of the collective aim to transform the organization from generally moderately satisfactory performance to satisfactory, and if possible, even highly satisfactory results in the near future. On the one hand, there is evidence of clear and significant improvement across most of the criteria assessed by evaluations carried out by the Independent Office of Evaluation of IFAD (IOE). But equally, it is evident that a number of important issues identified in the early ARRIs are yet to be effectively addressed.
Performance and trends
Three broad performance trends can be observed from the 2002-2011 project evaluation data. Ratings for two evaluation criteria – natural resources and the environment, and IFAD’s performance as a partner – have improved over the decade. A second group of evaluation criteria – sustainability, innovation and scaling up, and two rural poverty impact domains (human and social capital and empowerment, and institutions and policies) – show a marked
improvement since 2002-2004, but a more recent decline since a peak in 2006-2008. A final group of evaluation criteria – relevance, effectiveness, efficiency, and the performance of government as a partner – show flat and/or declining performance.
Three other features have not changed over the years. First, a pattern of high project relevance and reasonable effectiveness, but only moderate efficiency and sustainability. Second, the predominance of moderately satisfactory performance. Just 23 per cent of the 2011 ratings were satisfactory or better. And third, the importance of the country context. Performance in fragile states is markedly lower than in non-fragile states.
Eighteen country programme evaluations (CPEs) have been completed since 2006. These show clear improvement in all three non-lending activities since 2006-2008 – knowledge management, partnership building and policy dialogue – but little improvement in the last activity since 2007-2009. Notwithstanding improvement in knowledge management, more efforts are needed at the country level, including a larger allocation of dedicated resources for this purpose. Similarly, as underlined in most past ARRIs, it is time for serious efforts to engage more systematically with the multilateral development banks and selected United
Nations organizations at the country level, including the Rome-based agencies.
Two priorities stand out in respect of country programmes. Most CPEs reveal that IFAD-supported country programmes are largely a collection of individual investment projects. The first priority is thus for better integration of all IFAD-supported activities, which is critical to scaled-up and sustainable impact. The second priority is for a well-resourced country presence, preferably with more immediately outposted country programme managers, especially in large countries with sizeable portfolios.
Annual Report on the Results and Impact 2011
This is the ninth Annual Report on Results and Impact of IFAD Operations (ARRI) produced by the Independent Office of Evaluation of IFAD (IOE). It consolidates and synthesizes the results and impact of IFAD-funded operations, using as its basis independent evaluations conducted by IOE, and it raises issues and highlights lessons learned that can contribute to improving IFAD’s development effectiveness. This ARRI draws on a larger number of project evaluations and country programme evaluations than any other edition has thus far.
The performance of IFAD-supported projects continues to be satisfactory in the areas of relevance, effectiveness, rural poverty impact, innovation, and gender equality and women’s empowerment. IFAD’s own performance as a partner has improved in the past years as well, in large part because of its progressive transition to direct supervision and implementation support, the expansion of its country presence, and more systematic portfolio management. One qualification, however, is that moderately satisfactory performance remains the norm. There is therefore still scope for further improvement.
There are some areas where performance is not as satisfactory. The efficiency of IFAD- funded operations, natural resources management and the environment, and the sustainability of project benefits are areas where performance still remains weak, but has nonetheless improved over time. The performance of recipient governments, in contrast, has not shown much improvement during the past decade.
For the first time this year, the ARRI compares the project evaluation results by IOE, the Fund’s independent evaluation system, with those generated by IFAD Management in its self-evaluation systems. It is reassuring that the “disconnect” between the two is relatively narrow. This attests to the increasing robustness and reliability of IFAD’s overall self-evaluation system. The ARRI has, however, identified a number of systemic lessons and issues for Management to consider when further developing the project completion review process, which is a key component of IFAD’s self-evaluation system.
The benchmarking analysis conducted in the context of the ARRI concludes that the performance of IFAD-supported projects is on the whole slightly better than the agriculture sector operations funded by the African Development Bank, the Asian Development Bank
and the World Bank. In terms of benchmarking performance across the five geographic regions, IFAD operations in the West and Central Africa region remain most challenging compared to other regions.
Annual Report on the Results and Impact 2010
This is the eighth Annual Report on Results and Impact of IFAD Operations (ARRI) produced by the IFAD Office of Evaluation (IOE). It presents a synthesis of the results and impact of IFAD-funded operations, and raises systemic issues and lessons learned that can contribute to further improving performance in the future.
This year’s ARRI has devoted particular efforts to summarize the reasons for stronger or weaker performance (the why question), as derived from recent and earlier evaluations. It includes a new section, derived from country programme evaluations, on the performance of non-lending activities, and a more disaggregated analysis of performance using all six ratings, rather than the two broad categories of satisfactory and unsatisfactory.
However, emphasis continues to be placed on the trends in the three-year moving averages, rather than on evaluation data from one single year. That is, caution should be exercised in drawing conclusions about performance from a single year of evaluation data, as well as in making comparisons of the evaluation ratings from one year to another. This is because the sample of projects evaluated by IOE in a given year is relatively small and not chosen on a random basis. Using a three-year moving average however allows for the assessment of trends in performance over time, and helps smooth out biases that may result from the sample of project evaluations not chosen randomly.
As in the past, the ARRI is mainly based on evaluation of past projects. Therefore, it is fair to note that the assessment contained in this year’s ARRI may not necessarily reflect the type and performance of projects designed more recently. This is most probably the case of projects that were analysed in the moving averages calculated for all evaluation data from 2002 up to around 2006. However, the ARRI also provides an appreciation of the performance of more recent operations based on the analysis of 44 projects that were evaluated in 2007-2009, which is the most recent point in the three-year moving averages of the entire data set used as a basis for the ARRI. Finally, in an attempt to provide a more “real time” perspective in its analysis, this year’s ARRI assessed the relevance of selected recent country strategic opportunities programmes (COSOPs) and project designs. The analysis reveals that IFAD is learning from the past, as more recent COSOPs and projects have more realistic objectives and are more results oriented than older operations
Annual Report on the Results and Impact 2009
This is the seventh Annual Report on Results and Impact of IFAD Operations (ARRI) prepared by the Office of Evaluation (OE). It presents a synthesis of the main results and impact from evaluations conducted by OE in 2008. As in past editions and as agreed with the Executive Board, the ARRI also includes dedicated sections on two learning themes: (i) access to markets; and (ii) natural resources and the environment.
The seventh ARRI includes three new features. First, in addition to providing an account of the results achieved based on the evaluations undertaken in 2008, the ARRI analyses the three-year moving averages, starting from 2002, of the performance of IFAD operations. The use of moving averages allows for the assessment of trends in performance over time, and also rectifies any biases that may result from the sample of projects evaluated, which are not chosen on a random basis.
Second, it contains a quality-at-entry review. The main aim of this section is to assess the extent to which IFAD management is learning from previous operations and addressing the key areas of challenge identified in this and past ARRIs in new activities financed by the Fund. This section also serves to develop the ARRI one step further, by providing an assessment of recently approved activities, in addition to the traditional account of past operations.
The third aspect is that the 2009 ARRI is built on evaluations undertaken in line with the provisions contained in the new evaluation manual, which was considered by the Evaluation Committee1 in December 2008. One change in the ARRI – in accordance with the new manual – is the reduction in the number of domains covered by rural poverty impact criteria from nine to five. Another change brought about by the new manual is a more comprehensive definition of relevance.
Included in chapter VII is a brief review of the ARRI itself. In preparing this seventh edition, OE undertook a comparative analysis of annual reports similar to the ARRI produced by evaluation outfits in other development organizations. The main aim of this review is to identify features that might be of interest to IFAD for inclusion in the ARRI in the future.
This year for the first time the ARRI includes the calculation of moving averages using the entire ARRI data set since 2002. The three-year moving average analysis reveals, on the whole, a steady upward trend in results across all but a few evaluation criteria. This upward trend is further corroborated by the analysis of project performance by date of loan effectiveness, which confirms the hypothesis raised in past ARRIs that more recent projects and programmes perform better than older generation operations. These positive trends merit emphasis, and reflect the considerable efforts by the institution to implement far-reaching reforms and renewal over the past decade.
Annual Report on the Results and Impact 2008
This is the sixth Annual Report on Results and Impact of IFAD Operations (ARRI) produced by the Office of Evaluation (OE). The ARRI report consolidates and synthesizes the results and impact of IFAD’s operations based on a cohort of project, country programme and corporate-level evaluations conducted in the previous year –in this case in 2007.
As in the past, the objective of the ARRI report is twofold: (i) to present a synthesis of performance based on a common methodology for evaluation; and (ii) to highlight key learning issues and development challenges that IFAD needs to address to enhance its development effectiveness. While the primary audience of the report includes IFAD Management and staff, and the Fund’s Evaluation Committee and Executive Board, it is also of importance to the wider development community because of the issues it raises for sustainable agriculture and rural development generally.
The present ARRI report follows the structure of the 2007 edition. Sections 2-4 synthesize the main evaluation findings from the project, country and corporate-level evaluations carried out in 2007. Section 5 presents an analysis of the consolidated evaluation data from all 85 projects evaluated since production of the first edition in 2003. Section 6 presents the report’s contribution to learning, which this year covers the themes of country context and project-level monitoring and evaluation (M&E), as agreed with the Executive Board in 2007. Conclusions and recommendations are contained in section 7.
The report also includes a specific section on two key learning themes: (i) the importance of analysing country context issues thoroughly in order to strengthen IFAD’s development effectiveness; and (ii) a reflection on the opportunities and challenges in enhancing the performance of project-level monitoring and evaluation (M&E) systems.
Annual Report on the Results and Impact 2007
This is the fifth Annual Report on Results and Impact of IFAD Operations (ARRI) prepared by the Office of Evaluation (OE). It synthesizes the main results and impacts from the 15 IFAD-funded projects evaluated by OE in 2006, and also draws upon the findings of three corporate-level and three country programme evaluations.
In addition to providing a comprehensive account of the results and impact of the evaluated operations, this year's ARRI report focuses more generally on learning, with the purpose of identifying the main determinants of performance and results. In short, it attempts to ask and answer the broad questions: “What was the performance?” and “Why was performance as it was?”
A review of previous ARRI reports by a professional statistician noted that a comparison of 2006 ratings with previous years - and indeed any comparison from year to year - was unreliable because of the small size and non-random nature of the sample. As a consequence, this year's report places less emphasis on such year-by-year comparisons. As in the past, however, it provides a summary of the 2006 evaluation data, plus an overview of the performance and impact of a larger sample, namely the 73 projects evaluated by OE over the period 2002-2006 and contained in the four previous editions.
With regard to learning, this ARRI report includes a specific section (6) devoted to two themes: sustainability and innovation, which emerged in past reports as areas in need of attention. The report builds on the inputs generated in two in-house learning workshops with IFAD Management and staff, and makes recommendations on how performance might be enhanced in these two key areas.
Annual Report on the Results and Impact 2006
This is the fourth Annual Report on Results and Impact of IFAD Operations (ARRI) produced by the Office of Evaluation (OE). As in the past, the ARRI report consolidates and synthesizes the results and impact of IFAD's operations based on a cohort of project and country programme evaluations conducted in 2005. It presents an analysis of last year's evaluation findings and a comparison with results of previous years (2002-2004), with the objectives of contributing to accountability in terms of IFAD's performance, and of learning from evaluation findings. The ARRI report's key findings aim to prompt discussion of necessary corrective action and changes in the way IFAD conducts its business. The ARRI report does not focus on the follow-up action taken by the Programme Management Department to evaluation recommendations, as these are discussed in the President's Report on the Implementation Status of Evaluation Recommendations and Management Action.
This year's ARRI report introduces target scores for each evaluation criterion as an internal benchmark to illustrate how such scores can be used for performance and results management. This is an experimental exercise to demonstrate the usefulness of such a system rather than to suggest a specific target score. Performance targets will focus on how effectively IFAD is addressing the criteria that are important to development effectiveness. The assumptions used in this ARRI report are based on a six-point rating scale that introduces greater differentiation between ratings and allows for a more nuanced performance assessment 1. Using this scale, it is reasonable, on balance, to expect 5 per cent of projects to be highly successful and 50 per cent to be successful. If an organization cannot achieve a success rate of more than 50 per cent, questions about its performance may be justified. It is also true that given the difficult and innovative nature of IFAD's work, some relative shortcomings are understandable. Therefore, targets of 20 per cent of projects being classed as moderately successful, 15 per cent as moderately unsuccessful and 5 per cent being classed in the bottom two categories (unsuccessful and highly unsuccessful) might be expected. These percentages produce a target mean of 4.2, which is just above a moderately successful rating. The only exception is relevance (one of the three project performance criteria), for which a mean target rating of 5 was set, as clearly all IFAD projects should have this characteristic.
The ARRI report follows the same structure as previous years. Section II provides an outline of the evaluations conducted in 2005. Sections III-V provide a synthesis of evaluation findings for project performance, rural poverty impact and partner performance, which are the main evaluation criteria used by IFAD (annex I provides an overview of the methodological framework). Section VI summarizes overall achievements and discusses the contribution of evaluated projects to IFAD's strategic objectives and to the Millennium Development Goals (MDGs). Finally, section VII presents the report's key findings and recommendations.
Annual Report on the Results and Impact 2005
This is the third Annual Report on the Results and Impact of IFAD’s Operations (ARRI) produced by the Office of Evaluation (OE). As in previous years, this ARRI consolidates and synthesizes the results and impact of IFAD’s operations based on a cohort of project and country programme evaluations conducted in 2004. In addition, it presents the consolidated findings from all projects evaluated by OE since 2002.
The introduction of the new methodological framework for project evaluation (MFE) in 2002 provided OE with a common framework for use systematically across all IFAD project evaluations. A slightly revised MFE was produced in September 2003. Compliance with the framework has improved steadily since 2002, as is evident from the much more consistent set of project evaluation reports produced in 2004. However, independent evaluations continue to be constrained by the weakness of project monitoring and evaluation systems. As a result, ratings have not been given for all indicators in all projects for the last three years. This, together with the relatively small sample size and short time period, limits the extent to which data presented here can be interpreted as trends.
The MFE consists of three main composite evaluation criteria: (i) performance of the project; (ii) impact on rural poverty; and (iii) performance of the partners. Each main criterion is divided into a number of elements, or sub-criteria. The criteria are broadly consistent with those used by other international financial institutions (IFIs), the Development Assistance Committee of the Organisation for Economic Co-operation and Development, and members of the United Nations Evaluation Group, although impact domains differ according to the mandate of each organization. The MFE is explained in more detail in Annex I, which also illustrates how the MFE’s impact domains relate to the objectives of the Strategic Framework for IFAD 2002-2006 and to the Millennium Development Goals (MDGs).
The main objective of the ARRI is to present a synthesis of evaluation findings based on the MFE. Each report is also intended to provide a basis for discussion within the Evaluation Committee, Executive Board and IFAD about how OE can best present its findings and how the report can be improved. This ARRI has taken account of comments received from the Evaluation Committee in September 2004.
The report follows the structure implied by the MFE. Section II provides an outline of the projects and country programmes evaluated. Sections III-VI provide a synthesis from the project evaluations in each of the main evaluation criteria: project performance (section III), impact on rural poverty (section IV) and performance of partners (section V). Section VI summarizes overall performance. Section VII examines the contribution to IFAD’s strategic objectives and to the MDGs. The concluding section presents the key findings, suggests explanations for these findings, identifies main issues for the future and makes a number of recommendations.
Annual Report on the Results and Impact 2004
The new IFAD evaluation policy approved by the Executive Board in April 2003 emphasized the role of the independent evaluation function at IFAD in promoting accountability and learning in order to improve the performance of the Fund’s operations and policies. An Annual Report on the Results and Impact of IFAD Operations (ARRI) was an essential part of this policy. The independent evaluation perspective contained in the ARRI is intended to complement the annual self-assessment undertaken by the Programme Management Department (PMD) – and embodied in the Progress Report on the Project Portfolio – as well as any reports to be produced by PMD using the Results and Impact Management System.
This is the second ARRI. Last year’s represented IFAD’s first attempt at consolidating and synthesizing the results and impact of IFAD operations, based on the ten projects and two country programmes evaluated during 2002, as well as the two corporate-level evaluations undertaken in the same year. This 2004 ARRI does the same for the ten projects and four country programmes evaluated during 2003, as well as two thematic evaluations and one corporate-level topic undertaken in the same year. It broadly follows the same approach and structure, but with a few minor changes.
The introduction in 2002 of the new Methodological Framework for Project Evaluation (MFE) provided a common framework to be used systematically across all IFAD project evaluations. Following experience gained in the first year, a review workshop was held to discuss the practical difficulties encountered in using the MFE and the ways in which it could be improved. This led to revision of the MFE. Most of the ten project evaluations summarized in this report followed the revised framework. Some of the evaluations began before it was issued and thus used the earlier one. While minor problems of consistency and interpretation remain – and will be addressed in further guidance to be issued by the Office of Evaluation (OE) – the implementation of the revised MFE has led to a much more consistent and complete set of evaluation reports.
The first ARRI was intended to provide a basis for discussion within the Evaluation Committee (EC), Executive Board and IFAD on how OE could best present a synthesis of its evaluations, and to generate suggestions on how the report could be improved. This 2004 ARRI has considered the comments received from the EC and the Executive Board.
Annual Report on the Results and Impact 2003
The IFAD V: Plan of Action (2000-2002) recommended that the Office of Evaluation (OE) produce a systematic overview of the results and impact of IFAD’s operations, based on the evaluations it undertakes each year. One of the obstacles to such an overview was the lack of a consistent evaluation framework, but this obstacle has now been removed. With the introduction in 2002 of the new Methodological Framework for Project Evaluation (MFE), a common framework is now available for use systematically across all IFAD project evaluations. The main objectives of the MFE are to: (i) promote a more systematic assessment of impact at project completion; (ii) facilitate the production of a consolidated picture of the results, impact and performance of the evaluated projects during any given year; and (iii) facilitate the consolidation of insights and learning from evaluation.
The new IFAD evaluation policy approved by the Executive Board in April 2003 re-emphasized the recommendation made in the IFAD V: Plan of Action. It states, "The main purpose of the independent evaluation function at IFAD is to promote accountability and learning in order to improve the performance of the Fund’s operations and policies. Evaluations provide a basis for accountability by assessing the impact of IFAD-supported operations and policies. They are expected to give an accurate analysis of successes and shortcomings, i.e. ‘to tell it the way it is’. This feedback helps the Fund improve its performance." The policy states that such overview is to be presented annually to the IFAD Executive Board.
This Annual Report on the Results and Impact of IFAD Operations (ARRI), based on the projects evaluated during 2002 and using the MFE for the first time, represents IFAD’s first attempt at consolidating these results. As such, it is necessarily exploratory, experimental and a challenge for OE and IFAD. All multilateral and bilateral agencies are, to a greater or lesser extent, having to face up to a similar challenge: how to analyse, and report on, their overall effectiveness on the basis of improving, but still limited, evaluation and performance information. In generating insights and learning from evaluation, this ARRI also combines the results of two country programme evaluations (CPEs) and two corporate-level evaluations (CLEs) undertaken during 2002.
The report presents a mix of qualitative findings and ratings. Summarizing a large volume of evaluation material, and expressing judgements on impact and performance through ratings, is bound to be an approximation and to some extent an oversimplification. There are inherent dangers in reading too much into aggregate ratings, particularly in view of the small and diverse sample of projects covered. Great care needs to be taken in interpreting the results and drawing definite conclusions on the basis of this year’s data alone. That said, OE believes that there is value in presenting data clearly and transparently as a basis for discussion and as a contribution to constructive debate and learning within IFAD and outside.