Mountain Areas Development Programme (2008)

Albania  
December 2008

Completion evaluation

Introduction

Background

The mountain areas in Albania cover 70 per cent of the country's territory and have a considerably higher incidence of poverty than the lowland coastal areas. Agriculture, livestock and agribusinesses are the main employment and income opportunity in these areas, but low productivity, small farm sizes, limited access to financial services, and weak market linkages are among the factors that have inhibited effective and sustainable growth and poverty reduction. The IFAD supported Mountain Areas Development Programme (MADP) was designed to improve the living standards of mountainous households using a multifaceted strategy, which included financial services, agricultural development and institutional strengthening.

Political and social context

The MADP was designed and implemented during a period of significant political, economic and social transformation in Albania. Still emerging from one of the most isolated communist regimes, Albania was in 1997 thrown into substantial violent civil unrest as a result of the collapse of the pyramid schemes.

One year later the Kosovo crisis began, culminating in the arrival of hundreds of thousands of refugees impacting negatively on the social and food security situation, especially in northern Albania. Ethnic inspired unrest later spread to another neighbouring country, Macedonia, and between 2001 and 2002 trade and travel links were severely disrupted as a consequence. This impacted negatively on the already precarious Albania's situation, especially in the northern areas. In addition, the considerable political instability and uncertainty following both parliamentary and local elections also proved disruptive to the civil service, as they often resulted in changes in staff, which also affected MADP (as it will be later discussed).

Economic and poverty situation

The economy of Albania has generally performed well with strong macro-economic growth since 1993 averaging between five and seven per cent annually although it suffered a substantial economic contraction in 1997 as a result of the collapse of the pyramid schemes.1

Key drivers for achieving and maintaining these relatively high growth rates have included the rapid privatisation and trade liberalisations, a strong fiscal discipline and a relatively well educated labour force able to exploit the market opportunities that were opened in the post-communist period. Also, facilitating this benign outcome has been the establishment of a sound financial sector in the aftermath of the pyramid schemes crisis of 1997. Industry accounted for about 19 per cent of Gross Domestic Product (GDP) in 2006 down from an average of about 45 per cent in the late 1980s. The service sector is a smaller part of the economy than in most other post-communist countries in Eastern Europe. Agriculture's share of GDP has shrunk from about 33 to 23 per cent between 1996 and 2006, although around half the population derives a living primarily from agricultural production (EIU, 2007).

While strong economic growth has contributed to reduce poverty, Albania started from an extremely low base and still today remains one of the poorest countries in Europe with an estimated Gross National Income (GNI) per head of US$2,510 in 2005. According to a 2006 survey2, an estimated 18.5 per cent of the population live below the national poverty line of US$2 a day and 3.5 per cent of the population live on an income of less than US$1 a day, the latter being classified as extreme poverty. These data show an improvement from the 2002 Living Standard Measurement Survey (LSMS)3, according to which aggregated level consumption poverty was estimated at around 25 per cent, whereas extreme poverty was five per cent. This substantial reduction in poverty has been accompanied by regional convergence in poverty trends, with a sharp reduction in the poorest region, the Mountain areas, and comparatively slower, but still significant reductions in poverty in the Coast and Central areas. However, rural poverty is still significantly higher than urban poverty, with three quarter of all the poor living in rural areas.

Evaluation objectives, methodology and process

The Office of Evaluation (OE) of the International Fund for Agricultural Development (IFAD) has undertaken a Completion Evaluation of the MADP. The objectives of the evaluation were to: (a) assess the performance and impact of the project; and (b) generate findings and recommendations that would serve as inputs for the design and implementation of future projects in Albania with similar characteristics. The evaluation was conducted in line with the IFAD Evaluation Policy and utilised OE's methodology for project evaluations. The methodology focused on four dimensions: (i) the performance of the project measured in terms of relevance, efficiency and effectiveness; (ii) the rural poverty reduction impact of the project; (iii) the performance of partners including IFAD, the Government of Albania (GoA), United Nations Office for Project Services (UNOPS), UK Department for International Development (DFID) and others concerned; and (iv) sustainability and innovation. The evaluation mission conducted field work in Albania in June 2007. The evaluation team examined data obtained from the programme management units, interviewed key informants, conducted focus group discussions with beneficiaries using standard questionnaires and guidelines, and visited sites to see various activities funded by the Mountain Areas Development Agency (MADA) and by the Mountain Areas Finance Fund (MAFF).

Both quantitative and qualitative techniques have been adopted and the main findings presented in this report are the results of triangulation between different methods and sources. An important source has been the self-evaluation undertaken by the MADP, which has provided valuable analysis especially in relation to the more qualitative aspects of MADP's performance. The evaluation teams have some concerns about the integrity of some of the data sources (mostly from the Government of Albania), as will be discussed below.

The mountain areas development programme

Rationale

The MADP was designed to continue, strengthen and expand the activities of the two previous IFAD-funded projects: the North-eastern Districts Rural Development Project (NDRDP) and the Small-Scale Irrigation Rehabilitation Project (SSIRP), which were implemented in the poorest part of the mountain areas. The organization and management of MADP was designed to shift IFAD's support to mountain areas from the ad hoc spot application of area-based subsector projects to a long-term programmatic view of mountain areas development, to be based upon a rational and synergistic portfolio of investments.

Programme's key dates

MADP was approved by IFAD's Executive Board in December 1999, the loan agreement was signed in January 2000 and the programme became effective in July 2001. The actual start up of MADA's activities was further delayed due to several reasons, such as disagreement between IFAD and the Government of Albania on some of the staff appointments. MADA became operational only in May 2002 following the approval of its 2002 Annual Work Plan and Budget (AWPB) by the Agency's board of Directors. The Mid-Term Review (MTR) was conducted in October 2003. The project completion date was 30 September 2007, while the loan closing date is 31 March 2008.

MADP overall objective and components

The overall objective of MADP was to achieve a sustainable long term economic growth and development in the mountain areas of Albania, and to raise the standard of living of 37 500 mountain area-based households. The overall objective would be achieved by financing the following components: i) Programme Management: in order to ensure a systematic impact, the programme would establish an agency for MADA, which would be a small facility for programming, planning and fund management; ii) Rural Credit: through the establishment of a sustainable non-banking institution such as the Mountain Areas Finance Fund (MAFF), the programme would provide social and agricultural production credit on a sustainable basis to clients living and working in poor, marginal mountain areas; iii) Rural Infrastructure: this component would focus on the rehabilitation of small-scale irrigation schemes and construction of rural roads and village water supplies; iv) Agricultural Development has several subcomponents including: support to community management plans for pasture and forest land use; testing and vaccination of cattle and small ruminants; support to the private veterinary services; and provision of demand-driven extension services.

Financing

MADP's total programme cost was US$23.1million (See Table 1 in the main report). IFAD financed 59 per cent of total costs, through the provision of a highly concessional loan of US$13.2 million and grant of US$0.4 million. Albanian Government funded contribution was of US$2.9 million including foregone duties and taxes (12 per cent). The contributions of beneficiaries amounted approximately to US$1.2 million. The main co-financiers were: Department for International Development (DFID) that contributed US$1.9 million; the Netherlands Development Organisation (SNV) that agreed to co-finance US$0.4 million; the Italian Cooperation that provided US$1.0 million that was later supplemented with US $0.1 million4.

There were also supplementary funds, which had not been foreseen at the time of programme design. These included US$60,970 and US$40,500, which were secured from the World Bank-financed Agricultural Service Project (ASP) in Albania and from MADA beneficiaries.

These funds were used towards a joint MADA/ASP pilot initiative, namely the ‘joint competitive grant scheme'. Moreover, through it's Gender Mainstreaming Programme for Central and Eastern Europe and the Newly Independent States countries (CEN), IFAD provided a Technical Assistance Grant (TAG) of US$88,200 for the MAFF-initiated "Knowledge Generation and Skills Development Project for Rural Women and Youth".

MADP performance

Relevance

MADP has been mostly relevant and, perhaps more importantly, it has shown the flexibility to learn from experiences gained. It has adapted to changing circumstance and redesigned key interventions to increase its relevance.

The establishment of MADA and MAFF has been relevant as these are key institutions dedicated to reducing rural poverty in mountain areas. The programme objectives have been highly consistent with the needs of the rural poor who live in mountain areas, with the priorities of the GoA, with IFAD's 1998-2002 and 2002-2006 Strategic Frameworks and with IFAD's 1999 COSOP for Albania, which emphasised the need to move beyond time-bound project approaches towards a cogent and coherent programme for the development of marginal areas in Albania. In relation to the COSOP, it seems that MADP's original design and its 2003 MTR reorientation have informed both the 1999 and 2005 COSOPs for Albania rather than vice-versa, especially on modalities by which the overall poverty reduction strategy was to be implemented. Thus MADP has arguably also assisted in improving relevance of the Albania COSOP. Both COSOPs argued for channelling support through institutions that were permanent in structure and not through specific IFAD Project Implementation Units (PIUs). This design feature seemed highly relevant, as the experience with PIUs had generally proven unsustainable.

Over time, MADP has been able to make significant changes as signs emerged that some of the initial modalities were not optimally designed and were not in line with the evolving country's changing context. At project start, MADP's heavy reliance on collective actions and formation of associations did not prove to be appropriate to the Albanian context. Realising this, the 2003 MTR made efforts to restore the programme's relevance by increasing focus on fostering structural change in mountain areas and emphasising support to the private sector. The strategies for MADP activities and its institutions were further developed, in many cases revised and a strong focus was put on the new Institutional Building component. In addition, the Agricultural Development component was re-oriented (and renamed) as the Private Sector Development (PSD) component. The new approach for this component emphasized the commercial orientation of MADA and was intended to enable farmers and agribusinesses to fully exploit emerging opportunities. To this aim, Strategic Investment Plans (SIPs) were developed to strengthen the growth of strategic farming and agri-business activities with high profitability and potential for increased commercialisation. In relation to the provision of rural financial services, MAFF began gradually to emphasise loans to individuals and Small and Medium Enterprises (SME) with a much more direct incentive for productive and profitable investments, thus discarding the less relevant group-based approach. Since the MTR, the programme expanded its geographic targeting from poorer mountain areas to all mountain areas in the country, with a special emphasis on those activities with a high potential for increasing productivity and profitability.5

Effectiveness

In assessing MADP's effectiveness, an important consideration concerns the fact whether the interventions have actually taken place where the poor are located. MADP was designed to start in the south due to the presence of two previous IFAD-funded projects (Northern-eastern Districts Rural Development Project (NDRDP) and Small-Scale Irrigation Rehabilitation Project (SSIRP) located in Albania's northeast and centre-east areas, but then at the phasing out of these two projects, the MADP was supposed to move to the poorer northern mountain areas. However, after the closure of NDRDP and SSIRP projects, the investments and activity level of MADP remained largely concentrated in the relatively substantial richer southern mountain areas, allocating only around 16 per cent of investment funds (i.e. sourced from MADA) to northern ones. If MAFF funds are included too, the figure becomes 27 per cent, which is still a strong bias for southern mountain areas. There are poor people in the southern areas too, but the incidence and depth of poverty are significantly lower here and the programme could have been more effective in reaching poor people by taking a wider geographic coverage. The evaluation mission was unable to find technical evidence as to why MADA investment allocations were biased against the northern areas. Loan funds from MAFF, on the other hand, were essentially equally distributed between north and south.

MADP's effectiveness has been mixed but with an improving tendency throughout the programme period. MADP's programme objective was to improve the living standards of 37 500 poor mountain area households. According to MADP figures (2007) 56 488 households have received MADP services and their living standards have been raised accordingly. However, while living standards have generally increased rapidly during the programme period, the evaluation team would like to caution about drawing a too close causal relationship between MADP's interventions and this rise. Undoubtedly, MADP has played an important (and increasingly strategic) role, but there are some concerns about the degree to which necessary complementary actions by other actors (most notably the government's veterinary service and water user associations) have been sufficient to achieve the desired effectiveness. This especially concerns the interventions that started before 2003, most notably in the irrigation and vaccination sectors, which account for the majority of beneficiary households. The implicit assumption that these external actors would deliver the needed follow-up to MADP investments at times proved too optimistic and has in certain instances compromised effectiveness. In any case, exclusive focus on numbers may not capture important contributions from MADP both in terms of delivering tangible benefits to numerous households, but also in terms of piloting and demonstration activities especially within the private sector development and the credit sector, where MAFF has proven the viability of rural financial services and introduced new products. Other interventions concerning the rural infrastructure component and the vaccination activities have had more limited effectiveness and almost no-spill over effects and for all interventions the richer southern districts have been found to be the main beneficiaries.

Efficiency

It is difficult to evaluate MADP's efficiency at aggregate level. At programme's appraisal, an Economic Rate of Return (EIRR) was estimated at 37 per cent, heavily relying on the three main investments in irrigation, vaccinations and agricultural extension. The reach, coverage and economic lifetime of these investments were overestimated during the design phase. The economic life of the single largest investment (in irrigation) was set at 20 years at appraisal: this is evidently an overestimation as many of the schemes are already disintegrating with virtual no institutionally sustainable structures to ensure their maintenance. Already in 2001 UNOPS Supervision Mission claimed a MADP's estimated EIRR of only 21 per cent stating that this reflected the level of sunk costs in existing irrigation structure and conservatively estimated incremental benefits. In 2003 this figures was revised further downwards as many of the investments were deemed unsustainable, whilst benefits from incremental marketed output were likely to have been overstated. It was argued that the EIRR, while impossible to be accurately assessed, was likely to be near zero or negative.

Also reducing efficiency has been the substantial implementation delay of nearly two years at programme start. This affected especially MADA, whereas MAFF generally had somewhat higher efficiency, especially after it started to phase out the group-based loans. Based on various estimates of the economic life-time of the infrastructure and coverage of the vaccination programme, the evaluation team has made calculations of the EIRR and would argue that efficiency could have been significantly higher.

Impact on rural poverty

At a component level, the impact of the vaccinations interventions and rural infrastructure is disappointingly low, especially when considering the amount of funds invested (79 per cent of MADA's total investments). While there is still some uncertainty as to the ultimate impact of the vaccination programmes, it is clear that these have not managed to bring major zoonotic diseases under control. As for the rural infrastructure, most of the investments have proven to be unsustainable and based on flawed design. In addition, the heavy reliance on collective action through e.g. Water User Associations (WUAs) for Operation and Maintenance (O&M) proved ill-suited to the Albanian – highly individualised – social context. Similarly, many of the forest and pasture management associations have failed to maintain impact after the cessation of IFAD/MADA support.

Conversely, most of the activities that were initiated after the 2003 MTR have demonstrated significant strategically impact, albeit still on relatively small scale. The Strategic Investment Plans (SIPs) have facilitated the introduction of crucial agricultural technologies and the increased emphasis on a broader range of actors in the value chains has proved successful in raising agricultural productivity and, most crucially, profitability. The rural credit component, implemented through MAFF, has impacted directly on the beneficiaries' access to financial services, as they are now able to increase investments and smooth consumption patterns thus reducing exposure to e.g. seasonal or unforeseen downturns in the economy. Moreover, after a somewhat misguided start, the component has helped restore a healthier and sustainable credit culture, by following a strict commercial approach in its lending to individuals and Small and Medium Enterprises (SMEs).

Sustainability

An important determining factor for sustainability seems to be the degree to which individuals, companies or small cohesive groups of individuals (socially and/or commercially) have a clear and direct incentive to continue the activity after support is withdrawn. Perhaps the most powerful incentive is the profit motive, which explains the significantly higher degree of sustainability of e.g. SIPs compared to other various donor-established user groups/associations, such as the WUAs and the forest and pasture management user associations. The economic sustainability of many irrigation schemes, which have accounted for a significant amount of investments, is also questionable due to the low value mix of crops being under irrigation. These are often not profitable, as the income they generate is not sufficient to cover the full cost of irrigation schemes, including depreciations. However, it is possible to take comfort in observations from the SIPs and interventions in which privately owned irrigation schemes have worked and continue to work post-project.

With regard to Fora, both at district and national levels, it seems that their political and social sustainability is also limited, as they are substantially dependent on the income and funding for activities from MADA. With the end of MADA funding, the level of Fora's activity has dropped substantially, with the result that many Fora have either turned largely dormant, or sought other donors' assistance for a variety of purposes and at times somewhat opportunistically without strong strategic guidance. The sustainability of support to private and public veterinary services is also uncertain and improvements are needed in all aspects of vaccine production in future campaigns to not jeopardise previous MADA efforts.

As far as the two key institutions - MADA and MAFF - are concerned, MADA has become a more sustainable market-supporting centre of excellence for mountain areas development and the government is considering devolving substantially more legal authority to MADA; a clear indication that the institutional sustainability is improving. However, it will probably still need external funding for a number of years. In relation to MAFF, this was initially saddled with a high portfolio at risk primarily from the collective loans disbursed under the Village Credit Funds scheme (VCFs). Most of these VCFs proved unsustainable. This having been overcome, sustainability prospects appear bright provided MAFF is given the full ability to compete on equal terms with the growing competition in the financial sector. Privatisation will probably be the only long term sustainable solution.

Given the substantial initial focus on partly supply-driven approaches (especially in irrigation and rural road construction), sustainability has generally been jeopardised in many instances. However, this judgement does not do full justice to MADP's dynamics and to its successor programme the ‘Sustainable Development for Rural Mountainous Areas Programme' (SDRMA), which have seen clear sustainability improvements especially in relation to the specific priority interventions initiated after the 2003 MTR not least the strategic investment programmes and the re-oriented financial focus of MAFF.

Innovation, replicability, and scaling-up

The two major institutional innovative features of the MADP have been arguably the establishment of MADA and MAFF, as permanent locally embedded institutions.

MADA has become a market-supporting centre of excellence for mountain areas development and may eventually be developed into a regional development agency as seen in other European countries. This has clearly been an innovation in the Albanian context. MAFF is also an innovative feature, and it has produced innovative products such as flexible repayment regimes that have been well received among its customers.

At a component level, the introduction of SIPs was a major innovation within the programme, redirecting focus from supporting essentially economically doomed small-scale subsistence farmers, towards commercial farmers with a strong potential to up-scale. This is arguably also rather innovative in relation to IFAD's more traditional approach to poverty reduction, which tend to argue for continued support to small-scale farmers agricultural production. Obviously this does not entail that IFAD should refrain from working with the poor directly; the key point here is that the initial focus on promoting incremental improvements in the poorest farmers' agricultural production had virtually no sustainable impact and hence also no potential for up-scaling. However, in order to make a well balanced assessment of MADP's innovative features, it needs also to be highlighted that the programme's original design was initially based on some old-fashioned interventions, especially with regard to the infrastructure component that was using decade old designs for irrigation with no potential for up scaling.

Partner performance

IFAD played a crucial role in the design of MADP, drawing on the experiences of two previous IFAD funded projects. However, MADP design turned out to have some structural flaws that were partly based on unconvincing assumptions which had not been fully analysed by neither IFAD or by the Government of Albania. Later on, IFAD was instrumental in the much needed reorientation of the programme which took place after the MTR. IFAD strategically used MADP as a policy platform for advancing the dialogue on mountain areas development, which has catapulted the issue to the agenda of policy-makers and has ensured stronger government commitment. However, IFAD should have invested more resources in pledging a more balanced and pro-poor geographical orientation of activities and in ensuring stronger voice of the rural poorest in the advocacy activities.

Government of Albania

Has been the key domestic partner and has provided a generally enabling legislative framework e.g. for MAFF in the field of micro-financial regulation, but also for private sector development in general. Macro political changes seem at times to have impacted on the frequency and appointment of senior management positions, thus generating delays and some loss of institutional memory. In some instances, lack of complementary actions also contributed to reducing the programme's performance (in e.g. vaccinations) and the staff appointed by GoA did not always ensure proper pro-poor geographical targeting. Local governments have participated enthusiastically in the strategic development planning exercises piloted under the local area partnership sub-component.

UNOPS performance as the Cooperating Institution responsible for supervision was generally satisfactory, especially considering the relatively budget constraints under which it performed. Its supervision duties were managed with a reasonable degree of responsibility, proportionally to the allocated resources, alerting stakeholders to possible problems and providing part of the remedial resources such as technical inputs.

The communities participated in a number of contexts from WUAs to forest and pasture user associations, to Fora development with reasonable dedication, at least as long as they derived direct programme benefits. However, very few honoured the commitment to continuously provide user/community contributions and all displayed challenges in operation and maintenance. Also many ‘community leaders' made their request for irrigation rehabilitation not on the basis of the whole community's needs and aspirations, but often following political and personal preferences and priorities.

DFID has been mainly involved in providing technical assistance to a number of activities, including overall programme management, SIPs and local partnership initiatives. Concerns have been voiced about the degree of local involvement, capacity development and ownership of some products.

The Italian Cooperation has primarily provided funds in a timely manner and thanks to its financial support a number of workshops were held at the district, regional and national level with a wide range of stakeholders. In MADP's initial phase, the Dutch SNV provided TA to the forest and pasture management associations; these inputs have been appreciated as they facilitated the implementation of important activities such as the private sector development activities and tree planting.

Conclusions

The context in which MADP was initially designed and implemented deserves special attention. The severe political and economic disruption caused by the collapse of the pyramid schemes in 1997, the Kosovo crisis in 1998-99 and the trade disruption caused by the Macedonian crisis in 2001-02, clearly instilled a sense of urgency (and possibly emergency) among all stakeholders, which reduced focus on long-term sustainability issues and diverted the attention from the need to foster structural change in the mountain areas.

In addition to the two years delay in project start due to disagreements on senior staff appointments, there was also considerable project inertia with many modalities and concepts being somewhat inherited from previous projects, such as the adoption of group - based associations and activities that proved not to be suited to the Albanian context.

This led to the design of interventions with only limited impact and sustainability, often being supply- driven and with poor monitoring and supervision (e.g. vaccinations and irrigation). Combined with a partly misguided approach towards poverty reduction and too heavy emphasis on utilising discredited implementation modalities, the project performance in the first years was unsatisfactory.

To the credit of IFAD, UNOPS and GoA, during the MTR remedial actions and strategic reorientations were proposed in order to restore relevance and improve MADP's impact and sustainability. Emphasis was put on supporting the private sector and encouraging already emerging developments of land consolidation and commercialisation. Empirically, this has proven the only relevant growth strategy for mountainous areas. The alternative of promoting incremental improvements in the poorest agricultural production techniques on very small plots turned to be irrelevant as it has not been able to yield any substantial and sustainable impact on poverty reduction, but rather has further delayed an inevitable rural transformation process towards a higher degree of commercialisation and higher productivity. This strategic shift has successively promoted and increased relevance and looks set to be continued in the ongoing successor programme ‘Sustainable Development for Rural Mountainous Areas programme' (SDRMA), which is also supported by IFAD.

 The single largest achievement of MADP is arguably the establishment of the two core institutions of MADA and MAFF. They have proven relevant as key institutions dedicated to advancing the course of mountain areas in general and the poor living there in particular. They have also assisted in raising the profile of the mountain areas on the policy agenda.

The general knowledge of the specific needs and problems, but also opportunities and investment potentials, of these areas is now significantly enhanced among the public and policy-makers. However, both institutions could have performed even better had they been subject to less political interference and more robust impact evaluation.

MADP Performance Rating6

Relevance

5

Effectiveness

3

Efficiency

3

Project Performance7:

Rural Poverty Impact

4

Sustainability

3

Innovation

4

Overall MADP Achievement8

4

Partners Performance:

 

IFAD

4

Government of Albania

3

UNOPS

5

In light of the above, MADP's overall achievement has been rated as moderately satisfactory (4). While initial performance was not fully satisfactory, the programme, especially after the Mid-term Review, has made continuous attempts to improve its performance, most often with considerable success. Performance has thus improved over time and MADP has increasingly supported and benefited from the economic revival in the mountain areas that has occurred during the programme period.

Recommendations

Accelerate the strategic shift supporting a private sector led structural transformation

This should form the backbone of the mountain areas growth and poverty reduction strategy and all partners should remain clear that this process will produce both winners and short-term losers. In the agricultural sector this will entail support to land concentration, to commercialisation and industrialisation, a process that is likely to temporarily marginalise the least resourced farmers as they cannot meet the increasingly demanding standards of modern agriculture. This process is likely to be intensified as Albania seeks European Union (EU) approximation to food safety standards (in particular Sanitary and Phytosanitary Measures (SPS) and Hazard Analysis Critical Control Point (HACCP)), which in turn will force farmers to meet quantity, quality, timeliness and traceability requirements of new supply chains. Small scale, under-capitalised and often under-educated farmers have only limited prospects in this scenario, even with IFAD assistance.

Increase the poorest labour market mobility enabling them to exploit emerging non-agricultural opportunities

Perhaps too often, IFAD (both in Albania) has attempted to improve existing, mostly agriculture-based livelihoods of the poorest. Thus initial emphasis in MADA was to improve - marginally – the productivity of small-scale farmers by e.g. irrigation. As argued above such a strategy is likely to only delay an inevitable process. Instead, more efforts should be made to complement the above mentioned private sector led growth strategy with targeted efforts aimed at improving the poorest people's skills and competencies, enabling them to take advantage of the new opportunities emerging in both rural and non-rural settings. Retraining, vocational education and targeted courses could form part of such a complementary strategy.

Increase the voice of the poor mountain people in policy making and allocations of importance to them

At the moment the key vehicles for promoting voice and accountability - the Fora - are not representing the rural poor, and broadening Fora membership may undermine cohesiveness and sustainability. Going forward IFAD and MADA may need to devote more resources in analysing how to better ensure the representation of the poor in advocacy efforts.

Prioritise districts with higher than average poverty rates

It is not appropriate that MADA has focussed efforts on relatively less poor areas, leaving out more deserving ones. This needs to be corrected and will have to entail more investment in northern mountain areas. IFAD should closely monitor spatial disbursement patterns and not allow a repetition to occur.

Make a clear, sequenced and time specific privatisation plan for MAFF

While there was an argument for using public funds (IFAD and government) for reaching poor mountain households when the programme started, this argument is now starting to lose validity. Commercial banks are partly taking their cue from MAFF by investing heavily in mountain areas, and MAFF should be given the full operational and management freedom needed to remain competitive that only a full privatisation can offer.

Ensure more realistic analysis of incentives and political economy issues in the design of similar programmes

The MADP experience testifies to the need to critically analyse both economic and political incentives of all stakeholders (including possible losers) especially when designing interventions based on collective approach, be it credit, infrastructure or natural resource management. Initially MADP was institutional naive in its assumptions.

The MADP experiences also suggest that there are significant dangers in following donor fashions as the evidence from irrigation, forest management and micro-credit testifies. Too often such approaches have been used as a blue-print that have had relevance in another context, but not in Albania. Interventions relying on the establishment of new groups and associations need to be carefully evaluated, utilising both insight from the political economy of collective action and more simple incentive analysis.


1/ The highly leveraged and ultimately unsustainable pyramid investment schemes are estimated to have resulted in the loss of some US$1.2 billion in people's savings and triggered a crisis that brought the country to the brink of civil war.

2/ See Albania's Institute of Statistics (INSTAT) and the World Bank on ‘Trends in Poverty and Inequality', 2006.

3/ This survey found that rural poverty was significantly higher than urban poverty, by a factor of at least ten percentage points. Moreover, the northern part of the Mountain region (the North and the North-east agroecological area) had the worst poverty outcomes among the country's four regions (Tirana, Coastal, Central and Mountain). Almost one-half of the population in this area was poor and one in five could not meet basic food needs. Poverty also had a strong gender dimension as the transition period has had a disproportionately negative impact on women's economic and political status.

4/ The US$1.0 million and US$0.1m from the Italian Cooperation were financed through a programme called Facility for Farmers' Access to Markets (FFAM) in the Balkan Area.

5/ This also proved consistent with IFAD's ‘Targeting Policy' of September 2006.

6/ The ratings are based on the OE's six-point rating system: Highly Satisfactory (6); Satisfactory (5); Moderately Satisfactory (4); Moderately Unsatisfactory (3); Unsatisfactory (2); Highly Unsatisfactory (1).

7/ As per OE's Methodology, the Project Performance rating is the arithmetical average of Relevance, Effectiveness and Efficiency.

8/ As per OE's Methodology, the Overall Project Achievement is a composite assessment of the six evaluation criteria: relevance; effectiveness; efficiency; rural poverty impact; innovation and sustainability.

 

LANGUAGES: English

Southern Region Cooperatives Development and Credit Project (SOCODEP) (2008)

Ethiopia  
December 2008

Completion evaluation

Introduction

The Southern Region Cooperatives Development and Credit Project (SOCODEP) was undertaken in the Southern Region of Ethiopia between 1994 and 2005. This document is the completion evaluation report for the project. The evaluation mission took place between 17th September and 11th October 2006. It was conducted by a four-man team and the conduct, analysis and reporting of the evaluation has been carried out according to the Office of Evaluation (OE) guidelines for project evaluations.

Country background

Ethiopia

Ethiopia is a very large (1.1m km2), very diverse (geographically and ethnically), and extremely poor country. It ranks 170th out of 177 countries in the current (2006) Human Development Rankings. The predominantly rural population depends heavily on subsistence agricultural production, the vast majority of which exists under rainfed conditions. Cash incomes are very low except in coffee-producing areas, and access to rural financial services is very limited. Physical infrastructure (roads, power supplies, water and sanitation, and telecommunications) is very poorly developed. Population growth, increasing prevalence of the Human Immuno-Deficiency Virus/Acquired Immuno-Deficiency Syndrome (HIV/AIDS), environmental degradation, and increasing climate instability are among the trends which add to the challenges of poverty alleviation in Ethiopia. Politically, Ethiopia has undergone a virtually continuous process of change since the 1974 revolution in which Emperor Haile Selassie was overthrown. The Marxist rule of the Derg from 1974 to 1991 led to many reforms, but the authoritarian use of power over the peasantry did not change, or if anything, it became more restrictive. After the fall of the Derg in May 1991, a slow process of liberalisation of economy and political systems has taken place, but there is still a long way to go before the rural population will be fully able to participate in a free market and transparent democratic processes of government. There is no doubt that Ethiopia is a very difficult country in which to make progress in terms of development and poverty alleviation.

The Southern Region

At 232,000 km2, the Southern Nations, Nationalities and Peoples Regional State (SNNPRS) is one of the four largest Regions in Ethiopia (out of the 9 Regions and 2 City Authorities into which the country is presently divided). It is probably the most diverse Region in ethnic terms, and it contains some of the most remote and wettest parts of Ethiopia. At the time of project commencement in 1994, the Region consisted of 53 Woredas (districts), and had a total population of about 11m. Twelve years later, at the time of the completion evaluation the number of Woredas had grown to 133 and the total population to about 14.5m. Mean population density in the Region has consequently increased from about 47 to 63 per km2, but these averages hide a great deal of variation; at Appraisal in 1994, population densities were thought to range from less than 10 to over 500 per km2. Rainfall, temperatures, altitudes, cropping patterns, infrastructure, and cultures vary widely within the Region.

The project

SOCODEP was one of the first significant internationally funded interventions in Ethiopia following the fall of the Derg in May 1991. The Project aimed to respond to the then new legislation (Proclamation 85/1994) concerning Cooperatives, which ostensibly set out a means of turning the former Government-imposed and politically-dominated Producer Cooperatives of the Derg into farmer-owned viable business entities serving their members' interests. In particular, it aimed to make rural finance (specifically micro-credit) available to so-called Service Cooperatives and their members. The Principal Objective of the Project was to "increase agricultural productivity and raise income levels of the rural poor through support to Service Cooperatives' development in order to facilitate efficient provision of sustainable services to members". The Project had 7 specific objectives which are set out in Table ES1 and further elaborated in the reconstructed logframe in Appendix 4. T

ogether these objectives were designed not only to turn Service Cooperatives (SCs) into effective vehicles for the service of their members, but also to deliver some specific services (road construction, veterinary drug supply, health facility up-grading, and provision of water supply and sanitation services) directly through the relevant Government organs. The Water Supply, Health and Basic Sanitation Component (WSHBS) was separately financed by the Belgian Survival Fund as a later addition to the Project. 

Table ES1.  SOCODEP specific objectives


SO1

Provide a model for developing Ethiopian cooperatives under the new legislation, particularly with respect to improvement of financial intermediation services in rural areas, which could be replicated in other areas of the country.

SO2

Increased capital and income among the rural poor in the project area through off-farm income-generating activities particularly for women and families in densely populated areas with limited land for farm expansion.

SO3

Strengthen the SRAB to carry out its mandate with respect to cooperative development.

SO4

Provide credit to meet financial requirements for agricultural inputs and draught oxen and facilitate the supply of inputs through support to local traders and cooperatives.

SO5

Relief of livestock health constraints, particularly with respect to draught animals, through provision of veterinary drugs.

SO6

Improve access of rural families to services and markets by rehabilitating and maintaining rural roads.

SO7

(BSF Component) Reduce the burden of disease in 8 woredas of the SOCODEP area

Objectives and methodology of the evaluation

Overall objectives. The main objectives of the evaluation were to: (i) assess the performance and impact of the SOCODEP project; and (ii) generate a series of findings and recommendations that would serve the International Fund for Agricultural Development (IFAD), the Government of the Ethiopia, and other donors in designing and implementing similar projects and programmes in the future. The special focus of the evaluation, as highlighted in the Approach Paper, was to be on Cooperatives Development, Rural Microfinance, Socio-economic impact, and Institutional Capacity-Building.

Evaluation methodology. The evaluation followed OE's guidelines for project evaluations1.

The evaluation team conducted a field visit to the project area and interviews with key stakeholders.  The evaluation notes the limited range of reports and other documents available through the project and partners due to the frequent moving of the project office which made it challenging to find many documents.

However, the core documents were available and provided a sufficient source of secondary data for the evaluation. As per standard OE practice, a Core Learning Partnership (CLP)2 was constituted for the evaluation, which provided critical inputs and views at key stages of the evaluation process. 

Project performance

Implementation and outputs

Cooperatives restructuring (US$0.95m, four per cent of total budget at Appraisal). Cooperatives under the former Government (the Derg) were an instrument of Marxist ideology and coercion. A great deal of work would have to be done to turn these into autonomous businesses owned and democratically controlled by their members. The number of cooperatives restructured through SOCODEP under the new legislation exceeded the Appraisal target, 200, by about 30 per cent. Credit delivery (US$7.68m, 34 per cent of total budget at Appraisal).

Credit was to be delivered to cooperatives for their own businesses (flour mills, shops, produce marketing) and through the cooperatives to the individual members (for farm inputs, income-generating activities, small-scale enterprises, and work oxen). The Commercial Bank of Ethiopia (CBE) was to be the channel for disbursement of loans to the restructured cooperatives. Overall, the disbursement of loans fell below target, at EB 18.76m by the time of closure of this project component. This is a little over US$2.0m or approximately 30 per cent of target. This low number of loans disbursed was attributable partly to project design shortcomings and partly to problems encountered during project implementation. Specific factors included: CBE's lack of experience in rural credit implementation; delays in loan processing; lack of capacity in the restructured Service Cooperatives (SC) to borrow and on lend; and defaults in loan repayment.

Promotion of off-farm income generating activities (US$0.29m, 1 per cent)

Small enterprise promotion was undertaken through a pilot project involving training at the Rural Technology Centre, Sodo, and subsequent loans to individuals setting up in business. Training was provided to 226 beneficiaries, in bamboo work, tailoring, weaving, spinning, woodwork, beekeeping and pottery. Loans for subsequent business development amounted to EB 55,456 (approximately US$8,000), three percent of the target, in total over the entire project period.  The Women Income Generating Activities (WIGAs) performed much better providing 7,600 loans (76 percent of the target) to women. 

Institutional capacity building

Capacity-building, through training and provision of physical resources, was a major aspect of the project. Explicit capacity building inputs focused on Southern Region Agricultural Bureau (US$2.80m, 12 per cent of total budget) to enable the Bureau to deliver support to the SCs and CBE (US$0.39m, two per cent) to provide effective rural finance to the SCs. But there were significant allocations also to the other Regional Government organs involved (Water, Health, and Roads), as well as to the cooperatives themselves.

As far as physical resources are concerned, the project provided large numbers of vehicles (65 cars, three buses, 300 motorcycles), as well as office accommodation, office equipment and road-building equipment. Perhaps unsurprisingly, disbursement for these items proceeded quickly, with large overdrafts on the vehicles and equipment category being highlighted in the 1997 and 1998 United Nations Office for Project Services (UNOPS) Supervision Reports.

Road construction (US$4.70m, 21 per cent)

The target at Appraisal was for the rehabilitation and heavy maintenance of a total of 700 km of rural roads, to a design standard that would permit labour-based maintenance. In the end, 122 km was constructed, to a higher standard (and consequently higher unit cost) than originally agreed.  Thus reducing the amount of construction and rehabilitation possible and requiring machine-based maintenance in the future.

Veterinary revolving drugs fund (US$1.89m, 8 per cent)

This was intended to be a complement to other project activities – specifically to help to protect the livestock which were involved in ox loans and credit for small ruminants. The first purchases of drugs under this component were delayed significantly, taking place in 2001, and resulted in only about 19 per cent of the budgeted funds being disbursed by the time of closure of the project loans.

Water supply, health and basic sanitation (US$4.02m, 18 per cent) – Belgium Survival Fund (BSF) funded

Despite the assessment of a pre-Appraisal mission in 1993 that "…water supply is not a major problem in many parts of the project area … and in areas where needs are pronounced several NGOs are providing assistance", a formulation mission addressing this and the wider issues of health and sanitation was sent in 1996 and the component commenced in 1999.  This component has performed well meeting its targets related to health services (through construction, rehabilitation, equipping and training) and construction of water supply points in the 8 component Woredas.

Assessment relevance, effectiveness and efficiency

  • Relevance. SOCODEP addressed some real needs of the rural poor in southern Ethiopia (need for credit, need for improved market access, need for better health services and environmental health) and it was consistent with the government's regionalization programme. At the time of design the rhetoric of the new Government and the framing of the new legislation made working through service cooperatives attractive and promising. However the use of service cooperatives as the channel for services to the poor turned out to be a significant design weakness.   Similarly, the project had limited choices for partners with rural finance experience and should have been addressed with greater seriousness.
  • Effectiveness. The Project achieved some, but not all of its objectives. The Specific Objectives (SO) 1, 3, and 5 were mostly not met. These include SO1, providing a model for developing Ethiopian cooperatives under the new legislation, SO3 strengthening the Southern Regional Agricultural Bureau (SRAB) to carry out its mandate with respect to cooperative development, and SO5 relief of livestock health constraints, particularly with respect to draught animals, through provision of veterinary drugs.  Specific Objectives 2, 4, 6 and 7 were met to varying degrees and include SO2 increasing capital and income among the rural poor in the project area through off-farm income-generating activities, SO4 providing credit to meet financial requirements for agricultural inputs and draught oxen and facilitate the supply of inputs through support to local traders and cooperatives, SO6 improving access of rural families to services and markets by rehabilitating and maintaining rural roads, and SO7 (BSF Component) reducing the burden of disease in eight woredas of the SOCODEP area.
  • Efficiency. Given the difficult operating environment in Ethiopia the project could have been more efficient if it had been more realistic and less ambitious.  The vast geographic area, poor infrastructure and communication meant that the projected resources were not optimally used.  Also, the overall emphasis on the Project's numerical outputs (i.e. exceeding the number of restructured cooperatives) rather than the quality of those outputs, has resulted in a significant amount of wasted resources.  As a result, many of the project's activities were diluted to the extent that they failed to achieve their intended outcomes.  Efficiency would have been greatly enhanced if the project design had concentrated on a smaller project area, involving a smaller number of SCs, and provided greater intensity of effort and resources.

Project impact by component 

Cooperatives restructuring

The number of cooperatives restructured through SOCODEP under the new legislation exceeded the Appraisal target by about 30 per cent. However, the quality of the restructuring and re-orientation undertaken has not translated into an effective cooperatives sector in the project areas. The cooperatives encountered in this evaluation, and many others for which data was obtained, are suffering significant problems of poor management, misappropriation of funds and de facto bankruptcy. The findings of the evaluation in this regard are not new. To quote the internal review of the project carried out in 1999: "…with such limited capital, unprofitable business, increased misappropriation, weak management, absence of member education, and poor participation of members, the conceptual framework of promoting economically viable cooperatives … could not be achieved."3

Three years later, at the time of closure of the main loans, the situation had not changed significantly: "…the state of some cooperatives restructured under the project appear to be weak and in the worst case on the verge of bankruptcy … most restructured cooperatives critically need the support of the project …"4 . In short, although SOCODEP re-registered about 267 cooperatives under the new law, it failed to re-orient the officers and members of those organisations, and leave behind a strong and sustainable cooperatives sector.

Credit delivery

The disbursement of loans achieved approximately 30 per cent of its target. Although many cooperatives and individuals received and benefited from loans, many of the target populations did not. Loans for cooperatives' own businesses were generally unsuccessful. Because the cooperatives found that they could not compete with the leaner and fitter private sector, the majority of these ventures failed, and the cooperatives failed to repay their loans.  Loans to individuals were generally much more successful, with ox loans being particularly popular (although the required ox insurance was not popular, and it failed to pay out in the majority of cases of mortality). Loans to women for the multiplication and fattening of small ruminants were also especially valued, and repayment rates were high. In general, repayment of loans was better in the western part of the project area than in the east and south-east.

CBE learned in the course of the project that provision of rural microfinance is a very different activity to the provision of commercial, urban, loans. It found it was ill-equipped to manage this project component, and it withdrew in 2002. The transfer of this component to the Region's only Omo Micro Finance Institution (OMFI) in the last few months before closure of project loans was the only solution to a crisis.  Because of the difficulties experienced by CBE, and the poor performance of many of the cooperatives, the opportunities for cooperatives or individuals to access further loans has been curtailed. Real benefits enjoyed by individuals made a difference for a time, but they may not have brought about lasting change. 

Promotion of off-farm income generating activities

Small enterprise promotion was undertaken through a pilot project involving training at the Rural Technology Centre, Sodo, and subsequent loans to individuals setting up in business. The impact of this activity on a few individuals was significant; however, the extent of that impact was negligible.

Institutional capacity building

A great deal of training was carried out (estimates suggest that nearly 57,000 individuals received some form of training). Individuals have benefited to some extent, but the ability of the project stakeholder institutions to perform more effectively, during and subsequent to the project, is questionable. A lasting impact on the institutions involved has been severely undermined by the capacity buildings component's ad hoc nature 5 and by the frequent Government restructuring and re-deployment of personnel.

SOCODEP's capacity-building efforts largely ignored issues of attitude change, organizational reform, and policy dialogue. These were major omissions.

Road construction

The roads built are of good quality and have benefited those people living near them.  The Roads Authority has been left with a useful pool of heavy equipment for future road construction work. As far as SOCODEP outcomes are concerned however, the aim of this aspect of the project was to extend access to markets for cooperatives and their members. As with other project components, a few people have benefited from the component, and its main shortcoming has been its limited reach. 

Veterinary revolving drugs fund

The component was able to provide temporary relief for the few participants who were able to access the fund. At the time of the evaluation, drug shortages were being felt, and animal health problems were reported to be increasing, especially in those areas where tyrpanomiasis is prevalent.

Water supply, health and basic sanitation (US$4.02m, 18 per cent) – BSF funded

In some respects the WSBHS component has delivered the most visible and potentially successful aspects of the entire project, with its most effective aspects being the strengthening of health services (through construction, rehabilitation, equipping and training) and construction of water supply points in the eight component Woredas.  However, the functional sustainability of water points, and the utilisation of improved excreta disposal facilities at household level, are areas of concern. In particular, user fees for water point maintenance are too low, and access to spare parts for hand pumps is extremely problematic.

Integration across components

SOCODEP was a multi-component project, involving at least 7 Regional Government organs 6, three financial institutions7 , and numerous other stakeholders 8. There were important potential synergies between components and stakeholders. The cooperatives were fundamental. Their capacity to function as viable democratic business entities would determine the extent to which all the individual project beneficiaries would gain. In the event, their lack of effective re-orientation and capacity-building has limited the extent to which individual men and women have enjoyed the benefits of cooperative membership. Credit disbursement depended on the sound functioning of both the cooperatives and CBE, as the financial intermediary. Neither the cooperatives nor CBE were effective in channelling loans to individual members.

Institutional strengthening of Government organs was greatly limited in its impact, because of the unsystematic and incomplete way in which it was done, and because of frequent Government restructuring. Supply of veterinary drugs, construction of roads, enhancement of health services, and construction of water supplies and sanitation facilities were carried out as separate activities, and in the case of the first two, very little was actually achieved. The potential synergies were not realised, and the project as a whole could not be described as integrated.

SOCODEP Assessment

Table 2 summarizes the ratings attributed to the project by the evaluation, with the average ratings given to project evaluated in 2005 for comparison. On many criteria SOCODEP scored one point under the 2005 average. Impact on physical, financial and human assets is comparable with 2005 averages, but the performance of the Southern Region Government is assessed as particularly below the 2005 average. Because of fundamental weaknesses in project implementation, caused partly by unrealistic design, partly by insufficiently decisive management, partly by insufficient political commitment, and partly by externalities including frequent Government restructuring and reshuffling, SOCODEP is assessed overall by this evaluation as ‘Moderately  unsuccessful' (rating of 3).

Table ES2. SOCODEP project performance: rating summary

Evaluation Criteria SOCODEP Rating

 

2005 Project Evaluations Average
Project performance
Relevance 5 5
Effectiveness 3 4
Efficiency 3 4
Partner performance
IFAD 3 4
Government 3 4
BSF 4 -
UNOPS 4 4
Project impact
Agriculture Productivity 3 -
Physical and Financial Assets 3 4
Human Assets 4 4
Institutions and Services 2 -
Social Capital and Empowerment 2 4
Food Security 3 4
Environment and Common Resource Base 4 4
Markets 2 -
Overall impact 3 -
Sustainability 2 4
Innovation, Replicability and Scaling-Up 3 4
Overall assessment 3 -

Source: The Evaluation Mission 2007

Conclusions, recommendations and key issues for the future

Conclusions 

As with most multi-component projects, the project performance and impacts of SOCODEP have been mixed. There seems little doubt that the Project has had a net benefit to the Region, to individuals in Government, and to some of the target beneficiaries. The key questions relate to understanding how the benefits could have been greater, and what lessons can be learned for present and future projects in Ethiopia and further a field.

Context. SOCODEP was one of the first significant internationally funded interventions in Ethiopia following the fall of the former Marxist-Leninist regime (the Derg) in May 1991. The Project aimed to respond to the then new legislation concerning Cooperatives, which ostensibly set out a means of turning the former Government-imposed and politically-dominated Producer Cooperatives of the Derg into farmer-owned viable business entities serving their members' interests. In particular, it aimed to make rural finance (specifically micro-credit) available to so-called Service Cooperatives and their members. 

Design. The Project aim, as described in paragraph 31, was an imaginative attempt to respond to the then new legislation concerning Cooperatives.  However, this was an ambitious and in hindsight an unrealistic goal.  A number of factors contributed to the difficulties faced by the SOCODEP. First, the Region's size, diversity, poor infrastructure and poverty posed numerous development challenges.  Second, the project underestimated the obstacles to and rate of beneficial change and the capacity of the government for implementation in the post Derg period. 

Quality of project delivery. SOCODEP concentrated on delivery of numerical outputs, such as  cooperatives restructured, credit disbursed, trainings delivered, drugs purchased, ilometres of road constructed, water points built, and so on. Insufficient emphasis was placed on the quality of these outputs. For example, insufficient consideration was given to the intensity and duration of activities required to achieve the desired quality standards. In particular, the human factor of individual and group (community, cooperative, institution) attitudes were addressed minimally. For example, the design was too optimistic about the speed with which the former model of cooperatives, centrally controlled by the government, could be turned around into a member-owned and member-controlled viable business model. To turn around a failing, politically-established cooperative to become a viable business serving its members, or to bring about community ownership and management of a water point demanded a great deal of attention to quality of the investment, not just numbers.

However, some of the activities such as upgrading of health facilities and training of staff and community health workers performed better.

Relevance, effectiveness, efficiency. The project objectives and activities were relevant to needs of the rural poor in southern Ethiopia (e.g., in terms of the need for credit, improved market access, better health services and environmental health) and were consistent with the government's regionalization programme. The Project was moderately ineffective achieving some, but not all of its objectives. Similarly, the project was moderately inefficient. Given the difficult operating environment in Ethiopia, the project could have been more efficient if it had been more realistic and had less ambitious objectives and coverage. The vast geographic area, poor infrastructure and communication meant that the projected resources were not optimally used.

Sustainability. SOCODEP's benefits are unlikely to continue, partly due to the lack of a defined exit strategy. Moreover, institutional sustainability is limited and on-going access to credit, and water supply is not assured.

Innovation. At the time of design, SOCODEP's focus on cooperatives and credit represented a response to the apparent liberalization of national politics and economics, and to the change in cooperatives legislation. Unfortunately, the country context changed rapidly and the design became less relevant given the new context. Despite the positive efforts at the Mid-term Review (MTR), the design adjustments were not adequate given the changing realities. On another issue, the BSF component introduced an effective monitoring and evaluation system, which however was not integrated into the other project components. As the project was overstretched and its components were not integrated, SOCODEP offered little opportunity for the learning being generated to be feed back into the project. Hence, the evaluation considered the project to be moderately unsuccessful in terms of innovations, replicability and upscaling.

Policy dialogue. SOCODEP was largely responsive to policy changes and government-led restructuring. There is little evidence however of the project contributing to IFAD's effective engagement in policy dialogue in the country.

Participation. Probably the single greatest assurance of sustainability at the level of households and communities is through real commitment to beneficiary participation. However, the evaluation found that ensuring beneficiary participation in an area with a weak tradition of participation is challenging and requires greater commitment in terms of time and resources. As such, approaches which build on existing social capital (i.e., using indigenous Community Based Organizations), rather than working through structures imposed from above and outside the beneficiary communities, are most likely to succeed in the short and long term.

Integration. This evaluation report has highlighted at a number of points the lack of integration between the numerous stakeholders and components within SOCODEP. Although integration is not easy, particularly given the restructuring of government organs and redeployment of personnel, it is the only way to create synergies which can maximize the impact of limited budgets.

Management. The management model used by SOCODEP limited its effectiveness and responsiveness to rapid contextual changes that occurred during project implementation.  The Project Coordinator, attempted to harmonize and synchronize the work of several Government organs and other stakeholders over whom he has no real authority. And support from IFAD through supervision mission mounted annually by the designated cooperating institution the United Nations Office for Project Services (UNOPS) was insufficient. However, it should be recognized that during the time of SOCODEP, IFAD did not have modalities such as direct supervision or field presence to support project implementation. Although, IFAD was responsive in using the tools it had at the time, for example by undertaking an early and useful MTR and facilitating the inclusion during implementation of the important BSF component. 

Weak linkages between partners during implementation. The linkages between the key stakeholders have mainly been achieved by the efforts of the Project Coordinators (four in total, at various stages of the Project), IFAD's Country Programme Manager (CPM), and UNOPS. The Project Coordinator's role was challenging because of his inherent lack of authority, and the CPM's role is distant from the day-to-day project management issues to do more than provide general support and guidance. Also, annual visits by a cooperating institution are insufficient to rescue an under-performing project. Consequently, the effectiveness of the partnership was limited. Partnership with the private sector was not a viable option in the early years of the Project, but this option could have been pursued as implementation progressed. The extent to which partnerships outside of the project were developed by the implementing stakeholders is limited.

Recommendations

Design. It is recommended that consideration be given to interventions which are far more focused in terms of numbers of beneficiaries to be reached and geographic coverage, within the overall framework of IFAD's targeting policy. This would ensure greater synergies across activities and ultimately deeper impact on rural poverty.  Similarly, the project duration should be long enough to achieve the desired results and in particular take into account the time needed to implement attitude and cultural changes. Project management structures should be kept simple to ensure the integration and harmonization among different implementing agencies.

Quality of project delivery. It is recommended that greater attention be given in future project design and implementation to country context issues, and the identification of indicators of quality, and actions necessary to achieve real and lasting impact, alongside those relating to numerical outputs.

Policy dialogue. It is recommended that more explicit attempts be made to engage in policy dialogue with Government and other development actors, where appropriate and required involving a wider range of national and international specialists, rather than just IFAD and cooperating institution staff.

Participation. It is recommended that future IFAD-funded projects and programmes in Ethiopia pay more attention to people's participation, especially as it has not been a tradition of development practice in Ethiopia in the past.

Integration. In future multi-component projects, it is recommended that greater attention be paid to the linkages between the components and between those agencies responsible for delivering them. The evaluation team is in favour of projects which involve multiple components addressing the diverse needs of target populations – but the difficulties of integrating such efforts should be carefully considered.

Management. Management is more than coordination or supervision, and it should be addressed with greater rigour in future projects and programmes particularly in challenging contexts as found in SOCODEP. Projects need to be much more decisively managed. It is recommended that new approaches be explored, either through IFAD itself taking a more hands-on role during execution, facilitated by the Fund's field presence officer, which allow closer monitoring and follow-up to implementation.

Role of the field presence officer. The field presence officer can, among other tasks, provide implementation support to IFAD-funded operations and has the potential to enhance partnerships and policy dialogue in Ethiopia. Hence, the country presence should be further strengthened, so that it can play a greater role in enhancing IFAD's development effectiveness in Ethiopia.

Questions for the forthcoming country programme evaluation (CPE)

In view of the CPE for Ethiopia which is planned for 2007, a number of key questions arise from the present evaluation, which should be addressed in that context. These are:

Have the present project identification, formulation and appraisal processes encouraged the setting of unrealistic targets and spreading project of activities too thinly? If so, how can these tendencies be avoided in future?;

Are the existing project identification, formulation and appraisal processes sufficiently participative, in a country in which participation is not a strong tradition?

If not, can they be made more so?

To what extent have the detailed capacities of project stakeholder institutions been routinely assessed at the formulation stage, in order to design appropriate capacity-building programmes? What improvements can be made to this process?

What has been the impact of the BSF contributions in Ethiopia?  How can the partnership be enhanced in future activities?

How can the present model of project management and supervision be modified to create a significantly greater degree of the Programme Coordination Unit (PCU) authority and effectiveness, without de-coupling projects from the implementing institutions? Should separate project management structures be set up, perhaps using consulting firms, or can the existing PCU framework be made more effective? How could the field presence officer be more effective? What model best fits the Ethiopian context? Should clearer guidelines be framed, setting out responsibilities for taking actions on supervision and MTR recommendations? Where does the buck stop in terms of project management?; and

In view of the weak performance of Monitoring and Evaluation (M&E) in some Ethiopian projects, how can a monitoring culture be encouraged, and how can manageable frameworks be developed and implemented for monitoring of project performance, reflecting both quantitative and qualitative achievements?


1/ This included assessing the project against internationally recognized evaluation criteria, namely: (i) project performance, including relevance, effectiveness and efficiency; (ii) impact on rural poverty; and (iii) performance of partners involved in the project, including IFAD, government institutions, and others.

2/ Members of the partnership included: Ministry of Finance and Economic Development, Bureau of Finance and Economic Development, Bureau of Agriculture and rural Development, Southern Nations, Nationalities and Peoples Regional State (SNNPRS), Department of Cooperatives, SNNPRS, Rural Women's Affairs Team, SNNPRS, Bureau of Health, SNNPRS, Bureau of Water Resources, SNNPRS, Planning and Programme Department, Rural Roads Authority, SNNPRS, Sodo Rural Technology Centre, Commercial Bank of Ethiopia, Omo Microfinance Institution, the Association of Ethiopian Microfinance Institutions, Former SOCODEP Project Director, field presence officer, and the IFAD Country Programme Manager.

3/ SNNPRS Cooperative Office (1999) Comprehensive Analaysis of Current Situation and Requirements of Cooperatives in SOCODEP Areas, April 1999, Awassa.

4/ UNOPS Supervision Report, April 2002.

5/ The evaluation team could find no evidence of any systematic training needs assessment.

6/ Those responsible for Agriculture, Cooperatives, Veterinary Services, Water, Health, Finance & Planning, and Roads, organised in different ways at different times.

7/ CBE, OMFI, and Ethiopian Insurance Corporation.

8/ Cooperatives and their members, contractors, and consultants.

 

LANGUAGES: English

Apuseni development project (2008)

  
September 2008

Completion Evaluation

Introduction

The economy

The Apuseni Development Project (ADP) was designed and became effective at a time when Romania was still experiencing the aftershocks of the breakup of communism. The macroeconomic situation was volatile, with hyperinflation, negative real interest rates, commercial lending rates over 100 per cent, and an economy and financial sector still dominated by state entities and state intervention. In 2003, 38 per cent of the rural population lived in poverty, compared with 14 per cent in urban areas, with rural per capita incomes in the order of US$330 per annum.

Only in the last five years has economic reform, in the form of liberalization, privatization and deregulation, been systematically pursued as part of an overall coherent reform agenda.

Agriculture and rural development

Land privatization was one of first reforms to be carried out in the early 1990s as the country moved towards the adoption of a market economy. Combined with the collapse of markets, and a lack of working and investment capital, the resultant restructuring of the farm sector led to a reversion to low input, subsistence farming. The share of agriculture in a declining gross domestic product during the 1990s fell from about 20 per cent in 1990 to 12 per cent in 2002, while the proportion of agriculture in employment rose from 30 per cent to 40 per cent in the same period.

Rural finance

Because of the legacy of an extensive cooperative network, there was a system of financial intermediaries in rural areas after the 1989 revolution. However, agricultural credit faced major problems during the 1990s as a result of land reform and financial sector restructuring. Most agricultural credit was channeled as highly subsidized directed credit by the National Bank of Romania (NBR) through the Agricultural Bank. This inhibited the entry of other institutions. As loans were backed by state guarantees, there was little insistence on creditworthiness examination and collateral. Financial intermediation, i.e. the transformation of savings into credit, functioned poorly. In 1997, the National Bank ceased its involvement in the provision of directed credit and introduced basic changes in credit policy.

The Ministry of Finance announced a phased reduction over 3 to 5 years of all directed credit; credit allocation was made the responsibility of the banks, not the state; and defaulting clients, whether parastatal or private, were barred from credit until the default was cleared.

Savings mobilization and credit capacities of the credit cooperatives were limited by cooperative regulation. Their union, CentroCoop, had failed to function as an instrument for effective financial market integration; and BankCoop was reluctant to grant loans to the credit cooperatives. At the same time, when saving was made attractive by NBR after a prolonged period of negative real returns on savings, cooperative members absconded in large numbers and shifted their savings to commercial banks. Besides, cooperatives were viewed as a relic of the communist era and had a negative connotation, and BankCoop which was intended by the IFAD project design to be the main conduit for the IFAD loan, was closed in 2000.

The project

The ADP, funded on intermediate lending terms, is the only IFAD-funded project in Romania. The project was identified in 1995 and appraised during 1997. The project loan was approved by the IFAD Executive Board in September 1998 and became effective in November 1999.  The closing date for the project is 31 December 2006, following two extensions totaling two years.

Total project costs were planned to be US$34.11 million. The IFAD loan consisted of US$16.46 million comprising a revolving credit fund of US$16.0 million, and US$0.46 million for institutional support and project administration. A variety of co-financing was anticipated from the participating banks, beneficiary contributions, and donor co-financiers.

Most of this co-financing did not materialize, though the Federal Republic of Germany provided technical assistance, but not in the form of co-financing. A co-financier to provide grants linked to the IFAD loans was not found. As of 31 July 2006, IFAD loan disbursement at SDR 10.1 million (US$14.5 million) had reached 82 per cent of the total loan amount.  Loan repayment from the Government of Romania (GOR) to IFAD is currently ongoing and will be completed by December 2018.

Project objectives and components

Despite its title, the project is essentially a credit project. The overall objective was to:

"…improve and stabilize the economic environment of the rural communities of the Apusenis through the promotion and credit-funding of on- and off-farm enterprises and the provision of rural development services."1

The project had two components:

  • Rural credit, the immediate objective of which was "to establish a sustainable mechanism for the provision of capital and development credit for financing of rural enterprises, utilizing a revolving credit fund; and
  • Institutional support the immediate objective of which was "to strengthen rural development services".

The evaluation

The evaluation aimed to assess the project performance in terms of relevance, effectiveness and efficiency, impact, sustainability and innovations.  An evaluation mission was conducted in-country from 30 September to 20 October 2006.  The evaluation mission held discussions with officials of the Ministries of Agriculture and Finance, with staff of the World Bank, the Commercial Bank of Romania and the Carpatica Commercial Bank. The mission visited five judets (administrative districts), met with present and former staff of the Department of Agriculture and Rural Development, with the branch staff of the participating banks, and with fourteen borrowers. In an ancillary study of backward linkages in the dairy sector, thirteen borrowers and twenty non-borrowers were interviewed by a local consultant.

Project performance

Project rationale

The Appraisal Report implicitly assumed that lack of working and investment capital was a key constraint on rural productivity and livelihood enhancement. The ADP established a revolving credit fund (RCF) to provide development and seasonal credit for a range of improved on- and off-farm primary production and service activities.

Processing units for agricultural and livestock products were also included to add value and create local employment.

Disbursement was to be through local banks and cooperative institutions. The project would also support governmental institutions to promote the credit line and to provide technical and business support to beneficiaries.

The Appraisal Report referred positively to "the experience which IFAD had garnered in transitional economies …[whereby] … small-scale credit and rural financial services …provide the most appropriate niche for IFAD assistance; and in which it will have the most immediate impact".

Project implementation

For the first half of its life, the project was essentially non-performing, though there were vigorous efforts to stimulate it into life. The project became effective in November 1999, but by the time of the first project supervision mission, undertaken by the United Nations Office for Project Services (UNOPS) one year later, no loans had been disbursed. The second UNOPS supervision mission stated that the project remained "in a state of crisis" as only three loans had been approved 20 months after loan effectiveness and twelve months after the signing of the Subsidiary Financing Agreement with the sole participating credit institution, that is, the Commercial Bank of Romania. Three years after the project launch, there was a noticeable improvement in loan uptake, though at the two-thirds point through the project, only 16 per cent of the IFAD loan had been disbursed.

The design process

Some of the responsibility for this underperformance must be laid at the door of the design process. Following usual practice, the design process consisted of three stages (i) inception/identification, (ii) formulation/preparation, and (iii) appraisal. It could be argued that project start-up was also part of the basic design process in this case since it carried out some design adjustments. The design system used at IFAD places weight on ensuring continuity between the different stages. An effort is usually made to employ some of the same consultants from one stage to the next. In the case of the ADP, this continuity was maintained through the deployment of the team leader/credit specialist and through all four design stages.

The issue in this arrangement relates to the operation of checks and balances vis a vis the assessment of project design parameters and assumptions, through the project design cycle. In this case, the "appraisal" function was about "confirmation" of design features, rather than a "critical but constructive evaluation" of assumptions and parameters. The system hence emphasized stability in the evolution of project design, and was about "confirmation" rather than "objective evaluation" at appraisal.

The main sources of design review at IFAD are the Technical Review and the Operational Strategy Committees (TRC and OSC). However, the checks of these committees are carried out at the same time at the formulation stage. In this case, these checks were not followed through from formulation into appraisal, with the result that there were no changes in project design from one stage to the other.

Project assumptions relating to the macroeconomic and institutional environment were consistently overoptimistic, and fundamental design issues identified both by the TRC and the OSC, relating to effective demand by poor borrowers for project loans and the difficulties of identifying participating credit institutions, were not addressed. These aspects should have been the subject of risk assessment but were not.

And at the general level, the logframe was inadequately specified and was of minimal use as a managing and monitoring instrument.

Changes during implementation

As a result of the early failure of the credit component to disburse funds, mainly due to issues related to credit demand, credit delivery, communication and technical assistance (TA) for capacity building, radical changes were made to project design with the agreement of the GOR and IFAD. These affected operating parameters (terms and conditions) and implementation modalities. While the central objective of rural poverty reduction was retained, the geographical coverage was expanded three times, the life span of the project extended twice, and the targeting and impact approach was fundamentally altered. The geographical coverage was extended from six to fifteen judets, and prioritizing smaller borrowers was abandoned alongside the acceptance of heavily securitized lending approaches which prevented the poor from borrowing. Loan use was liberalized, bank co-financing dropped, borrower contributions reduced, and the life span of the project extended.

According to the project mid-term review carried out in 2003, the effect of the changes was to "seriously erode the smallholder focus".  Having inherited a credit project designed during macro-economic instability and with no clauses in the loan agreement related to lending targets and lending policies for enhancing access of smallholders to credit, the efforts of the IFAD and UNOPS missions in 2000 and 2001 were directed towards making the credit line operational rather than focusing on the issues related to smallholder focus. Hence, the credit line component started off as a general credit line rather than a project addressing the core concerns of IFAD related to smallholders' access to credit.

At the end of the project, disbursement had improved to the point where the project loan will be fully disbursed but with a lending portfolio which has gone mostly to the non-poor.

Relevance

The project's development objective was highly relevant to the needs of rural communities in mountainous areas. At the time of design, two fifths of the rural population was living below the poverty line of US$3.30 per day, and the mountainous areas were judged to be particularly disadvantaged. Yet, the project presented faults at the level of immediate objectives and operational design. The rationale for involving the poor in a formal credit arrangement was not spelt out and the target groups' "effective" demand was uncertain. The modalities and conduits for dispensing credit to small borrowers were quickly proved not to be viable. Over time, the project design has become more relevant to an environment which is macro-economically and institutionally more stable, and government rural development policies which are aligning to EU membership, and large scale enterprise development. The introduction of major design changes in the project in 2003 addressed in part the original design issues and reinstated the relevance of the project.

Effectiveness

The effectiveness of the project is mixed. It has eventually been successful in establishing a system for disbursing its loans, which is a significant achievement, although it has not been able to do so to the target group of small borrowers. Major modifications were instituted in project design with the Loan Amendment of 2003; these were driven by an impact hypothesis which was not adequately analyzed and explained, due to lack in time and by pressures for disbursement. It should be noted that the project continued having difficulty in disbursing its loans even after these modifications. After six and a half years, just over 1,000 loans have been disbursed, a figure which constitutes a small portfolio. Thus, the project has spread itself thinly through the 15 judets.

Given the paucity of loans in a given community, the effectiveness of the project in terms of its original intention of transforming poor communities in marginal areas is low. However, for most borrowers, the loans received appear to have been effective in terms of additional income and growth of assets, less so in terms of employment generation.

Efficiency

The slow start-up of the project generated efficiency losses through financial and human resources being tied down in a non-performing use. As far as the IFAD funds are concerned, the opportunity costs were limited since the project quickly moved to an arrangement whereby IFAD reimbursed the Government after end-loan pre-financing by the participating credit institution. The direct operational costs provided by the project for administering the loan are relatively low at an average of US$250 per loan. However, this does not take account of the operational and overhead costs incurred by participating banks and the Ministry of Agriculture judet agencies. The overall efficiency concerning the involvement of the Commercial Bank of Romania as measured by its IFAD lending relative to its overall outreach is minimal as BCR has five million clients out of which less than 1,000 are IFAD borrowers.

In contrast to the low efficiency in terms of the transaction costs of the banks and the involvement of other agencies, efficiency in terms of returns to investment as proxy-indicated by the low loan arrears ratio, has been excellent. The project has shown that rural and agricultural lending is feasible, and that the risks involved can be managed.

Performance of IFAD

IFAD underestimated the difficulties of operating in Romania, and overestimated its transition country experience. Contextual difficulties were compounded by lack of in-country presence and country analysis capabilities. For these reasons, IFAD was a weak player in the foreign assistance field. Weaknesses manifested themselves at the design stage where the process appears to have been mechanistic, with inadequate attention to macroeconomic, financial sector, and institutional aspects, and associated risk assessment. The quality assurance system of checks and balances operated through TRC and OSC were not fully utilized. However, to its credit, once it was clear that the project had major implementation problems IFAD moved pro-actively. The rationale for moving from a pro-poor to a growth-based project borrower strategy was not adequately analyzed.

During 2002, the IFAD Country Programme Manager (CPM), together with the Deutsche Gesellschaft fur Technische Zusammenarbeit (GTZ), seriously considered a link up with the Kreditanstalt fur Wiederaufbau (KfW) small and medium enterprise lending activity. However, the Ministry of Public Finance (MOPF) disagreed and the option was dropped. Thus, the opportunity to move to a more flexible lending strategy was lost. Subsequently, with a change of CPM, IFAD, along with its cooperating institution (UNOPS) agreed to the full collateralization of loans under the project. Little emphasis was put on credit technologies that would comply with modern micro-finance through commercial banks, such as cash flow lending. While at the same time, KfW continued to establish new cash-flow-based lending technologies into emerging Romanian banks.

Thus, IFAD abandoned the pursuit of a lending strategy which would accommodate poor borrowers. Because of the sole participating bank's lending practices, IFAD agreed to the full collateralization and securitization of loans, a barrier to poor borrowers with few assets and little income. Little emphasis was placed on credit technologies which would comply with modern lending practice to poor borrowers. However, once the new lending strategy was adopted, it was pursued with vigor by IFAD, UNOPS and MOPF. IFAD scores well on communication and information sharing, coordination, flexibility, and supportiveness, but only within the boundaries of the project.

However, four changes of CPM during the life time of the project has not been a good basis to provide for continuity and coherence. Of particular concern is the fact that different CPMs had different views on how the project should be reshaped, reflecting a lack of guidance and strategy.

Performance of Government

The National Plan for Agriculture and Rural Development prepared in 1999 did not provide a good back-drop for ADP. The Plan, which was heavily oriented to Romania joining the EU, contained no policy and approach for a rural finance system, which addressed the poor or otherwise. Given the transitory state of government institutions, government was unable to play an active role in project design. Once the project deficiencies became clear, the MOPF was galvanized into action, and played a crucial role in steering the project onto a track which would increase disbursement. The transfer of responsibilities of the Project Implementation Unit into the Project Coordination Unit in the Ministry of Public Finance was an unprecedented measure to resolve the issues encountered by the project. The Government scores well in relation to its pro-active coordination at the country and judet level during the course of the project crisis. However, its policy backstopping in relation to rural finance was weak, and its relationship with GTZ, a main partner, was at times tense. After a slow start, the Ministry of Agriculture played a key role in promoting and executing the IFAD loan product, though performance varied widely between judets.

Performance of UNOPS

UNOPS was the cooperating institution, responsible for loan administration and supervision. Because of non-performance by the project, its focus of attention became project redesign.  Since supervision missions are provided only once a year, its ability to play a decisive role in this area was limited. While it was a good team player and offered constructive advice, UNOPS should, likewise IFAD, have done more to articulate and analyze the shift from a pro-poor borrower to a growth based lending strategy. Given the early difficulties of the credit component, it is questionable that a rural finance or credit specialist was not mobilized until four years after project effectiveness. UNOPS assisted in tackling many of these issues proactively with a series of revisions (which required numerous loan and SFA amendments), including the streamlining of project implementation modalities, the elimination of CARD as a project agency, and the eventual inclusion of additional participating banks.

Performance of participating credit institutions

The Commercial Bank of Romania is the original, and for five years, the sole participating financial institution in the ADP. Being the largest commercial bank in the country, its reasons for participating in the project are somewhat unclear, and its approach towards rural lending somewhat ambivalent judging by the declining trend in numbers of IFAD loans disbursed over time, although the current performance may have been influenced by its ongoing privatisation process and review of all ongoing programmes being carried out in this regard. Until very recently, BCR was a state commercial bank, operating along conservative lines, influenced by a combination of National Bank of Romania regulations, and its own traditional banking culture. BCR applied a fully collateral- and guarantee-based approach to the IFAD loan product, which, while in line with NBR regulations at that time, was a barrier to rural borrowers both poor and non-poor. Nevertheless, BCR deserves commendation for having been one of the key players in enabling the ADP to happen at all, and in working with IFAD and Government in the early years of the project's life to overcome a number of difficulties which impeded implementation.

The Carpatica Commercial Bank joined ADP in 2005. BCC is a young, much smaller bank, with a more innovative and flexible approach to small-scale lending, placing more emphasis on cash flow based lending and loan appraisal. For the moment, it has taken up the slack in the declining IFAD lending portfolio and is helping to keep up the overall level of disbursement. BCC has been a welcome arrival to the ADP.

Partnership performance and organizational incentives

Relationships between partners were characterised at times by lack of consensus and unclear partnership arrangements. Most fundamentally, the shift from a pro-poor to a growth-based lending strategy was not properly articulated, and was a failure of partnership responsibility. Ultimately, the partnership will have fully disbursed the project, but to a certain cost to its development and immediate objectives.

Project impact

Agricultural productivity and beneficiary analysis

There is strong evidence of improved productivity for the non-poor individual and corporate borrowers who now constitute the primary beneficiaries of IFAD loans. Many of these are early movers taking advantage of the post- communist era vacuum of appropriately-scaled rural technology for a market economy. The Project Impact Assessment exercise, undertaken by IFAD in 2006, indicated that the financial results of all farms were positive, and that borrowers had succeeded in obtaining improved gross margins. This finding is corroborated by the Backward Linkages Survey commissioned by IFAD's Office of Evaluation, by the separate program of interviews carried out by the evaluation mission, and circumstantially by the very low arrears rates associated with IFAD loans.

The spread of benefits is narrowly confined. ADP has impacted on about 1,000 non-poor borrowers, the primary beneficiaries who will have most to gain in term of the ability to build household physical, financial and social assets. However, the investment activities of the primary borrowers has created around 1,300 jobs which are mostly taken up by the unemployed, who are likely to be asset and income poor. In addition to this group of poor secondary beneficiaries, there is a group of tertiary beneficiaries in the dairy sector. These are the milk suppliers to two milk-processing plants who have taken out substantial IFAD loans. These processors have provided an assured market for about 700 non-borrower milk producers for whom an estimated 1,050 further jobs have been ensured or created. About half of these jobs are in households owning less than five cattle, who are categorized as poor. Thus, about half of these jobs would be for family labor in poor families. Thus in total, a range of 1,900 to 2,300 jobs have been created for the poor, compared with an original target for poor beneficiaries of 6,000 – 6,500.

Physical assets

Because of the revised project strategy, only one third of the number of poor beneficiaries originally targeted have benefited from the project in terms of income or employment generation. Data to assess whether this has allowed these poor benefiting from the project to build their assets is limited. For the unemployed who take up jobs and whom it is assumed to be poor, it is likely that their wages will be used for consumption. For the (tertiary) non-borrower beneficiaries involved in producing milk for the processors, income generated would be used for consumption (clothing, health and schooling costs, etc.), but there is a possibility that some may be able to invest in asset building.

Financial assets

Again, because the project shifted its targeting strategy to the non-poor, it has had limited impact on the financial assets of the poor. Those poor who have benefited indirectly from the project are the unemployed who are unlikely to have surplus income after taking care of basic consumption needs, or small milk producers who are non-borrowers. Some of the poorer non-borrowers may be able to accumulate to invest. Neither of the two groups has fully benefited from financial services for the poor.

Human assets

The main impact of ADP with respect to human assets (nutrition, health, knowledge) has been in terms of building up knowledge and exposure of formal banking systems, norms and procedures by non-poor borrowers, and an exposure of banks to the need to develop products geared towards borrowers seeking small amounts of credit. It is not obvious that the project developed skills in the participating credit institutions in creating products that meet the needs of a new client base. A limited group of non-poor borrowers, on the other hand, has developed knowledge of the procedures involved in formally accessing credit. Training of banks staff, government agencies and borrowers has also taken place.

Social capital

As the project shifted its targeting strategy to the non-poor, it has had little impact on the empowerment of the poor.

Institutions and services

In a development sense, the project has been effective in providing access for the first time to borrowers who were hitherto not integrated into rural financial markets. This is an important achievement. However, because of the revised project orientation, when measured against improving the enabling environment for the poor, little has been achieved, most crucially with pro-poor credit terms and conditions.

Markets

In the dairy sector, IFAD loans for both individuals and companies have played a role in helping to restructure the dairy industry in those judets where loans have been disbursed to processors. During the 1990s, the state-owned dairy sector processors attempted to survive and transform, but ultimately failed and were liquidated. Production scales and technology were outmoded and inappropriate for the market economy. New processors including those financed by the IFAD project emerged to replace the defunct state processors, and these and their suppliers, who include a substantial proportion of poor, have been helped significantly by IFAD loans in two judets, Alba and Hunedoara.

Sustainability and ownership

The rating for this is determined by whether the results of the project will be sustained in the future without external assistance. Bearing in mind that the results of the project have limited impact on the poor, the assessment is nevertheless that some supporting factors (ownership, technical and financial sustainability) make the prospects for the systems put in place by the project in the medium term quite good, even though the future of the revolving credit fund remains uncertain.

Innovation, replicability and scaling up

The project was driven towards disbursement through a conventional, non-innovative, commercial banking operation. Yet, some innovative elements are visible in the project, which could be taken and applied elsewhere.  Had it stayed the distance with a smallholder and small enterprise focus, it may well have found a solution of lending to small borrowers.  Essentially the project is not to be repeated since it didn't build a system which supported the rural poor, although it was instrumental in introducing a rural finance system in Romania.

Conclusions and recommendations

Overall assessment

ADP was designed and implemented under difficult conditions of macroeconomic and financial sector volatility, and institutional uncertainty. However, the evaluation raises a number of questions about the design system that is operated by IFAD. These relate to the trade-off between maintaining continuity between the different phases of design, and the objective assessment that is needed at appraisal to confirm the main design parameters and assumptions of the project.

This evaluation points to the major shift in the project lending strategy from a small to a medium- and large-size borrower emphasis. It also points out that the impact paradigm that was implicit in moving from smaller to larger borrowers, and the assumed benefits from backward and forward linkages, were not adequately analyzed as a basis for the shift in strategy. Nor was an associated growth-based operational strategy put in place to ensure that poverty impacts would be maximized. And given the fundamental change in the impact paradigm of the project, there should have been a formal re-appraisal.

The table below shows the project ratings for performance, impact and overarching factors:

Performance ratings of the Apuseni development project

Evaluation Criteria

Evaluation Ratings a/

Relevance

4

Effectiveness

4

Efficiency

4

Project Performance b/

4

Impact

3

Sustainability

3

Innovation, replication, and scaling up

3

Overall Project Achievement c/

3

Performance of IFAD

3

Performance of Government

4

Performance of credit institution: BCR

3

Performance of credit institution: BCC

5

Performance of cooperating institution:  UNOPS

 

a/    IFAD uses a rating scale of 1 to 6, where 1 represents the lower score and 6 the highest.
b/    The rating for project performance is, as per OE project evaluation methodology, calculated as the average of relevance, effectiveness and efficiency.
c/    The overall project achievement rating is, as per OE methodology, given by the evaluation team taking into consideration its assessment of project relevance, effectiveness, efficiency, rural poverty impact, sustainability, innovation, replication and up-scaling.

Recommendations agreed upon by all partners

There are no specific recommendations for the project per se since further IFAD loans are not in the pipeline, given the fact that Romania has become a part of the European Union. However, the following recommendations are offered to IFAD and the Government of Romania for the design and implementation of future projects and programmes similar to the ADP.

Design process

The evaluation has highlighted a number of issue areas in the IFAD project design process. These relate to the appraisal function, to the internal accountability process involving the Technical Review Committee (TRC) and the Operational Strategy Committee (OSC), to organizational incentives relating to disbursement as the driver of performance, and to the issue of re-appraisal of a non-performing project.

IFAD is currently revisiting its overall design process through the Action Plan to ensure the quality and enhancement of its products, performance and processes. Within this context, the Fund should:

  • Review the role of appraisal in the overall design process, distinguishing between its use for "critical assessment" or for "confirmation" of design parameters and assumptions;
  • Ensure an appraisal function which is independent of formulation;
  • Review the manner in which comments provided by TRC and OSC are dealt with in relation to project appraisal and finalization, possibly having OSC comment at the appraisal stage;
  • Review organizational incentives relating to disbursement as an implicit driver and measure of portfolio and loan performance; and
  • Consider a trigger mechanism for automatic re-appraisal of non-performing projects, possibly based on supervision performance ratings.

It is expected that some of the above recommendations will be addressed by the new independent Quality Enhancement mechanism (already operational) and the Quality Assurance mechanism (recently established in January 2008) which will come under the responsibility of the IFAD Vice-President's Office.

Partners involved: IFAD

Targeting and monitoring and evaluation

Several findings on the issue of Targeting and Monitoring and Evaluation (M&E) stem from the evaluation of the ADP. Recommendations to this regard include the following:

  • The need for IFAD to develop a clear target group at the stage of project/programme design in a defined area and remain focused in line with the provisions in its targeting policy;
  • Consider, where appropriate, the undertaking of a baseline before implementation start-up;
  • Identify quantitative and qualitative Objectively Verifiable Indicators (OVIs) and include these in the log-frame; and
  • Ensure that an M&E system is a core activity of any project.

Partners involved: GOR, IFAD

The revolving credit fund

One question raised by the evaluation with regard to the sustainability of the credit component relates to the future of the Revolving Credit Fund, that is, to ensure that it will continue to revolve as planned until 2018.  The project partners seem willing to continue using the revolving funds. This implies that lending will be on a declining scale, given the schedule of repayments to IFAD and the fact that there has been no contribution of own bank resources. Representatives of BCR confirm that they will stay with the project until its end and will utilize all the remaining available funds. Furthermore, BCR expects to expand its rural portfolio, building on the experience with the IFAD loans when using EU restructuring funds in all its branches; Banca Carpatica (BCC) has also been very active in this regard and has utilized access to ADP funding to support the expansion of its branch network.

Based on the above, the following recommendations pertain to the Revolving Credit Fund (RCF):

  • Avoid extending the RCF to additional banks; and
  • The PCIs should consider leveraging any funds (or funds from other donors) to maintain RCFs and/or develop new product or services of relevance to the target group

Partners involved: GOR, PCIs


1/ ADP Appraisal Report, IFAD, 1998, p. 11.

 

LANGUAGES: English

Community-based Rural Development Project (2008)

Burkina Faso  
May 2008

Interim evaluation

Introduction

Burkina Faso is a land-locked country in West Africa with a population of 13.6 million, of which 81 per cent live in rural areas. In 2006, GDP per capita was US$1 213 and the Human Development Index was 0.320, which was the second lowest in the world.

According1 to Burkina Faso's poverty reduction strategy paper (2003), 46.3 per cent of the population live below the national poverty line of US$0.35 per day, and 94 per cent of poor people live in rural areas.

Community-based Rural Development Project (PNGT2) 2 was originally formulated by the Government as a follow-up to the Programme National de Gestion des Terroirs (1994-1998), which was financed by the International Development Association (IDA). The project was identified and appraised by IDA between 1999 and 2000, and IDA was its main cofinancier. IDA also acted as IFAD's cooperating institution and was responsible for loan administration, project supervision and implementation support. For IFAD, the project financing type was therefore "type C".3

The Ministry of Agriculture, Water and Fisheries was the main executing agency.

At appraisal, the estimated total project cost was US$114.85 million. The IFAD loan of US$11.5 million on highly concessional terms was approved by the Executive Board in May 2000 and became effective in May 2002. It was provided on a pari passu basis4 along with the first segment (of US$66.7 million) of an IDA adaptable programme loan (APL).5 This APL was envisaged to provide financing for three consecutive five-year project phases under the umbrella of a unifying national programme called the National Programme for Decentralized Rural Development (NPDRD). The expected Government and beneficiary contributions to PNGT2 were respectively US$14.3 million and US$9.15 million. The Government of Denmark pledged a grant of US$4.2 million for the project's monitoring and evaluation (M&E) system.6 The project closed on 30 June 2007 after an extension of one year.

Evaluation objectives, methodology and process

The interim evaluation of PNGT2 was requested by the Executive Board as part of the annual work programme and budget of the Office of Evaluation (OE) for 2007. According to the IFAD Evaluation Policy, an interim evaluation is mandatory when the IFAD operational division concerned is envisaging a second phase of a project. The evaluation was conducted by OE from March to November 2007.

The first objective was to assess the performance and impact of the project against the project objectives presented to the Executive Board and in the loan agreement. The second was to generate lessons and recommendations for the next phases of the project and other similar operations in Burkina Faso. More specifically, as per OE's project evaluation methodology, the evaluation sought to: (i) assess project performance in terms of relevance, effectiveness and efficiency; (ii) assess the project's impact on rural poverty, the sustainability of benefits, and the project's contribution to innovation, replication and scaling up; and (iii) assess the performance of key project partners, including IFAD, IDA and the Government of Burkina Faso.

Each of the above-mentioned criteria has been rated on a six-point scale.7

The evaluation contained three distinct phases: (i) the preparatory phase, which entailed a desk review of documents, a preparatory mission held in March 2007, and a self-assessment by the project team; (ii) the field work phase, which consisted of the main mission fielded in June 2007, during which various stakeholders at all levels were interviewed and achievements were observed in 6 provinces out of the 26 covered by the project; and (iii) the report-writing phase, which entailed data analysis, report preparation and discussion of evaluation findings and recommendations during a stakeholder workshop held in Ouagadougou in November 2007. 

Project design

According to the report and recommendation of the President (RRP) for the project adopted in May 2000 by the Executive Board,8 the goal of the project was to reduce poverty and promote sustainable development in rural areas, breaking the spiral of rural poverty characterized by natural resource degradation, reduced production and lower quality of life. Specific objectives were to achieve: (i) improvements in the cost-effectiveness of publicly funded investments at the local level; (ii) increased management capacity of beneficiary groups and their institutions; (iii) greater absorptive capacity of rural areas; and (iv) better access for poor people to productive infrastructure and inputs, social facilities and means to preserve their environment. The project had five components, which are presented in the table below with their relative cost and component objectives.

Project components, relative cost and immediate objectives

Component

Percentage of Total Project Cost

Component Objective

Local capacity-building

7

Improve organizational, technical and managerial capacities of rural communities and their institutions to manage their development in a participatory way and to carry out village and intervillage level subprojects

Local Investment Fund

52

Finance village and intervillage-level subprojects using a local contract management (maîtrise d'ouvrage locale) approach, to improve rural peoples' access to social and economic infrastructure and services

Institutional capacity-building

25

Develop an enabling institutional environment at the provincial and national level for decentralized rural development

Land tenure security pilots

4

Promote sustainable and equitable improvements to the rural land tenure system to contribute to peace and social equity, and natural resource restoration and preservation

Project administration, coordination and M&E

12

Ensure good project coordination and administrative management

From 2004 to 2006, a UNDP project to support stakeholder consultation and local governance was integrated into the project.9 A Global Environment Facility (GEF) project to improve integrated management of lowland ecosystems was also incorporated into the project from 2005 and is still ongoing.10 Both projects provided funds and personnel for specific interventions under the overall coordination and administration of PNGT2. 

The project covered the whole country, but local capacity-building efforts and community investments were initially aimed at 2 000 villages in 26 provinces (out of the 45 provinces in Burkina Faso). In 2004, it was decided 11 to extend project coverage to all villages in those 26 provinces and, ultimately, the project reached over 3 000 villages. The project financed also a small number of subprojects in seven additional provinces, in collaboration with development projects by other donors. The project relied strongly on contractual agreements with national and local service providers from both the public and the private sector, such as consultancy firms and public technical services, and also on the rural communities, to execute its main activities following a faire-faire12 approach.

Implementation results

Local capacity-building

To pave the way for the Local Investment Fund (LIF) component and as part of the project's contribution to the decentralization process, the project relied on village land management commissions (CVGTs) to plan and manage rural development activities at the village level. The numerous consultancy firms under contract by the project for facilitating local development planning and CVGT establishment used a four-step approach. These were: community sensitization; participatory planning at the village level; formal establishment of the CVGTs; and training and coaching of CVGT members. As such, 2 986 local development plans (LDPs) were prepared in a participatory way, as compared with the 1 500 anticipated at appraisal.13 These LDPs were subsequently translated into annual investment plans (AIPs) with the help of the provincial project coordination teams.

In all, 2 981 CVGTs were established by the project, thereby significantly exceeding the appraisal target of 1 200 commissions.14 CVGT members received close to 260 000 person days of training, covering essential topics such as the functions of the CVGT, basic bookkeeping, subproject design, participatory monitoring, and facilitation and negotiation techniques. A minimum of coaching by local facilitators, on average three days per annum per village, focused on the facilitation of village meetings for annual planning and monitoring and, to a lesser extent, on the utilization and management of community investments.

To ensure sustainable operation and maintenance (O&M) of community infrastructure, the project supported the creation and basic training of O&M committees. Other local capacity-building interventions included: (i) literacy training, reaching 58 000 people at the beginner level and about 36 000 at the intermediate level; (ii) HIV/AIDS sensitization through the establishment of, and support to 231 departmental and village committees to combat AIDS; and (iii) environmental education for rural children, including the preparation of a teacher's guide and the training of over 4 500 primary school teachers in its use.

Local Investment Fund

The LIF was set up to cofinance social and productive investments in rural areas.15 The village-level facility of the LIF cofinanced subprojects included in the AIPs that were entirely managed by the village communities through the CVGT. The province-level facility was meant to cofinance more important infrastructure works to benefit large numbers of villages. Altogether, 9 622 contracts were signed between the project and CVGT, and funds were transferred to CVGT accounts opened in local financing institutions. Close to 18 250 subprojects with a total cost estimate of US$55.5 million were implemented under this component.

Over 3 000 villages benefited from project investments, which included, in order of importance in terms of financing: social infrastructure, investments for improving agricultural and livestock production, and natural resource-related activities (see table 2). The evaluation estimated that about 30 per cent of the activities planned for in the LDPs were effectively executed with project support. This percentage is satisfactory considering no limits were imposed on the volume of activities and investments that the villages could plan for.

Institutional capacity-building

This component aimed at developing a favourable institutional environment for decentralized rural development at the national and provincial level, by supporting both the human and the physical capacity of several key institutions.

In the first years of implementation, to ensure the effectiveness of the two components presented above (paragraphs 9 to 13), the project trained over 1 300 people, mostly associated with service providers under contract by the project (consultancy firms, trainers and local facilitators).16 These people were then directly involved in the implementation of activities supported by the project.

The project supported the 45 provincial technical consultation platforms (CCTPs), which were composed of public administration, technical services and development partners and responsible for identifying public investment priorities, and coordinating and harmonizing development approaches in the different sectors. Although the project's most substantial contribution probably was the financing of quarterly CCTP sessions, it also provided essential training to members on topics such as decentralization laws and processes, communication skills, and local development planning.

At the national level, the project actively took part in policy dialogue regarding decentralization in support of rural development. For instance, capitalizing on the project's community-driven development experience, project staff participated in technical review committees for the preparation of guiding documents for decentralization, the general code for local government, and the decree on the creation of village development councils. The project also commissioned several studies, developed guidelines for local development planning, supported the organization of national workshops and prepared training modules regarding decentralization.

Land tenure security pilots

Aiming at ensuring land tenure security for all users, this component supported: (i) the preparation of a practical guide for securing land tenure; (ii) the operation of a national forum for building knowledge and sharing experiences on approaches to land tenure security, namely the National Commission for Securing Land Tenure in the Rural Areas; (iii) the drafting of a national strategy and action plan for land tenure security; (iv) the development of an approach for securing land on which investments cofinanced by the project were built; and (v) the piloting of an approach for peaceful conflict management between farmers and herders through specialized village commissions.17

Project administration, coordination and M&E

Project staff was hired in adequate numbers and with the skills-mix foreseen in project design. The number of project staff was not reduced after two years of project implementation as originally planned, because the decision was taken during implementation to cover all villages in the 26 provinces of direct intervention (see paragraph 8). Training was provided to project staff, inter alia, on administrative and financial procedures, local development planning and M&E, but the emphasis was placed on learning-by-doing and experience-sharing because of the relatively limited budget for formal training.

Financial management of the project was found problem-free by supervision and audit missions. Several adjustments to project financing were made during project implementation. The ceiling of the special account had to be raised from CFAF 0.75 to CFAF 4.5 billion to allow a smoother flow of funds under the LIF component. In agreement with IDA and IFAD, government counterpart funding was reduced by half, given the difficult economic situation that the country was facing at the time. Finally, the ceiling for the amount of project cofinancing for subprojects was increased for smaller villages.18

The project set up an M&E system at three levels: (i) monitoring of outputs, with community participation through the CVGTs and supported by local facilitators; (ii) evaluation of outcomes, by means of annual surveys covering all 45 CCTPs and more than 500 villages (since 2006); and (iii) evaluation of rural development impact at the national level within the framework of the NPDRD. For the latter, a national socio-economic baseline survey was conducted in 2004, and follow-up studies were carried out in subsequent years. Several studies were prepared to allow for environmental impact assessment in the subsequent PNGT2 phases.

Project performance

Relevance

The evaluation found the project to be relevant. Although the project design pre-dated the major national strategies and policies regarding poverty reduction, rural development and decentralization, it was consistent with the main directions contained therein. The project design was also consistent with the main elements of the country strategic opportunities paper formulated in 1998. The project rightly aimed at meeting the needs of rural communities, initially in the most disadvantaged villages of 26 provinces, in terms of services and infrastructure through an innovative participatory and decentralized community- driven development (CDD) approach based on effective community empowerment. The IDA project appraisal document (PAD) – which was the guiding document for project implementation – was comprehensive and clear, although the logical framework had some limitations.19

The project was constructed as a national programme to support the ongoing decentralization process at the key levels (village, province and national) with a strong institutional capacity-building component. Local capacity-building relied mostly on a useful learning-by-doing approach and outsourcing to public and private expertise. Institutional arrangements were also found adequate, with adequately staffed, highly qualified national and provincial project teams. As mentioned in paragraph 19, the number of staff was not trimmed down after two years as foreseen at appraisal, so that sufficient human resources would remain in place to ensure support to the increased number of villages covered by the project. Administrative and financial procedures were found to be broadly adequate, with some relevant adjustments made during project implementation, such as those presented in paragraph 20.

The evaluation found, however, that project design and implementation gave little attention to the IFAD concern that all members of the target group – including the most vulnerable populations – must benefit from project interventions. The project lacked a targeting strategy and a monitoring system capable of capturing data in a disaggregated manner according to poverty and vulnerability criteria. Another limitation of the design was the shift in focus away from community-based natural resource management (NRM). This had originally been a major theme at the time of project formulation and still remains one of the most important issues for agricultural and rural development in Burkina Faso. It was also the evaluation team's opinion that the project missed out on strengthening intervillage coordination and experience exchange on subproject planning and implementation, in particular at the department level, an administrative level that has now become the rural commune, and the most important level of decentralized government.

Furthermore, the evaluation concluded that the project design had not allowed adequate resources for coaching the communities on participatory planning, design of sound subprojects and management of community resources. Owing to the fact that local planning was confined to village boundaries and to one-year execution periods, community NRM activities requiring intervillage consultation and coordination were usually excluded from the local development plans. Provisions to develop the capacity of small entrepreneurs and local service providers were also found to be insufficient, despite the fact that they were key players in subproject implementation and that the rural private sector was playing an increasing role in local development with the advancement of the rural decentralization process.

Effectiveness

Overall, the project was judged to be effective. At loan closing, on 31 December 2007, the project had disbursed 99.46 per cent of its total appraisal cost estimate, and 99.85 per cent of the IFAD loan.20 With respect to the project's specific objectives (as mentioned in paragraph 6), effectiveness was very satisfactory in terms of providing social facilities and collective productive infrastructure to rural communities (see paragraph 30 below). The project was broadly successful with respect to building planning and subproject management capacity of beneficiary groups (see paragraph 27). There is no doubt that cost-effectiveness of publicly funded investments at the local level was increased (see paragraphs ¿37 and ¿38) and that the absorptive capacity of rural areas was improved. The project also obtained significant results regarding institutional capacity-building at the province and national level (see paragraphs 33 and 34). Effectiveness was sometimes found to be weaker, however, in terms of qualitative aspects such as the strengthening of local capacity to manage community resources and infrastructure.

Effectiveness was also found to be rather poor for the land tenure security pilots component.

A succinct overview of project effectiveness under each component is provided in the next paragraphs.

Local capacity-building

Most CVGTs have acquired the necessary skills for subproject planning and management, through the successful combination of coaching and learning-by-doing. The participatory LDPs prepared present a generally good assessment of village socio-economic conditions and development constraints, and propose a long-term village development plan with concrete activities for developing productive and social assets, well matched to the priority needs of the village inhabitants. Effectiveness was also considered high for HIV/AIDS sensitization and environmental education.

The evaluation found, however, that LDPs took too little into account local initiatives and opportunities, and gave inadequate attention to feasibility of planned activities. The subsequent translation of LDPs into AIPs often resulted in a "shopping-list" of investments eligible under the LIF, picked from a standardized menu. There was a clear preference for hardware investments with a maximum input from the project and minimum effort from the communities. As a result, most activities requiring a strong time investment by the population – such as community NRM activities – became a lower priority.

The standardized approach to set up CVGTs resulted in significant differences in capacity and dynamism among the latter, strongly conditioned by initial conditions at the village level. In many cases, CVGTs were driven by a small group of key members who did not regularly consult with the community on important decisions. Poor bookkeeping by CVGTs indicates less effective training and coaching on that aspect. In a similar way, capacity-building efforts to ensure O&M of community infrastructure were too basic and uniform, regardless of the type of infrastructure to be managed. As a result, the evaluation found that probably less than half the community investments have a functional and sustainable O&M arrangement in place.

Local investment fund

As the implementation results presented in paragraphs ¿11 and 13 show, the village-level facility of the LIF component was highly effective in quantitative terms. This was, in addition to adequate component design, mostly due to the existence of good quality local expertise (entrepreneurs, building site controllers, etc.) and adequate supervision of subproject works by CVGT subcommissions. The evaluation found the social and productive infrastructure built under the project to be of good quality. The evaluation tried to obtain a better understanding of a number of key elements in the local subproject management approach, two of which – cost-sharing and contract management by the village communities – are briefly discussed in box 1 below.

Some NRM investments and activities were less effective. For example, herd passageways, tree plantations and improved lowlands were often found in a poor state, mostly because of inadequate maintenance and management arrangements, or unresolved land tenure issues that usually went beyond village boundaries.

The province-level facility of the LIF was less effective due to long delays in project preparation and execution, because, inter alia, studies and works were subject to national tenders, coordination among provincial public agencies was difficult, and the local capacity to carry out major infrastructural works was lacking. As a result, the province-level facility was mostly used to finance village-level investments in villages where the annual cofinancing ceiling had been reached.

Summary of evaluation findings on community cost-sharing and contract management

Community Cost-sharing

Depending on the type of subproject, the village community had to share a predetermined portion of the costs, labour force and primary building materials, to promote greater accountability by the CVGT to the community, and stronger ownership by the community. The financial contribution from the village had to be deposited in the CVGT bank account before the project would transfer its part of the financing for the subproject. The collection of financial contributions was often problematic due to coincidence of the collection period with low cash reserves at the household level, lack of trust in the CVGT or disinterest in the activities foreseen in the annual investment plan by certain groups within the community. Because cost-sharing by the community was a condition for obtaining funding from the LIF, local solutions to collection problems were found in most cases, ranging from solidarity within social groups to outright underhand recovery of contributions from contractors. Several cases were observed where the community provided its financial contribution as required by the project, but made an underhand deal with the local entrepreneur (the contractor) to recover part of that contribution from the latter, once the contract was closed. In many villages, contributions were unequal across social strata, sometimes with the adverse effect of promoting patronage and local mechanisms that differentiate rights of access to public infrastructure.

For example, many cases were noted where a relatively well-off village inhabitant paid half or more of the village contribution to a project investment, such as a bore well or an input storage building. His family would then often have priority access to the facility and would quite frequently control its management.

In many other cases, social groups that were unable to contribute as much as the rest of the village to the costs of infrastructure built under the project, or that had difficulties in paying the service fee to use the facilities, would not receive the same access to those facilities as the relatively better-off.

In-kind contributions were the most difficult to collect, as these were not a precondition for project support but were part of the contracts with the local entrepreneurs. Shortages in many cases had to be compensated for by the contractors.

Community contract management

The tender for contracts to implement works under the AIPs was mostly managed by the CGVT in a transparent way and with close supervision by the project's provincial coordination teams. Irregular practices by contractors to win tenders were, however, quite frequent in certain parts of the country and documented by the project (bribery of CVGT members, below-cost tender offers, etc.). A subcommission of the CVGT was created to supervise works and a controller was hired for each building site.

Despite the often insufficient community contribution (see point above), the sample of small and medium private entrepreneurs met by the evaluation team said that they had considerably benefited from the business opportunities offered by the project. They also praised the swiftness of payment at completion of the works.

Institutional capacity-building

At the national level, through policy dialogue, the project effectively brought forward certain ideas that were taken up in the decree on the creation of village development councils (VDCs), adopted in 2007. These ideas included the principle of transfer of mandate and funds from CVGTs to VDCs and the provision for village general assemblies to hold VDCs accountable to the village inhabitants.

At the provincial level, stakeholders met by the evaluation team agreed that the CCTP sessions sponsored by the project allowed for greatly improved consultation as well as increased accountability among public entities and development partners regarding public investments and development interventions. This consultation, however, has remained to a large extent project-driven and so far has not led – at least in the provinces visited by the evaluation team – to either adequate coordination of interventions or harmonization of approaches and procedures, which still depend very much on approaches and procedures dictated from above.

The training of service providers under contract by the project and directly involved in the implementation of project activities undoubtedly had a positive effect on the knowledge and skills of these providers, and therefore on the quality of services provided to the communities.

Land tenure security pilots

Thanks to its mostly financial support to the National Commission for Securing Land Tenure in the Rural Areas (CNSFR) and other agencies involved in land tenure, the project contributed to the discussions and validation of the important National Policy for Securing Land Tenure, which was adopted in October 2007.

The pilot work on approaches for securing community investments and peacefully resolving conflict over land between farmers and herders in six provinces also showed promising results. Land tenure was secured for plots on which community infrastructure was built under the project. In one province, a large proportion of conflicts over land between farmers and herders, mostly cases where herds had damaged crops, could be resolved peacefully by village communities without intervention by public administration.

This component, however, has achieved only part of what was originally intended. Poor effectiveness in the earlier project years due to design, personnel and external problems, led to the component's reformulation following the mid-term review (2004), with significant scaling-down of its ambitions and transfer of part of the component's assignments to the CNSFR. However, the CNSFR has not yet succeeded in involving all public and civil society players with valuable expertise in the field of land tenure. The evaluation also found the practical guide for securing land tenure very theoretical and of rather limited applicability, drawing insufficiently on recent experiences and best practices. The pilot work on securing community investments and resolving conflict between farmers and herders was little exploited and remained disconnected from the institutional and policy support at the national level.

Efficiency

The evaluation did not attempt a detailed economic and financial analysis of the overall project, as many project interventions were directed towards institutional strengthening and promotion of participation and capacity-building of beneficiaries for which the economic benefits are difficult to assess. Economic rates of return (ERRs) were estimated by a study under the project21 for certain types of community investments, based upon a limited sample, and indicate positive ERRs overall, ranging from 3 per cent for vaccination centres to 265 per cent for compost pits. An overall portfolio ERR for the LIF was estimated by IDA,22 based on the relative weight of the main subproject types implemented between 2002 and 2006, and points to an ERR of 74 per cent. These estimates appeared optimistic to the evaluation team, which had found O&M problems for many, as well as under-utilization for some community investments (as discussed in paragraphs ¿28 and 50).

Subprojects carried out under the LIF showed good efficiency, with average savings of between 5 and 10 per cent on the standard costs foreseen in the technical and financial reference used by the project. Project estimates indicate that costs were between 6 and 66 per cent lower than sector comparators, with the largest savings evident in social infrastructure. This satisfactory efficiency was largely due to beneficiary cost-sharing and supervision of works, and to the CDD approach, which stimulated the emergence of competitive local entrepreneurs with low fixed operating costs.

Regarding capacity-building and policy dialogue, expenditure was typically high for a project that combined both CDD and policy assistance. Although the project's contribution was remarkable, the quantity of villages reached by capacity-building efforts clearly prevailed over quality on certain aspects (mentioned earlier in paragraph ¿28). The results achieved by the land tenure security pilots component did not justify the resources mobilized.

The institutional arrangements were found efficient, considering that operating costs were maintained below 12 per cent of total expenditure despite the fact that staff were not cut back after two years of implementation as foreseen, and the project was extended by one year.

Overall, on account of the less than satisfactory quality of certain project-supported capacity-building and policy assistance interventions and sustainability issues related to community resources, the evaluation judged the project moderately efficient. 

Rural poverty impact

The project has made an impact in several domains. Impact was most significant on: (i) physical assets: 37 per cent of the country's rural villages were provided with some essential basic infrastructure, thereby increasing the rural population's access to potable water, basic health care, education and some productive physical assets; (ii) human assets: improvements were registered in literacy rate and school attendance, health conditions and awareness, and technical and local development management capacity in the villages covered; and (iii) institution and services: a definite contribution was made to the country's decentralization process, which is expected to raise the quality of public administration and service delivery in rural areas through more effective people's participation in decision-making and greater accountability on the part of the Government and service providers.

The project had a moderate impact on social capital and the empowerment of rural communities in that it strengthened village-level capacity for participatory planning and implementation of subprojects andempowered rural communities to a certain extent in their relationship with public and private service providers. Agricultural productivity was significantly improved on lands treated by soil and water conservation works or fertilized by compost pits, although these lands represent only 2-3 per cent of the cultivated area of involved villages.

Impact on common natural resources management was weak, despite the fact that access to natural resources, in particular land, is fundamental to the livelihoods of most rural households. Soil fertility was improved at the individual field level where soil and water conservation works were successfully carried out, but the areas concerned are too dispersed to have any positive effect at the watershed level. More importantly, few solutions were found to land tenure issues concerning productive land (fields, pastures and forests).

The evaluation found that economic and social benefits remained below expectations for the more vulnerable population groups, including women, youth, herders and immigrants, which were specifically mentioned as priority target groups in IFAD's RRP.

The evaluation also found that the project missed the opportunity to strengthen the capacity of rural communities to coordinate and defend their interests beyond the village level.

This would have been useful in the light of the subsequent creation of the rural communes, which as mentioned in paragraph ¿23, have acted as the lowest level of local government in rural areas since 2005. Finally, harmonization and collaboration between province-level institutions outside the collaborative framework of development projects did not appear to have improved significantly.

Considering the above, overall rural poverty impact was found moderately successful by the evaluation.

Sustainability

A series of positive factors argue in favour of sustainability, namely: (i) a satisfactory uptake of the CDD approach by involved communities, with improved skills at the local level for subproject planning and management; (ii) the adoption, in most cases, of cost-sharing principles (see box 1); (iii) the usefulness of the majority of community-level investments, and therefore the existence of a local incentive to maintain those investments in good condition; (iv) land tenure security for community infrastructure; and (v) the opportunities presented by the next project phases that could focus more on consolidating local O&M capacity in coordination with the rural communes.

However, certain factors may impede sustainability of benefits, such as: (i) the economic non-viability or suboptimal use of some of the economic infrastructure (cereal banks, stores, etc.); (ii) the insufficient contribution by users towards maintenance of social investments (water points, meeting halls, etc.); (iii) the generally poor management capacity and inadequate remuneration of local O&M committees; (iv) unresolved land tenure security issues for productive and NRM investments; and (v) the lack of public funding to sustain public technical service support for local communities and to finance CCTP sessions.

On balance, the evaluation found sustainability of project achievements moderately satisfactory.

Innovation, replication and scaling up

Overall, the first phase of PNGT2 was innovative. The CDD approach followed by the project on a very large scale was both courageous and innovative in Burkina Faso in the sense that the approach gave village communities full responsibility for all technical and financial aspects of subproject planning and management. The three-tier M&E system of the project and the pilots in land tenure security were also the first of their kind in Burkina Faso. Several innovative refinements in procedures and methodology were tried out by provincial coordination teams, for example, in the type of support provided to villages for the awarding of contracts.

 Innovations were, to a certain extent, successfully replicated within the project itself, and scaled up outside the project through policy dialogue. The combination of highly visible field interventions with a strong institutional capacity-building and policy dialogue component, allowed the project to scale up key features of its CDD and provincial consultation approaches, by making sure that these features were incorporated into the rural decentralization process at the national level.

However, the participatory planning approach did not, in itself, promote much innovation because it did not allow for the full exploitation of local opportunities and initiatives (see paragraph ¿27). Likewise, the mechanisms for community investment O&M lacked innovation and were too standardized and simplistic to cater for the highly diverse types of investments and socio-economic contexts (see paragraph ¿28). Innovative experiences in land tenure security were poorly documented and their replication outside the provinces covered by the pilots has not yet been realized. 

Partner performance

IFAD

For IFAD, the project was categorized as financing type C (see footnote 3). The Fund provided a significant financial contribution to the project, but IFAD-specific objectives were inadequately taken into account in project design, which was led by IDA. Relevant issues23 were raised in the course of the technical and strategic internal quality assurance process during project design at IFAD. These issues were partly addressed in IFAD's RRP, but IFAD was not successful in ensuring that these concerns were adequately reflected in the final PAD.  

IFAD delegated loan administration, project supervision and implementation support to IDA, for which IDA performed below expectations (see paragraph 60). IDA did not give adequate attention to IFAD-specific concerns (see paragraph 24) nor did it promote IFAD experiences and expertise in the region during project implementation. IFAD's capacity to learn from the project has been reduced by the fact that six different country programme managers were involved between project design and completion. Finally, collaboration with other IFAD-financed projects in Burkina Faso was limited, despite the existence of promising opportunities.24 The evaluation rated IFAD performance moderately unsatisfactory. 

Government

The evaluation judged government performance satisfactory. The Government of Burkina Faso showed strong interest in and commitment to the project throughout implementation, and its continuous support certainly contributed greatly to the project's achievements, especially in terms of institutional capacity-building. The Government has made optimal use of the project's resources and experiences to advance its rural decentralization agenda. However, the Government did not fully succeed in ensuring harmonization and coordination among rural development partners and projects in the country.

Most of the project's achievements can be attributed to the good quality of project management at the national and provincial level. As mentioned in paragraph 19, the numbers and skills-mix of the project staff at the various levels were adequate. The project team succeeded, to a certain extent, in fine-tuning the project approaches according to the specific context of each province, and worked at the provincial level in close consultation with technical partners. Project M&E, as presented in paragraph ¿20, was also found to be adequate and innovative, especially in terms of steering the project at the national level. The M&E system has, however, not yet been transferred to the appropriate government agency to become the national rural development M&E system, as originally intended.

IDA

Overall, the evaluation considered IDA's contribution as cofinancier moderately satisfactory. As the main cofinancier of the PNGT2, IDA was the main driver behind the project's innovative and courageous design, which was largely based on other World Bank CDD projects in the region. As such, the main strengths – but also some flaws – in the design of the project, as discussed under project relevance, are attributable to IDA.

While the main ministry involved in project execution has only praise for IDA's support to the project, the other project partners, including IFAD, UNDP and the Embassy of Denmark, found that IDA made little effort to develop a true partnership among donors and showed little interest in other donors' specific objectives.

As IFAD's cooperating institution, however, IDA's performance was found unsatisfactory. While it has to be taken into consideration that IDA administration of the IFAD loan, project supervision and implementation support were free of charge for IFAD, IDA did not adequately respect its commitments. Delays in non-objection statements were frequent, communication with IFAD on both fiduciary and technical aspects was poor and often untimely, and supervision reports, although regular, were of poor quality. The "fluid" way in which IDA organized its supervision and implementation support missions,25 and the often late announcement of such missions to IFAD, made regular participation by IFAD difficult.

Summary of ratings

The table below summarizes the evaluation ratings for the project.

Performance of the Burkina Faso Community-based Rural Development Project

Evaluation Criteria

Ratings

Relevance

5

Effectiveness

5

Efficiency

4

Project Performance26

4.7

Impact

4

Sustainability

4

Innovation, Replication and Scaling Ip

5

Overall Project Achievement27

4

Performance of IFAD

3

Performance of Government

5

Performance of IDA

3

Conclusions and recommendations

Conclusions

Overall, the evaluation assessed the project as moderately successful. The first phase of PNGT2 supported relevant interventions at the village, provincial and national level that responded well to the needs of the majority of the rural people, while remaining aligned with government policies and strategies. This can be attributed to innovative and effective approaches at both the macro level (national programme for institutional capacity building) and the micro level (CDD), implemented by a high-quality project team and skilled service providers.

However, participation by vulnerable groups in local decision-making, project activities and benefits fell short of IFAD's expectations. The project lacked a clear and effective targeting strategy, and therefore its participatory local planning (CDD) approach did not effectively reach the most marginalized and disadvantaged groups, such as women and young people (see paragraph 49).

The sheer number of villages covered by the project meant that quantity often prevailed over quality, and a highly standardized planning and management approach was favoured over more flexible and adaptable mechanisms. This resulted in poor valorization of local development initiatives and opportunities, inadequate attention to the more complex NRM issues and fragile O&M arrangements for many community investments (as analysed in paragraphs ¿27 and ¿28).

Impact on shared NRM was unsatisfactory, because these activities were usually not a priority in local development plans for two reasons: (i) subprojects were confined to individual villages and one-year periods, which is often not appropriate for NRM initiatives (see paragraph ¿24); and (ii) rural communities gave preference to hardware investments (such as infrastructure) that required relatively less time and effort on their behalf (see paragraph 28). The idea of financing larger subprojects through a province-level facility was relevant, but planning and implementation procedures for this facility were found inadequate (as described in paragraph 32). The first phase of PNGT2 also demonstrated the limitations of an isolated and overly ambitious land tenure component (see paragraph ¿36).

The principles of cost-sharing in community investments and charging maintenance fees for services were intended to promote ownership and better management by beneficiaries. Cost-sharing, however, as analysed in more detail in box 1, induced sometimes irregular practices (such as underhand recovery of part of the community's contribution from the local entrepreneurs by those who had contributed). It also led to local mechanisms that differentiate rights of access for the rural population to public goods in favour of the less poor.

Government performance was satisfactory, but both IFAD and IDA performed below expectations. IFAD was unable to attract sufficient attention to its priorities of combating poverty vulnerability and ensuring adequate targeting of the rural poor, as recommended by internal reviews within IFAD, prior to project approval. This was mostly due to IFAD's weak involvement in project design, and supervision and implementation support (see paragraph 55 and 56).  

Recommendations

The evaluation makes five recommendations. The first recommendation relates to IFAD's further involvement in a subsequent PNGT2 phase. The other four recommendations relate to specific strategic issues to be considered by the Government and IDA for the next phase of the PNGT2.

Recommendation 1: Further IFAD involvement. The evaluation recommends IFAD's continued participation in the next phase of the PNGT2. In this regard, it is important for IFAD to engage in a dialogue with the Government and IDA to ensure that the recommendations emerging from this evaluation are taken on board by the main partners fully and in a timely manner. Additionally, in the next phase, IFAD should be more involved in project supervision and implementation support, and ensure that there are opportunities to promote learning across IFAD-supported projects, including PNGT2, in the country.

Recommendation 2: Inclusion of the most vulnerable population groups. Through their support to the next project phase, project partners should ensure that the poorest, most marginalized and most vulnerable among the active rural population fully participate in project interventions and fully share in project benefits. In particular, it would be useful to: (i) improve the project's understanding of the mechanisms of social and economic exclusion affecting the most vulnerable social groups; (ii) develop approaches for local planning and M&E that ensure full participation by vulnerable or marginalized groups and assign unequivocal priority to reducing vulnerability among such groups; and (iii) provide incentives and earmark resources for specific subprojects aimed at reducing vulnerability and exclusion of the rural poor.

Recommendation 3: Empowerment of rural communities in the recent decentralization context. Village and community-based planning approaches should place greater emphasis on endogenous potential and initiatives to develop local resources. The next project phase should support a changing role for public technical services, helping them devote greater attention to providing advice and assistance to local government and community initiatives. These service providers should help develop appropriate mechanisms to promote effective accountability on the part of VDCs and elected local officials in developing, financing, implementing and monitoring subprojects.

Recommendation 4: Sustainability of local investments. To ensure the sustainability of project investments, the next project phase should contribute to: (i) the development of mechanisms for cost-sharing in the construction and O&M costs of community infrastructure that are equitable and adapted to the specificities of different types of investments; (ii) the promotion of rural people's access to the means (inputs, technical advice, etc.) to maximize returns on investments; and (iii) the development of an enabling regulatory framework to finance O&M of public infrastructure in rural areas through the future communal fiscal system.

Recommendation 5: Natural resource management. The CDD approach should be adjusted to accommodate more easily community NRM subprojects that go beyond the geographic boundaries of one village or one rural commune, and take longer than one year to implement. The next project phase should help develop appropriate accompanying measures and financing instruments to support consensus-building processes among communities, villages and rural communes. These processes would aim at bringing users and decision-makers together to adjust the NRM rules to ensure sustainable exploitation or protection of shared natural resources. Land tenure security should be mainstreamed into all project components as a cross-cutting theme.


1/ Sources: World Bank: World Development Indicators 2007; United Nations Development Programme (UNDP): Human Development Report 2007.

2/ The project is known in Burkina Faso as Deuxième Programme National de Gestion des Terroirs – phase I. The project name mentioned on official IFAD documents in French is Projet national de gestion de terroirs ( phase II).

3/Project financing type C means that: (i) the project was initiated by another development partner; (ii) this development partner provides project cofinancing; and (iii) the partner acts as IFAD's cooperating institution.

4/According to the pari passu principle, for each disbursement on the IDA loan, an amount equal to one sixth of the IDA disbursement is disbursed from the IFAD loan.

5/An adaptable programme loan is a World Bank Group lending instrument that provides phased support for long-term development programmes by means of a series of loans. Progress in each phase of the programme is reviewed and evaluated on the basis of agreed indicators. The targets set for these indicators (triggers) have to be reached before a subsequent phase can be initiated and corresponding funding is made available.

6/ The Government of the Netherlands had originally envisaged a US$9.0 million grant for the project, but withdrew before project effectiveness following a change in development assistance policy in the Netherlands.

7/ Ratings are given on a scale from 1 to 6, with 6 = highly satisfactory, 5 = satisfactory, 4 = moderately satisfactory, 3 = moderately unsatisfactory, 2 = unsatisfactory and 1 = highly unsatisfactory.

8/ Project objectives were formulated differently in the project appraisal document (PAD) prepared by IDA. According to the PAD, the project development objective was to increase the productive capacity of the rural sector and improve the effectiveness of public investments by developing the institutional and organizational capacity necessary to enable local communities to plan, implement and manage their own development process. Specific objectives of the project in the PAD are to (i) build local capacity to plan and implement rural development and accelerate the pace of public transfers for decentralized rural development, and (ii) support the implementation of the country's decentralization framework. The component objectives in the PAD correspond to those presented in the RRP (EB 2000/69/R.17/Rev.1) presented to the Executive Board.

9/ The UNDP grant-financed Consultation and Local Governance Support Project had an estimated cost of US$2.4 million.

 

10/ The GEF grant-financed Sahel Integrated Lowland Ecosystems Management Project had an estimated cost of US$1.8 million.

11/ This decision was made by the project steering committee following a supervision mission by National Assembly members in 2004 that found project activities highly relevant and effective.

12/ Faire-faire means literally "make-do". Following this approach, the project sources important parts of its interventions out to service providers under contractual agreements. In PNGT2, for example, the creation of village-level institutions and participatory planning were outsourced to national consultancy firms.

13/ At appraisal, it was expected that at least 75 per cent of the 2 000 beneficiary villages would receive capacity-building support and have adopted a local development plan.

 

14/ At appraisal, it was expected that at least 60 per cent of the 2 000 beneficiary villages would have representative and participatory bodies.

15/ Individual income-generating activities were not eligible under the LIF.

16/ The project hired local facilitators to assist the CVGTs with planning and monitoring of project activities. Their number increased along the project implementation period in parallel with the number of villages covered. On average, each local facilitator supported five villages.

17/ In following a joint ministerial order on conflict management between farmers and herders dated 2000.

18/ Originally, the investment ceiling per village was calculated on the basis of a fixed annual amount per inhabitant of CFAF 5,000, regardless of the size of the village. As a result, the smaller villages could not receive sufficient funding for community infrastructure subprojects. After the mid-term review, the project adopted a new formula for calculating the cofinancing ceiling, which attributed to all villages a fixed yearly amount of CFAF 5 000 000, and – counting from the 1001th inhabitant upwards – an additional fixed amount per inhabitant of CFAF 3 000.

19 For example, there was a lack of clarity between objectives and outputs, and inadequate indicators for quality, efficiency and sustainability of project interventions and results.

20/ A financing gap of approximately US$14 million created by the withdrawal of the Netherlands, an overestimate at appraisal of IFAD cofinancing and a decrease in the Government's contribution after the first year of implementation were compensated for by IDA and higher than expected beneficiary contributions.

21/ Konate, S. April 2007. Economic and financial analysis of community investments under PNGT2, Final Report.

22/As part of the preparatory work for the formulation of the second phase of PNGT2.

23/ These included: (i) focusing IFAD support on one component only; (ii) clarifying mechanisms to ensure that marginalized population groups would not be excluded from local planning and, as a result, from project benefits; (iii) putting more emphasis on NRM; (iv) providing adequate support to O&M of infrastructure; and (v) clarifying IFAD's role in project supervision.

24/ Such opportunities included: strengthening the local private sector (Rural Microenterprise Support Project), financing income-generating activities and supporting true community-based NRM (Community Investment Programme for Agricultural Fertility [PICOFA] and Sustainable Rural Development Programme) or supporting rural financial organizations (PICOFA and others). IFAD project coordinators met by the team in Burkina Faso agreed that many opportunities for collaboration and mutual learning had not been capitalized on.

25/ This entailed IDA experts often visiting the project separately, more or less as they saw fit.

26/ The rating for project performance is, as per OE project evaluation methodology, calculated as the average of relevance, effectiveness and efficiency.

27/ The overall project achievement rating is, as per OE methodology, given by the evaluation team taking into consideration its assessment of project relevance, effectiveness, efficiency, rural poverty impact, sustainability, innovation, replication and up-scaling.

 

LANGUAGES: English, French

Community-initiated Agriculture and Resource Management Project (CARD) (2008)

Belize  
April 2008

Completion evaluation

Country background

Physical and social features. Belize is a country of 22 965 km², located in Central America, bordering the Caribbean Sea, between Guatemala and Mexico. The population of the country stood at 291 800 inhabitants in 2005, up from 211 000 in 1994, with one of the lowest population densities (12 inhabitants/km2) in Latin America. The country is divided in six administrative districts: Belize, Corozal, Orange Walk, Cayo, Stann and Toledo. During last decade, the country has been struggling against the severe damages of several hurricanes: Mitch (1998), Keith (2000), Chantal and Iris (2001) causing considerable impact on socio-economic assets and development.

Rural poverty. In 2003, Belize ranked 91 on the global Human Development Index (HDI) scale, i.e., in the mid-level range of countries with medium human development. According to the 2002 Poverty Assessment Report, 24.5 per cent of households live under the poverty line.  Poverty has a much greater incidence in rural areas (33.7 per cent) than in urban areas (17.2 per cent). Children are more likely than any other population group to suffer from poverty (39.0 per cent), especially in rural areas (51 per cent) than urban areas (27 per cent), and for indigenous Mayan children (83 per cent). The Government of Belize (GoB) continues to put the primary focus of its strategies on the fight against poverty – foremost through the education, health and productive sectors and with the southern districts as a key focus.

Small farmers. The majority of small farmers1 mainly produce maize, beans and rice. In remote rural areas, small farmers practice a traditional shifting (slash & burn) cultivation system called Milpa which contributes significantly to domestic food production. Milpa cultivation is predominantly found in indigenous communities in the southern districts. In response to increasing population pressure, the Milpa system has been evolving towards a more sedentary pattern in recent years, using a combination of perennial crops and food crops. At the time of project design, there was virtually no access to credit for small farmers in the project area. The public and private sector commercial banks were not interested in the risky agricultural productive sector or in costly small rural clients, whereas credit unions (CUs) were providing services in some rural areas.

The project

The Community-initiated Agriculture and Resource Management Project (CARD) was initially conceived as the second phase of another IFAD-funded project, the Toledo Small Farmer Development Project (TSDF). The performance of the TSDF was assessed as moderately unsatisfactory2 but due to the consistently higher rates of poverty in the southern districts of the country, and the emerging benefits in agricultural productivity, a second phase was recommended. In view of the lessons drawn from the TSDF evaluation, a more participatory, bottom-up, community-led approach was designed to enable communities to decide on their own development alternatives and opportunities. During project appraisal, the new project was renamed as the CARD project rather TSDF Phase II, to reflect the substantial design changes.

CARD proposed a more integrated and participative approach for the rural poor and the indigenous communities to become involved in their own development. The overall goal of the project was to reduce rural poverty in southern Belize and raise production, employment and income of the rural poor. The purpose was to develop the productive potential of balanced sustainable land use systems and ensure support services to poor smallholder families in the southern region. CARD became effective in June 1999, under a co-financing agreement with the Caribbean Development Bank (CDB). CDB was also designated Cooperating Institution and assumed major responsibility for project supervision. The Ministry of Agriculture and Fisheries (MAF) was the Executing Agency (EA). The Development Finance Corporation (DFC) acted as the financial agency for the implementation of the Rural Financial Services (RFS) component. At the time of closing in 2006, the IFAD loan had only reached 58 per cent disbursement equal to the Special Drawing Rate (SDR)1 017 000 compared to the approved loan amount of SDR 1 750 0003 .

Project area and target group. At IFAD appraisal, the original target area was defined as rural communities in Toledo and Southern Stann Creek districts. A phased approach for targeting of specific communities was proposed, with an aim to commence a pilot process in eight communities and gradually increase by ten communities per year to reach 54 communities 4 selected among the poorest sections of the population. Targeted households were classified according to three groups:

  1.  smallholders and artisanal fisheries households
  2. subsistence Milpa farmers and
  3. women from smallholder families. An additional emphasis in targeting was to seek to address the needs of the indigenous Mayan communities who comprise 11.9 per cent of the population in Stann Creek and 65 per cent of the population in Toledo.

The evaluation. The objectives of the evaluation were to assess the project in terms of: (i) overall performance; (ii) rural poverty reduction impact; (iii) innovation and sustainability; and, (iv) the performance of IFAD, CDB, the GoB and other implementing partners. The evaluation was conducted as a collaborative process by IFAD and CDB5

As there had never been a functional monitoring system, the evaluation was constrained in generating quantitative assessment of project performance, despite benefiting from the comprehensive and realistic Project Completion Report (PCR)6. Consequently, the evaluation conducted assessments on three levels to cross-verify the minimal secondary data as follows:

  1. assessment of a representative sample of community level activities7
  2. organizational level assessment
  3. and project level assessment, covering policies, partnerships, management and operations. Stakeholder meetings were conducted with key project partners such as Government of Belize (GoB) agencies, CDB and IFAD's Latin America and the Caribbean Division (PL), the Toledo Development Corporation (TDC), local and national NGOs, as well as with direct project participants in the targeted communities.

Project performance

Design features. There was an in-depth assessment with the communities and with potential development partners during the design process to identify a more relevant and appropriate project design. The design incorporated many of the lessons from the TSFD8 . CARD proposed a more participatory approach for enabling the communities to draw better benefits from development opportunities, while at the same time safeguarding cultural identity and increasing involvement in national development processes. The CARD design recognized that technical support for agriculture development was insufficient to address the complexity of poverty. An approach was required that acknowledged the interplay between resource management, production systems and culture. The project financed four components: (i) Community Promotion and Local Organizations Strengthening; (ii) Technical and Marketing Support Services and Small-scale Infrastructure Investments component; (iii) Rural Financial Services (RFS); and, (iv) Project Coordination to manage the overall programming and activities.

Changes during implementation. The essential elements of the design did not change during implementation but the implementation mechanisms were continually adjusted. The fundamental change in the project implementation was that the recommendations to appoint appropriately skilled staff to the Project Management Unit (PMU) did not eventuate.

As a result of delayed implementation and poor results, by the 2002 mid-term review (MTR), a major project restructure was required. Following continued slow progress, a decision was made by the Project Steering Committee (PSC) in early 2004 that the focus of the project should not be on attempting to implement the project as designed but rather to focus on limited activities that had been showing potential for success. This led to virtual cessation of work within the communities and a focus on activities with producer organizations and the two participating credit unions. A project extension9 was proposed by IFAD in 2005 and was considered feasible10 but unfortunately, due to the fiscal constraints of the GoB at that time, and the concerns over the poor performance of the project, the extension was not approved by GoB. In consequence, the IFAD financing was closed by the original closing date of June 2006.

Implementation and outputs

Community development and participatory planning. Activities were meant to strengthen local leadership and organizations and identify social and economic constraints to development of communities. Participatory project planning and management skills were introduced to improve the capacity of local organizations to plan and manage their own priority sub-projects.  Strong interest was showed by community members at first. Participatory planning processes were well received, but few plans led to implementation of the sub-projects identified. Financing was provided by CARD for only 59 sub-projects11 across the project area, most of which were not identified by the communities and/or which were too small to achieve the desired results (community sub-projects provided tended to be standardized by the project, e.g. three pigs per community, three sewing machines for women's groups, etc). The evaluation found very few examples where the community sub-projects had been implemented in accordance with the wishes of the community. Even training was mainly conceived as one-off, top down, educational activity. Conversely, in the isolated cases where community priorities had been supported, the results were very positive, with groups demonstrating improved financial and organizational capacity, as well as recording increases in their enterprise activities.

Technical services and marketing assistance. Based on the community plans, extension and technical assistance for improving technology on existing farms or diversification into new crops or non-agricultural activities were to be planned. Small-scale infrastructure was to be supported to improve farm access, storage and post-harvest processing. Results were mixed. The assistance for coconut growers and honey producers was initially successful, but eventually failed due to crop disease and other technical problems. CARD's most successful work was with the Toledo Cacao Growers Association (TCGA), a Mayan farmer's organization that produces organic cacao beans under a guaranteed rolling contract with Green and Black (UK-based customer)- which has also invested heavily in development of the Toledo cacao industry. As a result, 300 000 cacao grafted plants were transplanted, expanding the total acreage by about 800 acres. TCGA has expanded from 130 to 937 members. Late in the project, other similar initiatives were planned with the Livestock and Grain producers associations but these were not able to commence implementation before the end of the project. Infrastructure activities were largely confined to installation of culverts on a Ministry-led process of installation that involved minimal participation by the communities. The first installations were successful but the second batch has not yet been completed.

Rural financial services. The RFS component was designed to focus on small, short term loans, based on group guarantee, savings and other forms of innovative collateral. A credit fund was to be provided to the credit unions (CUs) as intermediary financial institutions (IFIs) for retail lending. An institutional fund was designed to strengthen the capacity of the IFIs to operate micro credit schemes. Several factors delayed the start-up of component activities including late hiring of the Rural Finance Expert (RFE) and the lack of eligible IFIs willing to operate in the project area. Negotiations with the two prospective CUs identified during appraisal failed. Eventually, two small local inexperienced credit unions: the Citrus Growers Credit Union (CGCU) and the Toledo Teachers credit union (TTCU) were accredited under relaxed criteria. There were also product design features that took a long time to resolve. Yet, the CUs managed to approve 1 056 loans amounting to BZ$1 478 193. The majority (62 per cent) of Toledo Teachers' loans went to agriculture, while nearly half (46 per cent) of Citrus Growers' loans were for commerce or retail trade. Loan delinquency is currently high in both IFIs with a portfolio at risk ratio of about 20 per cent, for Toledo Teachers and 33 per cent for Citrus Growers in December 2004, compared with the target ratio of = 12 per cent at appraisal. Portfolio at risk ratios are reported to be decreasing as the CUs consolidate their programs. Loan delinquency is mainly caused by external factors such as delays in payments of farmers by the Belize Marketing Board. However, both CUs acknowledge that they lent out loans too fast, too soon.

Project coordination. The design of CARD as a rural development project covered a broader scope of activities than was usually handled by MAF. For this reason, it was established a separate PMU that reported to MAF and which was largely autonomous in operations. In implementation, the PMU struggled to fulfil its role. The ‘innovativeness' of the project's approach made it difficult to find staff with the adequate experience. Staff members were not provided with basic, practical tools for implementation. The planned, gradual phased approach of the design was never implemented, nor was the plan to intensively support the communities on an on-going basis throughout the project period.

On the contrary, the project approach was fragmented, responding to isolated requests for support for vehicle use, or small non-viable sub-projects. The highly stratified administrative set up under the MAF, used by the project was at odds with the original design and made decision making slow. Procurement processes were a consistent issue with the project, in terms of overly bureaucratic documentary requirements, practical difficulties in complying with requirements due to low availability and quality of available suppliers and service providers. Poor communication processes between partners was also a continuing issue. The internal monitoring system was unreliable and incomplete for all stages of the project and the limited project data has constrained evaluation of project performance. Operations improved substantially late in the project when a Project Director (PD) was recruited who had project management experience and who was able to manage the coordination and communication mechanisms of the PMU more effectively.

Assessment of project performance
relevance

The design of the CARD project was found to be relevant to the target group and the Belize National Poverty Elimination Strategy focusing on the poverty "hotspot" of the Toledo District; supporting ecologically sustainable agricultural practices and addressing concerns with the traditional Milpa agricultural system. The focus on areas of highest poverty, on small farmers operating at subsistence level and indigenous people was also consistent with the policy directions of the GoB, the IFAD Strategic Framework and, retrospectively12 , the IFAD Regional Strategy for Latin America and the Caribbean. The design project responded to the stated interests of the communities that had participated in the design phase. The community-initiated approach allowed the communities to have a say in their own development and was designed to build from the activities of previous projects. The issues with the results of the project design are more in relation to underestimation of the capacity for translating the design into implementation. Similarly, some sub-projects were supported by CARD that were not relevant to the communities, but this was due to operational deficiencies rather than a lack of relevance of the project per se.

Effectiveness

There was potential for the project to be effective but the potential was not realized for most initiatives within the project period. At the community level, based on the sample of project activities reviewed, the effectiveness was minimal. Feedback from the communities was that CARD wasted their time with unproductive meetings, did not respond to their articulated needs and provided small, inconsequential sub-projects that made no difference to their living conditions. This is confirmed by the sparse project information showing that CARD financed small-scale, fragmented activities over a large number of communities rather than following the recommended phased, intensive interventions in a gradually increasing number of communities. In contrast the interventions at the organizational level, by the end of the project, had proved more effective. Credit unions have grown and expanded their activities and are showing signs of combating their issues with high arrears. The support for the TCGA has been very effective and has been the main contributor to progress towards achieving CARD's development objectives. Hence, CARD is assessed as moderately ineffective.

Efficiency

As had been identified during the project design process, human resource issues were a major risk to efficiency. Experienced and capable staff is known to be difficult to attract and retain in projects in southern Belize. The risk became reality in implementation. Delayed recruitment of staff, poor installation of basic procedures and lengthy procurement procedures meant that the project started badly and was never able to recover. Most major operational issues were not resolved until late in the project period. After the MTR, substantial efforts were made by all partners to improve efficiency by provision of short-term training and technical consultancy. However, few results were evident from the support provided apart from the increased performance with the credit and cacao development activities. The long periods without key staff such as the PD, Accountant and Monitoring and Evaluation Officer meant that internal project processes were inadequately installed, new staff members were insufficiently oriented, vital management information was not generated, and decision-making was not strategic. Loan disbursement only reached 58 per cent by time of loan closing. Overall economic analysis for the project is not possible in relation to the community sub-projects, or the economic results of the rural finance as sufficient data is not available. However, analysis of the cacao production suggests a positive economic benefit of over 40 per cent return on investment (ROI) given the prevailing production system, levels and market. However, even though the cacao production strongly contributed to the success of cacao project, success cannot be solely attributed to CARD. Also, by the time of evaluation, this potential economic benefit has only been achieved by a limited number of farmers. The estimated average costs per beneficiary are in line with other IFAD projects (US$387) but the operating costs to total investment are high (43 per cent of loan amount). Therefore, the project is rated as moderately inefficient. It is likely, given the improvements later in the project, that if an extension had been granted, a greater level of efficiency would have been achieved.

Performance of partners

CARD had a complex stakeholder mix. All three financing partners must share a level of responsibility for the divergence of the project operations from the design and the resulting poor performance. Much valuable project time was lost at the project management level in delayed recruitment, in unclear procedures and in insufficient orientation and training. Unclear definitions of roles and responsibilities and lack of clarity in communication between the partners was a continuing issue.

Performance of IFAD

IFAD played the predominant role in the design of the project but did not sufficiently follow up the issues in translating the design into implementation early in the project. The elements of policy dialogue identified in the design were not progressed in implementation. Given the known lessons from the TSFD, and the re-emergence of the same issues, as well as the need for IFAD to take responsibility for the designs proposed, IFAD's performance is assessed as moderately unsatisfactory. IFAD responded well after mid-term when the need for more intensive support was identified and the technical assistance (TA) for the microfinance activities was helpful. However, the TA provided for the staff training and Monitoring and Evaluation (M&E) was of limited effectiveness due to lack of follow-up to identify whether the theory was being put into practice.

Performance of CDB

CDB acted as the IFAD cooperating institution13 for the whole project period. CDB financial processes were conducted in accordance with the agreed procedures and financing flowed effectively once the required documentation had been submitted by PMU. However, those procedures were found to be unduly complicated and bureaucratic for the small level of financing required for individual contracts. 

Insufficient orientation was provided to the incoming staff. As a result, procedures were not adequately understood or followed. This proved a major bottleneck in expediting implementation and hence performance is assessed as moderately unsatisfactory. After MTR, CDB also responded by providing a more intensive support with improved processes; and by reducing delays of payments when cashflow was constrained, through the provision of some direct payments to suppliers.

Performance of GoB

The GoB was effective in complying with most of the provisions of the loan agreement except for effective recruitment processes. In terms of financial commitment, allocation of the agreed counterpart funding during the initial stages of the project was satisfactory. Yet, the performance of the PSC, under the Chair of MAF was patchy. At times, the PSC was seen by the PMU as attempting to micro-manage the project14 . At other times, MAF did not call a meeting for close to 12 months, between year 2003 and 2004 and again with a large gap in meetings towards the end of the project, with inconsistent attendance from representative agencies. The PSC was responsible for the unsatisfactory staffing performance of the project, including appointment of inexperienced PDs for the majority of the project period. The government improved the policy and regulatory environment for microfinance. As a result, the performance of GoB was also considered moderately unsatisfactory.

Performance of other partners

Other partners. The relationships with services providers in the early stages of the project was not adequately handled by CARD, nor adequately supported by the three financing partners. Service providers, in the main, fulfilled the requirements of the terms of reference but those requirements were rigid and not always in the best interests of the communities or groups. Yet, in the later stages of the project, the partnership between service providers and producer groups was considered to be instrumental in achieving substantial benefits. The communities themselves showed tenacity in endeavouring to become involved with the project but frustration and cynicism set in, when support for priorities was not forthcoming. Local organizations were largely by-passed in project activities.

Project impacts

Rural poverty restriction impact

In reviewing the impact of CARD, there are several over-riding factors that need to be highlighted. The overall impact at the community level was very poor. The expected impact in terms of rural poverty was not achieved, and in some cases, negative social impact such as failed community groups, disillusioned leadership, frustration and confusion were noted. The few community-based activities that have been successful are those that are closely linked to the culture and to income generation, particularly the deer dancers and the community-based natural resource management activities. The three impact indicators related to social, human, gender and institutional developments are rated as moderately unsatisfactory. Conversely, activities that were more intensively supported at the latter stage of the project through the rural finance activities, or producer organizations such as the cacao production and honey production generated more benefits. The four indicators related to economic development are rated as moderately satisfactory. Consequently, overall the project impact has been rated as moderately satisfactory.

Physical assets have been improved through the road infrastructure improvements but outcomes are limited. The project made minimal contribution to improve food security, despite interests of community to strengthen and diversify food sources. Social capital development through training and skills development was exceptional in the case of the organic cacao industry where traditional knowledge and technical skills combined to generate a quantum leap forward in industry development.

Other aspects of training and skills development were dismal. Training was inappropriate, inappropriately delivered and in the main, was not followed-up or assessed. Activities designed for institutional development only showed noticeable outcomes for the TTCU and the TCGA. Both organizations had long-term support from the project and this is clearly displayed in the robust growth of both organizations. Human capital improvements were again achieved mainly in the case of rural finance and the cacao industry.

Agriculture development impact in terms of basic crop production, has been low in comparison to expectations. Yet, the commercialization of the cacao production has had a very positive result in encouraging agricultural production on individual farms that is consistent with the culture and current production systems as well as being commercially viable. Financial assets have been strengthened through the increased access to rural finance and as a result of the income generated through cacao production. Access to markets has also been increased, particularly for cacao and livestock. For most other sub-projects, the marketing aspect has been neglected. Few of the sub-projects implemented directly targeted environmental protection. Yet, those targeted at ecotourism, organic production or tree planting have demonstrated positive outcomes.

Sustainability, innovation and replicability

The likelihood that the small initiatives supported at the community level will be sustained or replicated is varied. Few of the activities reviewed by the evaluation are considered as fully sustainable. Just over 50 per cent were considered moderately sustainable15 . The main thrust for sustainability will be realized through the cacao industry where the known market potential is considerable and the system is already fully operational. Few project initiatives were innovative, yet there are a few successful innovations that will be both sustained and have potential for replication resulting in a rating of moderately satisfactory. The cacao production and the complementary organic production system is an improved traditional process that is now being replicated by most of the TCGA members. There is potential for using the system for other organic crops. Some initiatives commenced by CARD have the potential for implementation/scaling up under the current European Union (EU)-financed Belize Rural Development Project (BRDP). The BRDP design has incorporated some lessons from CARD and yet is still experiencing similar implementation issues. The communities visited during the evaluation already perceive it as a continuation of previous failures – a project that has ostensibly been designed for them but to which they cannot access.

Overall assessment

CARD has been largely perceived in Belize as an unsuccessful project by the Government and by the beneficiaries. The investments in the communities did not generate the expected performance in terms of outputs. Overall, the magnitude of project benefits for the communities has been below expectations. Yet, there were several initiatives that were rated as highly satisfactory and which have far reaching potential for southern Belize.

Aggregations of the impact, sustainability and innovation ratings show that where outputs were generated, results were positive. The overall project achievement for CARD is assessed as moderately satisfactory.

The table below shows the rating of the CARD project.

Performance ratings of the CARD project16

Evaluation Criteria

Ratings

Relevance

5

Effectiveness

3

Efficiency

3

Project Performance */

3.6

Impact

4

Sustainability

4

Innovation, Replication, Up-scaling

4

Overall Project Achievement **/

4

Performance of IFAD

3

Performance of CI

3

Performance of Government

3

Source: IFAD evaluation mission 2007
*/   This is an average of relevance, effectiveness and efficiency.
**/ Determined based on the ratings for relevance, effectiveness, efficiency, impact,
sustainability, innovation, replication and up-scaling. 

Conclusions

Project success was seen where suitably qualified staff members with practical experience in field-based rural development were appointed, where community interests were supported, and where ready markets were available. Although the design was assessed as valid, there were several major gaps in project implementation that were instrumental in the poor performance including: (i) poor human resource management; (ii) a fragmented, externally driven approach, rather than the recommended strategic, phased approach; (iii) lack of timely and sustained follow-up on project interventions; and (iv) ineffective monitoring and evaluation.

There were other weaknesses in the project, but if these four aspects had been adequately addressed, most of the other challenges could have been more easily overcome.

Lost opportunities. Cases of lost opportunities abound throughout CARD implementation. Some sub-projects stimulated activity but when support was not sustained, momentum declined. Provision of sub-projects that had not been identified or prioritised by the communities resulted in loss of ownership. The model proposed for the community development aspect was never adequately implemented and therefore at this stage cannot be correctly judged as successful or not.

Access to markets and market potential. Marketing support was seen as a major need by both groups and individuals; by men and women. Once the markets have been opened, producers were ready to actively invest their own time and resources in production. Poor to no results were seen in all supply-driven initiatives where there was no specific market and/or marketing process identified. One of the reasons for the exemplary success of the cacao production was because there was a confirmed buyer for the production that was not only willing to buy, but who was also an active partner and financier in the cultivation process. There are other products in the south, particularly organic products that have substantial market potential, and which are appropriate to the local culture and environment. More could have been done to harness that potential.

Local institutional development. CARD has left an institutional legacy amongst the IFIs and TCGA that is likely to be sustained. Long term inputs to producers' organizations, particularly the TCGA, yielded multiple benefits in social capital generation as well as in economic and institutional development. The credit unions were an effective mechanism for developing rural credit for small farmers in the south and have potential for further growth. Yet they are still at a vulnerable stage of development. The development of the local community institutions is still well below that envisaged at appraisal.

Community group expectations were unduly raised due to the repeated failure of sub-projects to deliver on the expectations that had been raised. The process of planning in the community was taken as a "promise". Training and skills development did not meet the needs of the groups.

Complexity and accountability. The CARD structure was too complex and was unable to function effectively. Decision-making was too centralized and remote from the operations of the project. The role of the PSC should have been to support and enable, rather than control and manage. Lack of orientation and accountability processes installed in the project decision-making committees, and between stakeholders meant that each did not adequately fulfil their role, or require that the project fulfilled its role effectively.

Similarly, a project must maintain supervision over the delivery of services, must check with the project participants to ensure satisfaction with the service and follow up to ensure that the service yields the required results. The partnership arrangements between GoB, CDB and IFAD were appropriate but communication and lack of clarity in roles affected the responsiveness of the partnership.

Outstanding policy issues. There were larger national issues which impacted upon the project and were not addressed. In particular, land tenure issues were identified in the logical framework as an important factor that would influence the eventual benefits of the project. Land speculation is increasing across the country. Those with more capability to access and pay for lands are leading the processes of surveying land with a view to obtaining land titles. The indigenous communities in particular fear loss of traditional lands and elements of conflict over land. Observations during the evaluation highlight that this is an escalating issue1 .

Similarly, recognition of ethnic diversity, and its relation to poverty was an issue raised by both the Government and the community. Some activities of CARD demonstrated effectively how cultural practices can be harnessed for economic good, and how ethnic diversity can be seen as a potential rather than a challenge. Lack of decentralized decision-making in both governance structures and support organizations creates a barrier to development for the south.

Recommendation 1

Investment in the southern districts of Belize. The advances made through CARD show that southern Belize has potential that is firmly in line with the GoB strategic objectives towards economic development through niche export marketing and ecotourism as well as in poverty reduction. The unrealized potential that was present in the south at the commencement of CARD remains unrealized but still realizable.

There are a number of social and economic assets developed through the project which still require support to ensure that the full benefits from project investments can be realized. There is a need for detailed negotiations with the BRDP to definitively assess whether or not these priorities can be accommodated. At present, there is a general statement that this will occur but no mechanism for transfer of the knowledge from one project to the next.

Niche marketing in organic products has substantial economic potential for the south and requires further investment. This requires an enabling environment by the Government including: careful consideration of developments that may compromise the organic certification of the area; continued support to the TCGA, possibly in extending their activities to market other organic production; and/or establishment of similar producers organizations for other organic products; facilitation of export approvals and other documentary requirements for export of organic products.

Recommendation 2

The reflows from the Rural Credit Fund need to be quarantined for continued support to the two participating credit unions. The Belize Credit Union League (BCUL), with the Central Bank, is likely to play a major role in future nurturing of the credit unions in the southern districts.

Policy dialogue is required to support the professional development of the credit unions' needs. There is still additional credit funds required. Given the positive performance of the credit unions and the identified issues concerning future development, the GoB can assist in further development in the southern districts of Belize by ensuring that the funds are available to the south in perpetuity.

Recommendation 3

Realistic implementation. There are a number of recommendations in relation to improved support to development projects in Southern Belize. These include: (i) intensive support at the start-up of a project to ensure that the design is effectively translated into implementation; (ii) more emphasis at the start-up of the project to install and operationalise an appropriate monitoring system that will respond to the management information needs of the implementers, partners, and assist in detecting when a project deviates from its planned approach; (iii) there is a need for future project initiatives to clearly state intentions and gauge and manage the level of expectations generated; (iv) the groups to be financed need to either be in existence prior to the project and/or have specific self-generated objectives and a common cause. The methodology that requires communities to form small groups specifically to access funds is not viable; and (v) long-term investment, similar to the one given to the TCGA, should be provided to develop local institutions - both local governance structures and specific interest groups - which is likely to yield better results than short-term fragmented sub-projects.

Recommendation 4

Investment in people. Improved communication mechanisms and opportunities for cultural exchange need to be encouraged so that there is an increased mutual understanding of the different cultures that comprise the national identity of Belize. Supporting cultural initiatives like the deer dancers and the community-based national park management initiatives contribute to economic benefits as well as transforming cultural differences into an advantage rather than a challenge and there are other opportunities that can be similarly developed. Future initiatives must not ignore the traditional processes but can learn from and integrate the traditional processes, melding them with available technologies to provide lasting changes in agricultural systems. It is not a matter of stimulating these processes, but rather of supporting the evolution that is already occurring.

Policy level review in decentralization is needed to assess the representation of different ethnic groups in decision-making for their own development and the relationship between the roles and responsibilities of different government and non-government institutions needs more definition. At the same time, there is a need for community leaders, both traditional mayors and local councils to receive more orientation, training and basic tools for leadership. Lessons learned from other countries are that the connection of indigenous people to land and other forms of self-determination is a sensitive yet highly important issue. It is not an issue that was addressed by CARD, yet it is an issue that continues to face many of the communities that it aimed to assist.

Recommendation 5

Professionalizing project processes.Project procedures were too complex and cumbersome. The allocation for appropriate procurement procedures needs to be in line with the expected type of sub-projects. Staff and communities must be guided by an overall strategy that is "user and culturally oriented". Introducing a skills transfer clause into the contract for a service provider can greatly increase the long term benefits to the project participants. There has been a recurring theme in project evaluations for Belize that poor recruitment processes and human resource management compromises project effectiveness. Experienced rural development implementers at the management and field level are needed to serve as mentors and provide on-the-job training to local project staff. Good practices for project management, procurement, field implementation and monitoring and evaluation could be transferred within the country. When staff turn-over does occur, the project partners need to have a commitment to proper orientation of newly appointed staff.

Recommendation 6

Productive partnerships. The partnership between CDB, IFAD and GoB was an appropriate arrangement that brought together expertise and resources towards a common objective. Some amendments in the implementing arrangements may have been of assistance to make the partnership more productive, for instance: (i) clearer distinction in the roles and responsibilities, particularly for CDB fulfilling a dual role as a financing partner of CARD, and as a cooperating institution for IFAD supervision; (ii) greater investment in establishing the PMU at the commencement of the project; (iii) more review of the specific procurement processes for CARD to fit with the small-scale procurement required; (iv) better communication mechanisms and practices to expedite critical correspondence, exchange views more regularly, increase mutual understanding of the challenges faced by the project and lead to more rapid identification of solutions.


1/ Defined as those having less than 20 acres of land.

2/ IFAD Office of Evaluation (1994), Belize Toledo Small Farmers Development Project Evaluation. The assessment of performance for TSDF at that time was "relatively poor". This has been translated in "moderately unsuccessful" (3) in accordance with current OE rating guidelines.

3/ CARD (June 2006), Audited Accounts.

4/ There were slight differences between the IFAD and CDB appraisal documents. The CDB appraisal proposed that CARD would work with the 48 poorest of the 72 targeted communities; not 54 according to the phased approach proposed by IFAD.

5/ Field work was conducted collaboratively in Belize from 7 to 25 May 2007, after a preparatory IFAD Mission in April 2007.

6/ CARD (2006), Project Completion Report.

7/ See Annex 5 for details of sample selection, assessment process and results.

8/ It also responded to most of the recommendations of the IFAD Technical Review Committee (TRC) and Operational Strategy Committee (OSC).

9/ IFAD (2005), CARD Project, Assessment of the Viability of a Project Extension.

10/ The evaluation reviewed the Feasibility Assessment for the Extension and confirmed that the case for extension was sound.

11/ CARD (2006), Project Completion Report (PCR).

12/ The Regional Strategy for Latin America and the Caribbean was produced in 2002 and highlighted strategies for empowerment of the rural poor, taking advantage of market opportunities and partnerships; these were already incorporated in the CARD design.

13/ CDB as a cooperating institution is contracted by IFAD to handle the fiduciary aspects and conduct the required supervision missions for the project. IFAD also undertook supervision missions sometimes with CDB, at other times without.

14/ Such as the case of distribution of coconut seedlings which was approved by the PSC in 2001, yet had not even been requested by the communities targeted. (See Monkey River Case Study in Annex IV).

15/ Of the 17 communities visited by the evaluation mission, nine communities were able to confirm that the activities that had been supported by the project had been partly sustained. The range of activities was varied, ranging from infrastructure (culverts installation) to livestock, sewing, and ecotourism. The definition of sustainability was merely that the impact of the specific activity could be discerned as continuing, and likely to continue. This needs to be considered in the context that the benefits to be sustained were at a low level, apart from the cacao production.

16/ As per OE's project evaluation methodology, a six-point scale has been used to attribute ratings to each of the evaluation criteria. Six-point rating scale reads as follows: 6-Highly satisfactory, 5-Satisfactory, 4-Moderately satisfactory, 3-Moderately unsatisfactory, 2-Unsatisfactory, 1-Highly unsatisfactory.

17/ Such as credit being accessed for survey of land, only to have more powerful people sequester the surveyors and submit alternative claims for land; new owners preventing villagers access to traditional hunting grounds, amongst others.

 

 

LANGUAGES: English

Development of the Puno-Cusco Corridor Project (2007)

Peru  
December 2007

Interim Evaluation

Introduction

The interim evaluation of the Development of the Puno-Cusco Corridor Project, was carried out between March and August 2006.1 This was the sixth project implemented by the International Fund for Agricultural Development (IFAD) in the Republic of Peru. It was approved by the Executive Board in December 1997 and commenced operations in October 2000 with a projected closing date of 31 December 2006 (and a possible extension of one year). The project cost was USD 30.8 million; 61.4 percent from an IFAD loan, 15.9 per cent in Government of Peru resources and 22.7 per cent to be contributed by users. The National Social Development Cooperation Fund (FONCODES) implemented the project under IFAD's direct supervision.

Major design features

The project's overall objective was to raise the incomes of the rural poor to contribute to eradicating extreme poverty. Specific objectives were to: (a) build a demand-driven market for nonfinancial goods and services; and (b) contribute to building up the market for rural financial services. The project area includes a corridor along the main road network between the cities of Puno and Cusco and lateral feeder roads, comprising 128 districts in 14 provinces (five in the department of Puno and nine in the department of Cusco). The target group included 30 000 families, half of whom were to benefit directly from the project, accounting for some 15 per cent of all rural families.

The project included three components:

  • incentives for strengthening rural markets, representing 67.6 per cent of total project cost and promoting the development of the technical assistance services market by: (i) transferring resources to users to hire technical assistance and training services; (ii) providing non-reimbursable funding for community investments in business development; and (iii) providing business development services to improve user access to business information and opportunities;
  • rural financial services, for 19.5 per cent of total project cost, to strengthen local financial agencies, create a fund to make loans to users, set up a guarantee fund to reduce risks for financial institutions and promote the adoption of new technologies and services by financial operators; and
  • project administration, monitoring and evaluation, at 12.9 per cent of total cost, for operating costs.

The project design was very relevant to the problems and potential of the rural poor in the project area, and to the priorities set by Peru's public authorities during formulation in the second half of the 1990s. Experience with prior IFAD projects was taken into account, analysis of the root causes of rural poverty was sound, project objectives were clearly stated, and actions and outputs planned for components were both appropriate to the problems identified and consistent among themselves. In addition, strategies that were innovative for Peru were put forward, such as: (a) using the economic corridor approach to define the project area (rather than the more traditional political and administrative entities); (b) focusing on economic relations between urban and rural areas and strengthening links between farmers and microenterprises in intermediate cities, rather than the traditional focus on farming activities in the rural environment; and (c) pursuing a strategy of developing the technical assistance services market and transferring resources to user groups to hire such assistance. Finally, the design properly identified the poorest rural groups and their problems, and underscored the importance of women in production and marketing.

On the less positive side, no specific strategies were proposed for each of the groups identified, on the assumption that all faced the same problem of limited access to technical assistance and financial services. This was resolved during implementation by means of specific strategies and actions to provide the poorest families and women with greater access to services and benefits under the project.

Major outcomes

When the evaluation mission took place, the project was in year six of its implementation. As of 31 December 2005, USD 17.3 million had been spent, including USD 10.9 million from the IFAD loan, USD 2 million in local counterpart funding and USD 4.4 million in cash contributions from users. The loan had been 55.1 per cent disbursed and, given Peru's public spending controls, just 44.1 per cent of the Government's counterpart contribution had been received.

During implementation the project underwent significant design changes, i.e.: (a) inclusion in the project area of La Convención Province (Department of Cusco); (b) reformulation of the financial services component to reflect changes in the environment, making it a programme to promote savings among rural poor women and competition between financial intermediaries to promote financial innovation; and (c) involvement by local governments in implementation, as a result of the decentralization process undertaken by the Government of Peru.

The main project actions and outputs were as follows:

  • Resource transfers to rural communities and groups of smallholders or microenterprises to hire technical assistance and training services and pay for study grants. As of 31 December 2005, the project had transferred a total of 11.36 million soles (USD 3.32 million) to more than 1 600 formal or informal farmers' groups to hire technical assistance, representing 45 348 individual users from 41 028 families -triple the projection in the original design-. A total of 22 389 users participated in study tours (triple the target set), and training in various areas was provided to all organizations with business plans and profiles, as well as to other groups with no need of more intense technical assistance. A total of 813 training actions took place, with participation by 12 108 men and 9 032 women.
  • Training for technical assistance providers. As of 31 December 2005, a total of 1 676 technical service providers (close to 50 per cent of those hired) had taken part in training offered by the project in various subject areas.
  • Creation or strengthening of marketing venues. Strengthening was provided to 14 existing fairs and 20 new venues were created (stockyards, agricultural and craft fairs).
  • Support for participation in trade promotion events for 1 356 users.
  • Support for registration of trademarks and certifications with advisory assistance free of charge, obtaining the maíz blanco gigante [Giant White Corn] denomination of origin, registration of 21 trademarks (with an additional 60 in process) and organic certification and health registration for eight organizations.
  • Organization of a telephone-based market information service, with 12 851 queries answered satisfactorily or very satisfactorily according to 86 per cent of respondents in a quality survey.
  • Promotion and user support for obtaining national identity cards, which benefited 13 274 people (424 young people and 13 300 adults, 5 985 of them women and 7 315 men).
  • Formalization of 406 user organizations through advisory assistance and subsidization of a portion of processing costs (significantly more than the 300 proposed in the logical framework).
  • Business development investments. As of 31 December 2005, 52 of these investments were approved, 16 completed and 36 under way. Business development investments focused on the construction of cart tracks, infrastructure for livestock fairs and markets, bridges and tourism. They took place under agreements with municipal governments, which contributed counterpart funds. Considering all 52 of these, total investment was 9.4 million soles (USD 2.85 million), of which counterpart contributions accounted for 22.3 per cent. More than 84 000 families benefited, close to double the target.
  •  Promoting savings among rural women. This took the form of incentives for opening and maintaining personal accounts at formal financial institutions in the region (Caja Rural de Ahorro y Crédito Los Andes in Puno and Credinka in Cusco). As of 31 December 2005, 3 477 accounts had been opened by the same number of women in 185 savings groups. This component is projected to reach 7 000 accounts by the project's end. Incentives totalled 476 000 soles (USD 144 000) and leveraged savers' own resources in the amount of 1 329 192 soles (USD 403 000).

Project performance

The project was deemed highly relevant because:

(a) the strategy and support mechanisms to develop the technical assistance services market were consistent with Peru's public policy;

(b) adjustments were made in view of the decentralization policies adopted during the implementation period, to work actively with local governments;

(c) access to technical assistance -one of the major obstacles to increasing incomes- improved significantly;

(d) it demonstrated the key role of savings in families' survival strategies;

(e) it showed that lack of access to credit was not a hindrance to the viability of technical changes promoted;

(f) it used learning methods (based on the principles of "learning by doing" and the transfer of knowledge and experience between farmers) that were highly effective and appropriate to the circumstances, problems and potential of the target population; and

(g) it successfully added value to the users' cultural assets and identified market potential for products with cultural content.

From the point of view of achieving its objectives, the project performed well. The number of users reached was much greater than planned (more than 48 000, belonging to 44 000 families, almost triple the 15 000 families projected in design). Actions and impact were well targeted to the poor, and the number of women users exceeded the target, as did their participation in executive roles within user groups. The objective of contributing to the development of a technical assistance services market was achieved, the users' ability to pay improved (translating into significant levels of cash counterpart funds) and new capacity was developed to select, hire, supervise and evaluate technical service providers among the more than 4 000 users (more than 41 000 families) who took part in business plans and profiles. The supply of technical assistance was strengthened by the addition of 1 000 new non-professional service providers (trained farmers), the experience of working directly with the organizations of 3 400 service providers and participation by 50 per cent of them in training imparted under the project.

In addition, the project achieved its objective of improving the marketing of user products. Many users gained access to new markets and outlets, improved their business information management and, as a result, were able to charge higher prices. The objectives of the financial services component were achieved in part. The savings programme well exceeded the target, whereas the development of innovative goods and services fell short.

The project was deemed to be efficient in the use of resources, as evidenced by relatively low operating costs, a high proportion of resources spent on users, much lower unit costs for technical assistance and training than projected in the design, and reasonable costs per savings account.

Impact on rural poverty

Project performance can be considered successful in terms of having achieved the projected impact - in some cases surpassing design targets- in other cases achieving unforeseen impact. The project had a significant impact on rural poverty in several respects, contributing to achieving IFAD's strategic objectives for the period 2002-2006 and the Millennium Development Goals. Highlights are outlined below:

  • Production and productivity of farm and off-farm activities. The project promoted the widespread adoption of new production techniques among the 45 000 users who received technical assistance and training. This enabled them to become more productive and improve product quality, particularly in livestock raised for meat and milk, guinea pig production and crafts. They diversified production and converted to commercial production patterns from traditional family consumption. Features shared by all these changes were the introduction of simple techniques, and low investment levels and operating costs, since the users themselves produced most inputs. This favoured dissemination to a considerable number of farmers not served by the project, although generally without attaining the same productivity and quality standards.
  • Incomes and tangible assets. The more than 41 000 families who received technical assistance and training in business plans and profiles raised their incomes by more than 20 per cent. Part of the surplus was invested in tangible assets, mainly in production-related assets (livestock, machinery, looms, building construction) and, to a lesser extent, to upgrade housing. A sample of business plans and profiles showed an increase in tangible assets of 17 per cent machinery, 20 per cent livestock, 30 per cent product inventories and 4 per cent other assets.
  • Food security. Higher and more stable incomes, together with increased and more varied production through implementation of business plans and profiles, made more food in greater variety available to users. Greater savings availability and security among the women who participated in the savings incentives programme also contributed to this result. 
  • Environment. The project favoured the mitigation of adverse environmental impact for several productive activities through improvements in production technology. Highlights include better waste management by small-scale tanning operations, the use of natural rather than artificial dyes in textiles, and certified organic coffee and cacao production.
  • Human capital. The project had an important impact on human capital, mainly among the 45 000 participants in business plans and profiles, who acquired new knowledge and skills in managing resources, managing technical assistance, production techniques and marketing. The project also had a considerable impact on the 3 477 women who acquired new knowledge and skills in managing savings accounts and the world of finance in general. The project had a significant impact on the self-esteem of all project beneficiaries through the value assigned to their own culture, empowerment in managing resources and technical assistance, availability of savings and the ability to deal with formal financial institutions (for women). The latter had an effect on gender relations within the home.
  • Social capital and empowerment. The main impact on human capital consisted of strengthening community and group organization and cohesion through the competitive process, resource management and women's savings groups. Some of these groups were organized around existing organizations and generated strong local support networks for dealing with emergencies and obtaining information.
  • Financial assets. The women's savings programme had a positive effect on financial assets, particularly among the close to 3 500 women (and their families) who opened savings accounts with project incentives. They saw increases in their savings rate, in their information on, and knowledge of, the financial system (rules and regulations governing financial institutions), and in access to other services provided by the formal financial system. They had more cash available for investment and consumption as a result of the higher incomes generated by their business plans and profiles, most of which were successful.
  • Market access. Through technical assistance and community investments, the project helped lower marketing costs, attract new buyers and vendors to fairs and other local commercial venues and achieve new clients on major markets, nationally (e.g. Lima) and abroad.

Project actions and impact were well targeted to the poor population. Also, the number of women beneficiaries exceeded the targets set.

Prospects for sustainability were assessed as very good overall. The changes in production have a high probability of sustainability because the project worked on areas with increasing demand, good prices and profitability, and the users improved their capacity to address new problems. The technical solutions promoted by the project are also highly likely to be sustainable because they were appropriate to economic and social realities, favouring continuity following project completion. Also, the project effected substantive permanent change in the technical services market, giving the rural poor greater access to technical assistance. In terms of ownership, prospects that project achievements will be sustainable are high because local governments and users changed their attitudes to matters such as the management of technical assistance and savings behaviour.

Performance by partners

The most important areas of coordination and concerted action among the institutions participating in the project were the steering committee and the local resource allocation committees, as well as informal interactions. The partnership created around the project functioned adequately, ensuring good participation, coordination and exchange of information on project progress among the institutions concerned. This made a positive contribution to management, problem solving and discussion of innovative aspects and their prospects for replication.

IFAD played an active role in project design and execution, which was evaluated as very successful. Of particular note were the concern to include innovations from prior projects in Peru in the design phase and the satisfactory performance of direct supervision. IFAD's performance was strongly influenced by the presence in Lima since 1995 of an office and operations manager for Peru. This made for a much closer collaboration with the project and more direct relations with the Government, implementing agencies and other international cooperation agencies working in Peru. IFAD's active part in implementation was reflected in direct supervision and representation on the project's steering committee.

Performance by the Government of Peru and its institutions was assessed as partially successful. The institution for which the project was conceived, the Development Financing Corporation (COFIDE) declined to participate. The project was proposed to several ministries until it was taken up by the National Social Development Cooperation Fund (FONCODES) within the Ministry of the Office of the President. At the time the project was entrusted to it, FONCODES was fully devoted to infrastructure, making it necessary to adapt the project to its structure and the new institutional environment. This took a considerable amount of time.

FONCODES had well defined responsibilities in the loan contract as the project's implementing institution. Its performance was evaluated as successful. Its flexibility in setting up a central implementing unit to carry out the project was noted; this lent autonomy to the use of resources and removed interference in appointing personnel and other important aspects of implementation.

The Ministry of Economy and Finance played a role principally in providing counterpart funding in a timely manner and participating in management through a representative on the steering committee. Counterpart contributions each year fell short of design projections, as a result of policies to control public spending. This translated into loan and counterpart funding disbursements at lower than projected levels (55 per cent and 44 per cent, respectively). However, this did not jeopardize the project. Resources budgeted each year were adequate given implementation capacity and were transferred as provided for in annual budgets committed by the project (except in 2004). Participation in the steering committee was active.

The project coordination unit (PCU) performed very well, thanks to continuity and experience in the coordinator position, sound (and competitive) selection of technical personnel (from the project area, having solid training and experience and mastery of the Quechua and Aymara languages) and the location of the PCU and local offices in intermediate cities in the project area (rather than Lima). That operating costs were quite low in relation to total cost was attributable to having a small PCU with just 21 people, and to a concern with keeping unit costs down.

Local governments. The local governments contributed additional resources for project actions and took part in decision-making. Funds were contributed for business development investments (25.3 per cent of the cost of work done was contributed) and to implement business plans and profiles (supplies for investments were contributed). Participation in decision-making took place through representation on local resource allocation committees, which ruled on business plan competitions, communities and savings groups. The performance by local governments as implementation partners can be considered successful, although their interest in participating and adopting innovations varied.

Overall project evaluation and conclusions

The overall evaluation of the project is satisfactory. Project design and implementation were very relevant to IFAD's strategy in the country, and applied innovations and lessons learned from IFAD's earlier projects. Objectives, strategies and actions were also very relevant to Peru's public policies and the problems and potential of the rural poor.

The project was partially effective in achieving its objectives, attaining most but not all of the stated objectives. The supply of financial services was not strengthened with new products appropriate to the target population. However, the remaining objectives and targets were reached and many of them exceeded by far. The project was efficient in its use of resources. Operating costs were relatively low, a high proportion of resources were spent on users, unit costs for technical assistance and training were much lower than projected in the design, and costs per savings account opened by women were reasonable. From the point of view of impact, the project's performance was successful, achieving not only the anticipated results but also others not planned for. Sustainability is considered likely, and performance was also successful in terms of innovation, replicability and scalability.

The number of users far exceeded plans (more than 48 000, belonging to 44 000 families, nearly triple the 15 000 families called for in the design). The great majority of project users were very poor families, both as self-defined and based on the official poverty line. Moreover, women's participation in business plans and profiles (36.6 per cent) exceeded the target (20 per cent), as did their participation in executive positions within user groups. Also, the new savings promotion line created by the project was very favourable towards women.

The project had a significant impact on IFAD's three strategic objectives (building capacity among the rural poor and their organizations, improving equitable access to productive natural resources and technology, and increasing access to financial services and markets), contributing directly to the Millennium Development Goal (MDG) of eradicating extreme poverty and hunger by increasing users' incomes (by an average of 20 per cent) and improving food security. The project also contributed to improving education (MDG 2) indirectly, by increasing savings. Being environment-neutral, the project also contributed to the MDG of ensuring environmental sustainability.

 Prospects for sustainability were assessed as very good overall, and higher than the average for IFAD projects (according to the 2005 annual report on the results and impact of IFAD operations), particularly with respect to efficiency, sustainability, innovation and performance by IFAD.

Project Evaluation Ratings

Performance Criteria

Project Ratings a

Relevance

6

Effectiveness

4

Efficiency

5

Impact b

5

Sustainability

5

Innovation, Replication and Upscaling

5

Performance of IFAD

6

Performance of the Government of Peru

4


a/ IFAD uses a scale of 1 to 6, where 1 represents the lower score and 6 the highest.
b/Breakdown of impact ratings:  Productivity, 5; Income and Physical Assets, 6; Food Security, 5; Human Assets, 6; Social Capital and Empowerment, 5; Access to Markets, 4; Financial Assets, 4; Institutions and Services, 4; Environment, 5. 

Major issues for the future and recommendations

Based on the project experience, a number of issues and lessons were identified in connection with strategy and implementation mechanisms. Strategic considerations include:

  • Successfully introducing innovations requires a combination of factors: adoption of a long-term approach by IFAD through continuity in the strategy and its demonstration effect; intervention by a committed sponsor with access to decision-making bodies in the government; close monitoring, flexibility and adaptability in implementation
  • Good results in terms of user production confirmed that limited access to technical assistance is one of the main obstacles for smallholders and microentrepreneurs to increase their incomes.
  • Developing and strengthening market mechanisms for sustainable access to technical assistance by the rural poor population (or segment thereof) can be an effective strategy for reducing rural poverty.
  • The project experience confirmed the savings capacity of poor families and the crucial importance of savings in their survival strategy, particularly to reduce their vulnerability to unforeseen events (health, climate changes, etc.)
  • Lack of access to credit was not an obstacle to the viability of the technical changes promoted. The users relied on savings, family remittances and wage earnings to finance small investments.  
  • The development of technical assistance services can be adversely affected by the presence of institutions or other projects in the project area that operate on principles contrary to the market, e.g. providing free services indefinitely, with no profitability requirement and with no user participation in contracting and supervision. It is advisable for a project of the nature of the Corridor Project to be preceded by public policy commitments relating to technical assistance services in order to avoid any contradictions.
  • The savings promotion programme demonstrated the importance of a favourable context (low inflation, positive real interest rates, economic stability, banking legislation incorporating entities with different formats and diverse clientele, a microfinance system with a strong regional presence, the existence of deposit insurance). It also showed the importance of information on these favourable elements in generating confidence among savings programme participants.
  • Decision-making and management processes have an important influence on impact in terms of rural poverty. The transfer of resources, control in managing technical assistance and competitive recruitment can in themselves have more of an impact on human and social capital than traditional projects components such as training and technical assistance.

With regard to implementation mechanisms:

  • Applying the principles of "learning by doing" and transferring experience and knowledge among farmers was highly effective in promoting learning on production issues. This suggests the possibility of taking advantage of existing potential for innovation within the rural poor population itself, working through study tours and exchanges to disseminate advances and success stories, and working with "expert farmers" as low-cost, high quality non-professional service providers.
  • Resource transfers to users to hire technical assistance proved an effective way of promoting learning on managing resources and technical assistance. This mechanism should be complemented by a real transfer of power to users, without interference by the project or others, in selecting, hiring and supervising technical service providers, and by a cash counterpart contribution requirement to promote a feeling of ownership.
  • The use of public competitions with representation by local stakeholders and users can both be an instrument for transparent resource allocation and generate a positive impact on human and social capital.
  • Users' technical assistance needs must be well defined to avoid generalities and lack of definition that can lead to services being provided for years with no focus on specific problems, yielding poor results and therefore costly and inefficient. The business plans or profiles used by the project, or similar instruments, together with deadlines set for implementing them and increasing counterpart percentages for new applications, provided an incentive for hiring technical assistance based on well defined needs and oriented towards achieving economic results within defined timelines.
  • The project's experience with the savings promotion programme suggests that it is important for similar initiatives to combine group work with individual accounts as a way of promoting collective action and social capital, rather than focusing on the advantages of individual savings (greater savings incentive, higher self-esteem).
  • The project demonstrated that identifying the potential of the users' cultural capital can generate excellent results in terms of new goods and services with a high value that take advantage of specific market niches (adventure tourism, crafts, obtaining copyright for design, registration of traditional products). At the same time, valuing cultural capital in implementing a project (e.g. by giving preference to the Quechua and Aymara languages and traditional dress in competitions or by favouring Quechua and Aymara speakers in competitions) can make a substantial contribution to social capital and self-esteem and other aspects of human capital, which in turn can contribute to a potential positive impact on other aspects of rural poverty.

Based on the evaluation findings, the mission concludes that IFAD and the Government of Peru should take advantage of this valuable experience and its promising results. The following initiatives, inter-alia, could be undertaken:

  • Gain a better understanding of positive results by analysing the contributing factors, including sustainability. This could be done through studies or additional pilot experiences to test specific models or mechanisms. For instance, a detailed analysis could be done of successful organizations and technical service providers having maintained linkages after project support ended, or of local governments having applied innovations promoted by the project, thus improving their ability to serve their populations. The optimal scale of savings incentives warrants study as well. IFAD should allocate resources in support of these efforts to the extent possible.
  • Discuss the usefulness and possible incorporation into public policies of the project experiences and mechanisms, including an analysis of comparative advantages and possible obstacles such as contradictions between different public programmes and projects as to whether payment is required for technical assistance services or counterpart contributions are required for services.
  • Stimulate participation by all those working on rural development -public institutions, non-governmental organizations, universities, professionals- in researching project mechanisms and strategies, as well as in discussions on their usefulness. This could be achieved through broad dissemination of project results in the form of a series of publications and events.
  • Promote greater sustainability of achievements in terms of access by the target population to technical services through the establishment of partnerships between farmers' organizations and technical service providers to enable their self-sustaining access to technical assistance.

1/ The mission visited Peru from 13 to 30 March 2006; overall coordination of the evaluation was provided by Miguel Torralba, IFAD Evaluation Officer, who participated in the start and the end of the field work.

 

LANGUAGES: English, Spanish

Rural Microenterprise Development Programme (2007)

Colombia  
September 2007

Completion evaluation

Introduction

The population of the Republic of Colombia is 45.6 million (2005), of which 23 per cent live in rural areas. The country's economy is the fifth largest in Latin America, with a GDP per capita of US$2 688.1

The Government estimates that 52.6 per cent of the total population live below the poverty line, while this figure reaches 69 per cent in rural areas.

The completion evaluation of the Rural Microenterprise Development Programme was conducted by the Office of Evaluation (OE) of the International Fund for Agricultural Development (IFAD), in Colombia, from July to September 2006.

The programme was approved by the IFAD's Executive Board in September 1996, and implementation began in June 1997, with a scheduled closing date of June 2003. The closing date was first extended to June 2005, and finally to 30 June 2007, following two additional extensions. The programme cost was US$26.7 million, of which US$15.9 million was to be covered by an IFAD loan provided on ordinary terms. The Government of Colombia provided US$0.7 million in counterpart funding towards the programme. Following two amendments to the loan agreement, the total programme cost was reduced to US$20.2 million, of which IFAD's loan covered US$16.1 million and the Government's contribution, US$4.1 million.

The institution responsible for programme implementation was the Ministry of Agriculture and Rural Development and the cooperating institution responsible for loan administration and programme supervision was the Andean Development Corporation (CAF).

Evaluation objectives, methodology and process

The main objective of the evaluation was to assess the performance and impact of the programme and to generate a series of findings and recommendations for the design and implementation of similar projects in the future in Colombia. To that end, as per OE's project evaluation methodology, the evaluation of the Rural Microenterprise Development Programme sought to: (i) analyse the programme's performance in terms of its relevance, effectiveness and efficiency; (ii) assess the programme's impact on rural poverty; and (iii) assess the performance of key programme partners, including IFAD, CAF and the Government of Colombia. As per OE's project evaluation methodology, a six-point scale has been used to attribute ratings to each of the aforementioned evaluation criteria.2

The evaluation included two missions to Colombia: a preparatory mission in July 2006 to launch the evaluation and the main field mission, which visited the country in August-September 2006. The main evaluation mission held discussions, inter alia, with Government officials at the national and local level, as well as with programme staff and beneficiaries. It also had a chance to visit various programme sites during its fieldwork. In this regard, the mission visited a stratified sample of 15 associations of rural microenterprises (ARMEs) supported by the programme in three of the country's departments: Atlántico, Bolívar and Cauca. The criteria for sample selection were: geographic coverage (department), product line and the phase of support that the associations were receiving. Security conditions on the ground, access and logistics were also taken into consideration. 

Programme design

The overall objective of the programme was to increase incomes and employment in rural areas through support for the development of rural microenterprise, while ensuring gender equity and environmental protection. The programme's specific objectives were to: (i) facilitate the growth of rural microenterprise in commercial, productive, business-development and economic terms; (ii) foster the development of rural microcredit; (iii) help develop, build the capacity of and stimulate the market for the suppliers of services for rural microentrepreneurs; and (iv) generate applicable policy guidelines for the development of rural microenterprises as a strategy for combating rural poverty.

The programme had four components, namely: (i) technological services and training for rural microenterprises; (ii) financial services; (iii) institutional strengthening for first-tier financial intermediaries and suppliers of technological services; and (iv) programme coordination.

The programme covered the entire country, but four rural departments (Bolívar, Sucre, Cauca and Nariño) were prioritized initially, owing to their high concentrations of rural poverty and to the existence of favourable conditions for the development of microenterprise activities. Direct beneficiaries included some 10 200 rural families and 3 110 rural microenterprises.

Implementation results

At the time of the evaluation, the programme was in its ninth year of implementation. Estimated disbursements as of end-2006 totalled around US$18.9 million – 94 per cent of the revised overall programme cost of US$20.2 million. Virtually the entire approved amount of the IFAD loan had been disbursed, whereas 66 per cent of the counterpart contribution of US$4.1 million had been disbursed.

The programme was implemented in two stages, which were markedly different in terms of the modality and partners responsible for implementation. The first stage ran from 1997 to 2000, and the second from 2001 to 2006. The turning point in programme implementation came after an amendment to the IFAD loan in 2000, following a review mission organized by the Fund to address the difficulties encountered and the scant results produced by the programme up to that point. During the first stage, implementation was the responsibility of an inter-ministerial implementing committee, made up of the Ministry of Agriculture and Rural Development, the Ministry of Economic Development and the Mix Corporation for the Development of Microenterprise (MCMD).

During the second stage, the Ministry of Agriculture was given sole responsibility for programme implementation through the National Technical Coordination Unit for the programme, which was housed within the ministry. The new programme management committee consisted of the Ministry of Agriculture and the National Planning Department, that is, the institutions most directly concerned with rural development.

The two stages differed significantly with regard to the overall approach to programme implementation. During the first stage, for example, the technological services component was administered by MCMD, which selected training and technical assistance proposals formulated by suppliers, without the participation of rural microenterprises (RMEs). As a result, services did not always respond to needs of the microenterprises. In the second stage, the RMEs formulated their own business development plans without the involvement of other entities, and selected their own suppliers of technological services. Among other issues, an increased involvement of microentrepreneurs contributed to their overall empowerment. It also improved the effectiveness of technical services, which became more responsive to demand, and enhanced the efficiency of investments by eliminating payments for unnecessary activities and high administrative costs.

Once the MCMD was no longer involved in programme implementation, the allocation of resources for technical assistance was decentralized through regional competitive selection processes.

This decentralization also enabled the involvement of departmental governments, which promoted promising business initiatives for their regions, supported the RMEs in formulating their business development plans and cofinanced some projects. For the past few years, microentrepreneurs have been participating in the pre-selection process, contributing their practical knowledge and experience.

Through the financial services component, the programme provided loan funds to financial operators that were not regulated by the Superintendency of Banks, an option that was not available to the Fund for Agricultural Financing (FINAGRO). In addition, loans to RMEs not assisted by the technological services component were authorized, thus avoiding the difficulties often associated with tied-credit programmes.

The activities carried out under the programme were: cofinancing of training and technical assistance for microenterprises (technological services component); loans to microentrepreneurs and incentives for capitalization (financial services component); strengthening of suppliers of technological and financial services (institutional strengthening component); and additional assistance for the execution of microenterprise projects, such as participatory monitoring and evaluation, and assistance in the design of policies and instruments for rural microenterprise development (programme coordination component). A succinct overview of some of the key results under each component is provided below.

Technological services

Efforts to improve RMEs focused on four major areas: (i) introducing modern production technologies; (ii) increasing market access for RMEs; (iii) improving business management; and (iv) organizational strengthening of ARMEs. Support was provided following a gradual strategy in four annual phases; 69 per cent of RMEs received support only in the first phase, 19 per cent over two phases, and the remaining 12 per cent over three or four phases.

The Rural Microenterprise Development Programme cofinanced 199 projects to support the development of 380 ARMEs. The overall number of microentrepreneurs assisted totalled 20 167. Projects were carried out in 22 departments, covering most of the country. The cost of the projects averaged approximately US$43 000. The implementation rate rose from 3 projects in 1999 to 68 projects in 2006. A total of 20 167 individuals were reached (62 per cent of whom were men and 38 per cent, women.

Financial services

Activities under this component included the selection and contracting of 12 financial operators and the administration of resources for the Credit Fund and the Capitalization Fund. In addition, an agreement for administration of funds was signed with FINAGRO, as a second-tier bank.

The operations of the Credit Fund and of the Capitalization Fund began in 2001 with the establishment of loan funds with the financial operators, which were selected through a competitive process. Initially, the resources were disbursed directly by the programme, but since 2006, FINAGRO has administered the funds.

As of June 2006, 4 992 loans had been extended (US$3.59 million) and there were 3 397 active clients under the programme. Lending increased each year after 2001, in terms of both number of loans and amounts. The peak lending years were 2005 and 2006.

Institutional strengthening

A total of 160 technological service suppliers were identified, and their contact information was published in a "talent map" (available on the programme's website)3. The capacity of the financial services operators was strengthened through two activities: (i) an international seminar on microfinance held in Bogotá in 2005, and (ii) a study visit to Bolivia in 2006.

Programme coordination

The main outputs of this component were: (i) the production of eight reports documenting experiences under the programme; (ii) support for the drafting of two policy documents4 by the National Planning Department; and (iii) the implementation of a programme monitoring and evaluation (M&E) system, the methodology and instruments of which were transferred to the RMEs, to build their M&E capacity.

Performance of the programme

Relevance

The evaluation considers the programme to be highly relevant (with a rating of 6), as its objectives and its strategy for rural poverty reduction through support for rural microenterprise were effective in meeting the needs of the target population. The programme also helped overcome the agricultural crisis that Colombia was experiencing in the 1990s and supported efforts aimed at eliminating illicit crops. The programme addressed the weaknesses of the RMEs, helped to empower microentrepreneurs and encouraged the development of technological and financial service markets. It is also proving useful to the Government for formulating national policies for rural development, particularly those in the subsectors of rural finance and rural microenterprises, to the extent that it has been identified as a "flagship programme" by the Ministry of Agriculture. Last but not least, the programme is in line with the broad objectives contained in the Colombia country strategic opportunities paper, which was prepared by IFAD in 2003.

Effectiveness

The evaluation considers the programme to be effective with a rating of 5. It has successfully achieved its objectives, for example, in relation to the strengthening of RMEs, development of rural microcredit and stimulation of the market of service providers in a number of areas including production, marketing and management.

The strengthening of the RMEs translated mainly into growth in sales. This was observed in 88 per cent of the RMEs supported, which exceeded the target of 70 per cent. At the same time, 26 per cent of the RMEs increased their assets, although this result fell short of the target of 70 per cent. Additional indicators used to assess RME growth were:

Productive development. The vast majority of RMEs improved their productive processes from a technological standpoint; however, little headway was made with regard to diversification of production.

Organizational development. Gains were made in transparency and participation, thanks largely to the implementation of participatory monitoring and evaluation systems. The ARMEs that received the most phases of support made the greatest progress in this area.

Commercial development. There was significant integration of RMEs into regional production chains, particularly among RMEs undergoing their fourth phase of support (92 per cent).

Market orientation. This was key to changing the mentality of the microentrepreneurs, who adapted their production to market requirements and improved the quality of outputs as a result.

Business development. The programme helped to enhance competitiveness in response to changes in consumer behaviour and technological progress. The principal indicators were: (i) maintenance of financial records, which was noted in 68 per cent of the ARMEs and in a fourth of the RMEs; (ii) yearly operational planning, noted in 62 per cent of the ARMEs, and strategic planning, noted in some cases; (iii) quality control, noted in virtually all the ARMEs; and (iv) joint marketing, noted in 70 per cent of the RMEs.

The programme contributed to the development of rural microcredit by facilitating the entry of nine new financial operators into the rural microcredit market. These financial operators were also active at the time of the evaluation (60 per cent of the target of 15 operators envisaged at appraisal). A total of 4 992 loans were extended amounting to US$3.59 million, and in June 2006 active clients totalled 3 397 (exceeding the target of 3 000). The financial operators provided timely access to credit (disbursement times ranged between three and seven days, less than the target of ten days), with enhanced gender equity (58 per cent of the loan recipients were women, exceeding the target of 40 per cent). The financial operators also managed their loan portfolios effectively. This is evidenced by the low percentage (2.8 per cent) of the total portfolio-at-risk, in terms of outstanding loans exceeding their repayment date by 30 days.

Nevertheless, a high proportion of RMEs did not receive loans, mainly due to the fact that the financial operators did not know them and to the lack of coverage in areas located a long distance from the agencies. In addition, it is clear that a line of credit for commercialization – crucial for ARMEs – should have been implemented.

During implementation, FINAGRO became a new source of funding for financial operators interested in microcredit lending, thanks to the creation of its rural microfinance unit, which is currently administering the Credit Fund. Another unforeseen outcome was the creation of ten self-managed revolving funds administered by the ARMEs, with total capital amounting to US$95 489.

On a related topic, the market for suppliers of services to RMEs was stimulated and strengthened by the identification of providers, the facilitation of contacts with the beneficiaries through competitions, and the addition of new suppliers (some RMEs became service suppliers). A total of 148 ARMEs (far above the target of 30) identified and contracted suppliers of technological services in an openly competitive market. In addition, nine financial operators acquired rural microcredit technologies and expressed willingness to apply them; however, at the time of the evaluation, they had not done so and continued to employ urban microcredit technologies (possibly because of the time required to change the software used for loan management and to set up agencies and/or representative offices in rural areas).

With regard to the formulation of policy guidelines for the development of RMEs as a strategy for combating rural poverty, eight reports were produced documenting experiences under the programme. Assistance was also provided to the National Planning Department in drawing up the Opportunity Bank policy document5.

Efficiency

The programme was found to be highly efficient, with an evaluation rating of 6. For example, the cost of support per family (US$936) was approximately half that of other IFAD projects in Latin America and the Caribbean, which is attributable both to the RME development strategy and to the efficient implementation of the components. Operation cost was 16 cents per United States dollar of products and services delivered to microentrepreneurs – very efficient both in absolute terms and in comparison with other projects in the region, where operation costs are generally about double that amount.

Performance of the partners

IFAD

The Fund's performance was highly satisfactory, earning an overall rating of 6. The IFAD country programme manager for Colombia monitored implementation closely, in collaboration with Government authorities and the cooperating institution. Twenty monitoring missions were conducted by IFAD, most of them led by the country programme manager, resulting in two amendments to the loan agreement and specific recommendations for improving programme performance. IFAD demonstrated flexibility, ensuring that the necessary corrections were made in a participatory and timely manner. The support provided by IFAD's regional grants6 was highly beneficial.

Government of Colombia

The Ministry of Agriculture was the implementing agency. Other institutions involved were FINAGRO, the National Planning Department and the Directorate of Public Credit (an agency of the Ministry of Finance and Public Credit). Between 1997 and 2000, MCMD implemented the technological services component, and during that period budgetary allocation by the Ministry of Agriculture was clearly insufficient. After 2001, the ministry increased budgetary allocations steadily, encouraged regional coordination and designated the Rural Microenterprise Development Programme a flagship operation. Consequently, taking its overall performance into account, the evaluation considers the Government's performance as satisfactory, with a rating of 5.

FINAGRO

FINAGRO initially had legal difficulties in administering the Credit Fund, but succeeded in resolving those problems early in 2006. In the short period since then, lending has increased to its highest rate ever, and the Rural Microfinancing Unit has been set up. FINAGRO's performance is therefore rated as satisfactory (rating 5).

Cooperating Institution

CAF administered the IFAD loan and supervised implementation. At first, loan administration did not operate smoothly, which caused financial difficulties for the programme. However, the situation improved over time and the period needed for replenishing programme funds decreased substantially. Supervision comprised 16 multidisciplinary missions during the programme period, which were carried out in a participatory and conscientious manner. CAF's performance is rated as satisfactory (5).

Inter-American Institute for Cooperation on Agriculture (IICA)

IICA served as fiduciary agent for the programme, and was responsible for procurement. Its performance was adequate, conscientious and effective. In addition, IICA provided technical assistance for programme implementation, promoting a gender perspective and assisting in the organization of business round tables held within the framework of the programme. In 2006, IICA assumed the responsibility of auditing several projects under the programme. Its performance is therefore considered highly satisfactory (rating 6).

Impact

Rural poverty impact

The programme had significant impact in several areas, in particular on human assets and social capital. Improvements were noted in the ability of microentrepreneurs to run their businesses, especially in productive, managerial and commercial terms. In addition, the transfer of decision-making responsibility for the selection and contracting of services and for the management and administration of public resources helped to empower microentrepreneurs vis-à-vis their customers, their suppliers and the Government. The programme also helped to build capacity among community organizations, notably through the organizational strengthening of 159 ARMEs, which established effective and transparent managerial bodies – many incorporating young people – and participatory monitoring committees.

On the other hand, the programme only partially achieved its expected impact on family incomes, and the compliance of the main environmental pollution processes with national environmental legislation (vis-à-vis disposal of untreated wastewater, and waste and rubbish collection) has not been ensured. It is therefore rated as moderately successful (with a rating of 4).

Sustainability

According to the evaluation, the programme is likely to be sustainable, with a rating of 5. In this regard, the evaluation noted that the ARMEs have strengthened their technical capacity and social organization (for example, the evaluation noted greater participation in decision-making and more transparency in their governing and managerial bodies) and that they are working in expanding industries (for instance, virtually all of the microenterprises studied are profitable). In addition, within the framework of the strategy of developing production chain systems at the regional level, the RMEs are closely linked to one another and/or are in a direct business relationship with other more powerful agents in value chains. The programme contributed to creating an institutional fabric – which includes regional governments and private agents favourable to the development of RMEs – along with a clear sense of identification with the programme's objectives by everyone involved. Stimulating the technological services market enhanced the interaction between supply and demand, encouraging improvement in the quality of services and ensuring their continuity. Administration of the Credit Fund by FINAGRO will guarantee access to credit until 2014, but growth will be limited since FINAGRO can only disburse external funds to unregulated entities; hence, any increase in funds can only come from the resources of the financial operators.

Innovation, replicability and scaling up

The strategy of reducing poverty by focusing on the development of RMEs is innovative, both for the Government of Colombia and for IFAD in Latin America and the Caribbean7. Although conducive conditions for this new approach existed during the design phase, the Government was not really prepared to implement the strategy because its main experience lay with urban microenterprises. The programme introduced specific approaches, organizational mechanisms and processes, which today constitute a proven strategy for supporting rural microenterprise in Colombia.

In particular, empowering microentrepreneurs to set their own priorities and identify their own needs is an innovation that offers significant strategic potential and lays the foundation for an inclusive, participatory and dynamic process of rural development. The decentralization of the process for preselecting projects for cofinancing through regional competitions, the participation of regional governments and of the microentrepreneurs themselves in the preselection committees, and the introduction of a rural approach to microfinance in Colombia are also important innovations.

Programme approaches can be scaled up, thanks to the degree of credibility enjoyed by the programme among concerned government authorities. Indeed, the idea of direct transfer of funds to beneficiaries (without the involvement of NGOs) is already being discussed within the Ministry of Commerce, Industry and Tourism with an eye to incorporating it into the ministry's support programme for urban microenterprises. The programme also offers lessons for all actors in the Colombian financial system with regard to policies to support rural financial institutions. The reforms proposed in the Opportunity Bank policy document provide clear evidence of the willingness to change and reform in order to facilitate access to the formal financial system by the rural poor, who have not benefited generally from such services in the past. On the whole, the programme is considered successful in terms of innovation and opportunities for replication and scaling up. The rating for this evaluation criteria is therefore 5.

Overall assessment

The table below shows programme ratings compared with the average ratings presented in the annual report on results and impact of IFAD operations (ARRI) in 2005.

Rural Microenterprise Development Programme performance compared with the average performance ratings presented in the 2005 ARRI report

 

  Programme score ARRI 2005

Programme performance

   

Relevance

6 5

Effectiveness

5 4

Efficiency

6 4
     
Impact    
Physical and financial assets 4 4
Environment and natural resources 3 4
Human assets 6 4
Social capital and empowerment 6 4
Institutions and policies 5 4
     
Overarching factors    
Sustainability 5 4
Innovation, replication and scaling up 5 4
     
Performance of the partners    
IFAD 6 4
CAF – cooperating institution 5 4
FINAGRO 5 4
IICA 6 4

The Rural Microenterprise Development Programme has been successful, with an overall rating of 5. It is considered to have been very relevant in relation to the needs of the rural poor and is in line with the country strategic opportunities paper for Colombia. The programme was also effective and very efficient, especially in comparison with other similar operations. In terms of impact, it was moderately successful, making evident progress in several areas, including, notably, human assets, social capital, capacity for action of the ARMEs, market access and empowerment of women, although the increase in family incomes was not as high as expected and there were shortcomings with respect to the environment. There is strong evidence to suggest that the development processes initiated under the programme will be sustainable.

 

Summary of ratingss – The IFAD

Star

The programme contributed significantly to the achievement of IFAD's strategic objectives in Colombia, strengthening the capacity of 17 000 rural poor people and their organizations to play an active role in society and in the overall economy, and improving their access to technology, financing and markets under competitive conditions.

Conclusions and recommendations

Conclusions

The programme's objectives and the strategy for rural poverty reduction through support for rural microenterprises were timely, coinciding with Colombia's agricultural crisis in the 1990s, and apposite, in the light of the high percentage of the rural population (57 per cent) dedicated to microentrepreneurial activities.

Results were poor during the first stage of the programme (between 1997 and 2000) as a result of highly centralized and supply-driven programme design, which was conceived within the framework of the urban-oriented National Plan for Microenterprise Development.

The loan was amended in 2000 following a review mission organized by IFAD. RMEs would now formulate their own business proposals.

The increased involvement of microentrepreneurs contributed to their overall empowerment. It also improved the effectiveness of technical services, which became more responsive to demand, and it enhanced the efficiency of investments by eliminating payments for unnecessary activities and high administrative costs.

The programme contributed to better access to microcredit, but coverage is still limited and the range of financial services inadequate to meet the needs of RMEs.

The approach to business development as a strategy for poverty reduction was successful, and the operational strategy – giving microentrepreneurs a leading role in their own business development processes – was innovative within the Colombian context. The programme offers encouraging experiences for replication, particularly in view of the efficiency of the investment.

The evaluation notes that there are a number of key issues that should be carefully considered in the design and implementation of similar future projects and programmes.

Strategy

Genuine empowerment of the beneficiaries and their organizations as part of the strategy of business strengthening represented a major methodological innovation for the development of programmes to fight rural poverty.

Decision-making and information mechanisms

The programme has shown that the poor are capable of making their own decisions, the evidence of which was the microentrepreneurs' demonstrated ability to identify their own needs in their business development plans, to select and contract suppliers of technological services, to achieve economies of scale through their associations, and to tailor their production and their businesses to market demand.

The Government should gear its efforts towards providing the elements that microentrepreneurs need to manage information on markets and on their businesses.

Allocation of public resources

The allocation of public funds through regional competitive processes is another approach to be considered in the future. Not only did this enhance the effectiveness of public investment – by prioritizing the best projects – but it also improved the efficiency of the investment by ensuring that resources were channelled to relevant, timely, high-quality activities. The formation of preselection committees for regional competitions, whose members included regional government authorities, successful microentrepreneurs and subject-area specialists, demonstrated the advantages of decentralizing decision-making and involving the private sector.

Integration into production chains

The strategy of linking the programme-supported RMEs to production chains has made it possible to incorporate them into more dynamic links in the various value chains, thereby ensuring the RMEs a more stable market presence and encouraging improvements in the quality of their goods and services. This strategy has spurred the development of new microenterprises linked to the chains, notably service-providers, resulting in local business development and job creation; it has also meant that RMEs have not had to rely on technologies and processes that are generally available only in urban areas.

Cooperative and decentralized relationships

The involvement of a variety of public and private actors operating in the regions prioritized by the programme was very effective and led to productive interaction based on cooperative partnerships. The various stakeholders – microentrepreneurs, public institutions and service providers – were linked together in networks that broke with the traditional vertical pattern of a state that gives and a population that receives. There is scope to strengthen horizontal linkages and to continue deepening and decentralizing activities by increasing the involvement of public and private entities in the regions in activities that are currently being supervised directly by the National Technical Coordination Unit.

Market orientation

It is worth noting the importance of changing the mentality of the RMEs and their organizations, which have moved away from a historically "productivist" orientation. RMEs now examine the market before embarking on any production activity in order to improve their chances of success and sustainability. They are also adopting business practices that will enable them to compete on the market.

Rural financial services

The programme banished several misconceptions about rural financial services, showing on the contrary that:

  • Rural microentrepreneurs require credit in order to sustain commercial development processes, since they must finance their access to regional and national markets, accommodating delays in payment for products while still maintaining adequate cash flow in their businesses;
  • Microentrepreneurs are good clients of the financial system – the portfolio-at-risk greater than 30 days was 2.8 per cent – and they generate broad and diverse demand. This is attributed to the clarity of the terms of business established with the financial operators, which discouraged the culture of non-payment often found in government-run tied credit programmes; and
  • Lending to rural microentrepreneurs was profitable for the financial operators, even though they had to extend their coverage into rural areas, which raised overheads.

Recommendations

Recommendation 1:  reinforce support mechanisms for RMEs

Information mechanisms. Enhance the availability of business and market information to support both RMEs and providers of technological services through the use of the various systems that have been implemented by the Ministry of Agriculture. The challenge is to adapt these systems and make them more accessible and user-friendly, bearing in mind the knowledge level of the microentrepreneurs.

Horizontal linkage. Make a greater effort to create horizontal linkages among the microentrepreneur beneficiaries of the programme with a view to achieving better synergy among peers and adding to the reservoir of business knowledge that they have developed together. The organization of meetings, thematic workshops and study visits might be an efficient means to that end. It might also be worthwhile to consider creating a rural enterprise observatory, with the support of IICA, given its experience in setting up other observatories.

Early warning. Develop a more effective online system (an early warning system) for monitoring the progress of RMEs, in terms of both commercial and financial variables, in order to identify promptly any problems that they are facing and thus lower the "mortality rate" among rural microenterprises.

Social action. Emphasize social activities as a key component of strategies aimed at organizational strengthening. The equation "organization equals business" is generally very unstable. Given the potential for business failure in the face of various challenges (for example financial, commercial or external shocks), carrying out activities of a social character will give organization members a greater sense of value and ownership. Some of the organizations visited by the mission have made a major effort in this regard.

Recommendation 2:  strengthen financial services

Diversify the supply of financial services. Strengthen financial operators in order to enable them to diversify their supply of services and begin offering working capital and marketing loans, collective loans, savings accounts, insurance and fund transfers. Implement a programme for the development or transfer of methodologies to diversify financial products, employing a multiproduct approach based on detailed analysis of demand in each region.

Expand the coverage area of financial operators. Extend the coverage of agencies of financial operators to remote rural areas currently not being served by: (i) promoting partnerships among financial operators and consolidated ARMEs that operate self-managed credit funds. These associations have experience in the administration of loan funds and they can partner with financial operators in preselecting loan applications, linking collections of loan payments to deliveries of products and consolidating payments in bank accounts of the financial operator; and (ii) promoting partnerships through non-bank representative offices. The cost of establishing such an office is infinitely lower than that of opening up an agency. An incentive system for expanding rural financial services could be implemented by either, the Ministry of Agriculture, the programme or another institution or project through the establishment of representative offices and partnerships with associations of producers.

Stimulate the participation of savings and loan cooperatives in rural microcredit, since these institutions have a strong regional identity and enjoy a great deal of trust among the public. They are service-oriented and they have shown themselves to be very effective in assisting microentrepreneurs, provided they have access to product methodologies and improved technologies for the management and processing of information. They require specialized technical assistance programmes and support for horizontal and vertical integration processes8. Another advantage of cooperatives is that they can offer virtually all the services provided by banks, except for checking accounts, but their costs are significantly lower.

Specific recommendations for FINAGRO:

Improve FINAGRO's current capabilities for the management of financial operators through the implementation of a training programme. FINAGRO might create a technical assistance unit specializing in microfinance product methodologies for the rural sector.

Create a database of credit users in order to measure the impact of the component and include this information in the programme's monitoring and evaluation system.

Implement a monitoring system based on indicators of financial, administrative and managerial performance and of impact. Widely available computer systems could be used for that purpose (those of the Consultative Group to Assist the Poor or the World Council of Credit Unions, among others), enabling rapid interpretation and early detection of risks.

Consider whether FINAGRO's access requirements (bank guarantees and insurance policies) might be placing small operators at a disadvantage with respect to larger NGOs.


1/Source: The Economist. Country Briefing (2006).

2/ On the six-point scale, 6 represents the best score. For example, in assessing project relevance, the scale would read as follows: 6 (highly relevant), 5 (relevant), 4 (partly relevant), 3 (partly irrelevant), 2 (irrelevant), 1 (highly irrelevant).

4/ The "Opportunity Bank" document (National Economic and Social Policy 3424) and "Visión Colombia II Centenario: 2019" (the chapter "Aprovechar las oportunidades del campo").

5/ A policy for promoting access to credit and other financial services, seeking to ensure social equity.

6/ The Internet-based network of organizations and projects working with the rural poor in Latin America and the Caribbean (FIDAMERICA), Strengthening the Regional Capacity for Monitoring and Evaluation of Rural Poverty Alleviation Projects in Latin American and the Caribbean (PREVAL) and the Rural Microenterprise Support Programme in Latin America and the Caribbean (PROMER).

7/It is the only operation of the Latin America and the Caribbean Division devoted exclusively to rural microenterprise.

8/ Horizontal refers to partnerships or associations of cooperatives in a region, area or department; vertical relates to integration with second- or third-tier institutions that provide the cooperatives with services and the possibility of managing liquidity.

 

LANGUAGES: English, Spanish

Participatory Irrigation Development Programme (2007)

Tanzania  
September 2007

Completion Evaluation

Introduction

Country background

The United Republic of Tanzania is a country with 38 million inhabitants. In 2005, around 75 per cent of the population lived in rural areas, deriving their livelihoods mainly from agriculture and related activities. About 20 per cent of the rural population live in absolute poverty (i.e. below the food poverty line). In 2005, the United Republic of Tanzania was one of the poorest countries in the world, with a GDP at purchasing power parity of US$580 per capita. The agricultural sector accounts for 45 per cent of the country's GDP, and engages 82 per cent of the labour force. Only 15 per cent of the 40 million hectares of arable land is currently cultivated. The estimated potential for irrigation is 2.1 million hectares. Unreliable rainfall is a major constraint on agricultural development.

A number of policies have been formulated to address the country's development concerns, notably, the Vision 2025, which spells out the long-term economic and social development aspirations of improving the living standards of the people; the poverty reduction strategy paper, which was issued in 2000 and is the focus of most development interventions; the Tanzania Assistance Strategy through which donor assistance is harmonized; the sector-wide approach to budget support which is aimed at reducing duplication, encouraging donor harmonization and rationalizing the flow and allocation of funds; and the Agricultural Sector Development Programme which complements the country's poverty reduction strategy and rural development strategy.

Various donors, inter alia, IFAD, the World Bank and Danish International Development Assistance have been supporting irrigation development in the country. IFAD has developed two country strategic opportunities papers for the United Republic of Tanzania, the first in 1998 and the second in 2003. The latter was informed by a country programme evaluation undertaken by the Office of Evaluation (OE) in 2001-2002. In terms of investments, the Fund has provided US$193 million as loans for 12 projects in the country since 1978; these include operations in the irrigation subsector. Currently, five of these 12 projects and programmes are ongoing.

The programme

The Participatory Irrigation Development Programme is a six-year operation. The programme became effective in February 2000 and its current closing date is end-June 2007. Prior to the programme, from 1990-1997, IFAD financed another project in more or less the same geographic area, the Smallholder Development Project for Marginal Areas, which was largely aimed at promoting small-scale irrigation development.

The initial objectives of the Participatory Irrigation Development Programme were to: (i) increase the availability and reliability of water through improved low-cost systems of water control; (ii) raise agricultural productivity through better extension services; and (iii) build institutional capacity with the long-term vision of realizing the potential of smallholder irrigation development. Notably, these objectives were revised in 2001 to align them with the national Agricultural Sector Development Programme1 and it is against the new objectives and outputs that the Participatory Irrigation Development Programme has been evaluated. The programme has four main components, namely: (i) irrigation development; (ii) support to agricultural development; (iii) strengthening farmers' organizations and local institutions; and (iv) programme coordination.

The programme covers 12 important crop-producing districts2 in the central plateau regions. There is considerable unused arable land in the programme area, largely as a result of the lack of irrigation systems.

In the districts covered by the programme, 21-49 per cent of the population live below the basic needs poverty line, mainly because of poor soils and erratic rainfall. Women have limited access to agricultural resources and woman-headed households represent about 11 per cent of the programme target group. In general, the indicators for social and physical infrastructure are below the national average in the programme area.

The total programme cost was US$25.3 million, towards which IFAD provided a loan for US$17.1 million on highly concessional terms.3 The World Food Programme provided US$3.6 million in cofinancing and Irish Aid, US$0.8 million. The Government provided counterpart funds equivalent to US$3.1 million, and the programme beneficiaries also provided valuable contributions in the form of labour and locally available materials (equivalent to US$0.6 million).

The Ministry of Agriculture, Food and Cooperatives was designated as overall executing agency, while district councils were responsible for actual programme implementation. A programme coordination unit was established in Dodoma, inter alia, to ensure coordination among the line departments involved in programme activities.

Various training centres, universities, NGOs, consulting companies and private contractors were also involved in the programme, providing a range of services required for implementation. The United Nations Office for Project Services (UNOPS) was the cooperating institution. Furthermore, efforts were made to ensure close coordination and synergies with other ongoing IFAD-funded projects in the country such as the Agricultural Marketing Systems Development Programme and Rural Financial Services Programme. These two operations, in particular, were expected to support the programme in the areas of marketing and financial services.

Objectives and methodology of the evaluation

Objectives

The main objectives of the evaluation were to: (i) assess the performance and impact of the programme; and (ii) generate findings and recommendations that would serve IFAD and the Government in designing and implementing similar projects and programmes in the future. The evaluation was also to provide an opportunity for learning and exchanging views with multiple partners on issues related to participatory irrigation development and its contribution to broader rural poverty alleviation efforts in the country (see paragraph 12).

Methodology

This assessment followed OE's guidelines for project evaluations4 and a six-point scale has been used to attribute ratings to each of the evaluation criteria.5 The overall approach to the evaluation included a desk review of the relevant documents, a rapid assessment study,6 the main evaluation mission, which entailed almost four weeks of fieldwork and included interviews with key informants,7 focus group discussions and other participatory approaches involving beneficiaries in seven selected schemes from six districts.8 A final multistakeholder workshop was held in Dar es Salaam in October 2006 to discuss the evaluation's results and lessons learned, as well as to lay the basis for the Agreement at Completion Point (see Part B of this document). The evaluation process involved active interaction and dialogue among members of the core learning partnership (CLP).9

The evaluation mission categorized the 56 programme schemes into three groups, according to two important characteristics: (i) the availability of water; and (ii) the existence of irrigated farming traditions. Group A (covering 9 schemes) had permanent water availability and the community had a tradition of irrigated farming. Group A was therefore most suited to irrigation schemes. Group B (covering 32 schemes) had seasonal/partial availability of water, experience in rice-growing with water-harvesting techniques of bunding, and therefore was suitable for enhanced water-harvesting schemes. Lastly, Group C (covering 15 schemes) had partial availability of water (seasonal rivers), limited or no experience with irrigation or water-harvesting, and therefore qualified for water-harvesting schemes. The evaluation team visited samples of all three groups.

Programme performance

Design features: some general considerations

The programme was designed to include 18 districts identified during programme formulation and appraisal. However – owing to budgetary constraints – the programme steering committee in 2001 approved coverage by the programme of 12 districts only (which later became 13, see footnote 2).

The process of selecting10 districts, schemes/communities and individual beneficiaries was lengthy. It entailed involving potential beneficiaries in a process of consultation and dialogue for around five months before decisions were taken on the selection of schemes. This laborious process was a cause for discontentedness in those communities that were ultimately not included in the programme.

Programme design underestimated the cost of scheme construction, predicting an average of US$170 000 instead of the actual average cost incurred of US$380 000. This caused a shortfall in the programme budget, leading to delays in implementation. Eventually, the financing gap was filled by the Government, which provided an additional allocation of US$2 million as counterpart funding towards the programme.

Implementation and outputs

Targeting

In all, around 25 400 people benefited from the programme. In most cases, the poverty eligibility criteria (see footnote 10) were applied and the unit of selection was the household or the individual, except in schemes where people had plots along an already existing irrigation canal and thereby automatically became beneficiaries. In such situations, compulsory land redistribution was enforced when plots were larger than two hectare per person.11

In some cases, poorer segments within selected communities could not participate in the programme because of their inability to contribute the labour required by the programme.12 Women were particularly disadvantaged in contributing labour, since they were already overburdened with other priority household work. In six out of seven schemes visited by the evaluation team, the 30 per cent target of women beneficiaries was surpassed. However, the 50 per cent target of beneficiaries living below the poverty line was only met in two out of seven schemes visited.

Irrigation development

The planned output was improved water management systems in the programme areas. In general, it is noteworthy that the planned outputs have been achieved and in some cases exceeded. That said, the lowest achievement related to dam construction (50 per cent) and the highest related to the number of beneficiaries reached (187 per cent).

Although the programme made due efforts to undertake the required technical analysis in identifying irrigation schemes for development and/or rehabilitation, the basis for decisions was in some cases weak because of the lack of data (especially on hydrology). This led to the selection of some schemes where the available volume of water was insufficient and could not meet the community needs. As such, some schemes did not yield the results anticipated in terms of production over a number of years.

The evaluation notes that with approximately ten years of IFAD presence in the country's irrigation subsector, more attention should have been paid to the selection of suitable sites. For instance, although droughts and floods cannot be foreseen, their probable occurrence and consequences should have been more thoroughly factored in during the design process.

On another related issue, the tendering process for each scheme was longer than planned, mainly because of the lack of response from contractors. The programme therefore decided to construct some schemes through the use of Government structures. The tendering process led to delays in the construction work and revealed that there were only a few private contractors available to undertake such construction work within the stipulated time frame. In general, the capacity of the contractors was low, inter alia, in terms of technical know-how and resources.

Support to agricultural development

The planned outputs were improved agricultural development services and the rehabilitation or building of market access roads in the programme areas. Under this component, a total of 327 km of market access roads were constructed, which is 31 per cent more than envisaged. Extension officers and participating farmers were trained using the farmer field school13 concept, which led to enhanced knowledge among beneficiaries about improved technologies and varieties. The latter was an important factor in the increases achieved in agricultural production and productivity – for example in paddy – from an average of 0.5 to 2 tonnes per hectare. Increases, however, fell short of expectations at design, partly as a result of the limited availability of irrigation during low rainfall periods in the year.

On a less positive note, some of the demonstrations and trial programmes were seriously affected by drought. For similar reasons, the survival rate of trees planted along the roads constructed is only around 10 per cent, which was the result of both drought and animal damage. Tree-planting along irrigation canals was only 20 per cent successful. Finally, the least successful intervention was the construction of pit latrines in the farm areas (5 per cent). The demand for this item may have been overestimated at the time of programme formulation.

Strengthening farmers' organizations and local institutions

The planned outputs were improved capacity of local institutions among farmers, districts and the private sector; and the fostering of participation, equity and sustainability. In total, 56 water users' associations were formed (representing an achievement of 108 per cent) and 123 256 participant-training-days were provided, with women constituting 36 per cent of the participants. Moreover, a total of 44 savings and credit cooperatives (85 per cent of target) were established, and these served as the main vehicle for promoting savings and credit. The programme also provided support to private-sector development, for example, in terms of training of local artisans and contractors to construct and maintain or repair roads and physical irrigation structures. However, the identification of potential entrepreneurs to be trained was problematic because few people had the required qualifications.

District-level capacity-building focused on training of district staff, and members and leaders of water users' associations in programme implementation and review, labour-saving technologies, scheme operation, water management and other related matters. The evaluation notes that the training imparted was relevant, even though more attention could have been devoted to establishing effective links between district-level and community institutions so as to enable districts to provide technical support to communities.  

Assessment of Programme Performance

Relevance

The Participatory Irrigation Development Programme responded to the needs of the beneficiaries and was in line with Government's overall efforts to combat rural poverty by enhancing rural and agricultural development. The programme was integral to the Government's endeavours to achieve the Millennium Development Goals, and conformed with the country strategic opportunities papers (prepared by IFAD in 1998 and 2003) for the United Republic of Tanzania by aiming to facilitate the rural poor's access to irrigation in order to enhance incomes and livelihoods. The beneficiaries met by the evaluation team felt that the programme was highly relevant as it aimed to address the problems they were facing in relation to inappropriate physical infrastructure, water management, farming technology, food supply, housing, gender equity, linkages with authorities and the ability to organize themselves. All in all, the evaluation concludes that the programme was highly relevant, with a rating of 6.

Effectiveness

The effectiveness of the programme in meeting objectives varied. Overall, water availability and management have improved in the programme area among most but not all of the target group. Extension services for agricultural development saw general betterment, market access roads have been constructed and are serving their purpose, and the capacities of institutions have been strengthened, even though the sustainability of some are a cause for concern. As such, the evaluation considers the programme moderately effective, with a rating of 4.

Efficiency

It is noteworthy that in the earlier stages of implementation, some delays occurred as a result of the lengthy participatory approaches that were being promoted, for example, in the selection of target communities, and the training of water users' associations and savings and credit cooperatives. Also, the tendering procedures for selecting private contractors was time-consuming and the actual construction of the schemes was slower than anticipated largely because of the limited capacity of the contractors selected.

Developing sufficient capacity at the district level to fully meet the implementation needs of the programme required more intensive training than originally thought. In particular, the drawn-out tendering procedures and lack of capacity among contractors led to inefficiencies. Moreover, as stated previously, the overall budget was not sufficient to construct all of the planned schemes, mainly because of the underestimation of construction costs at design and unforeseen increases during implementation. In sum, given that certain activities of the programme proceeded fairly swiftly (for example, the construction of rural roads and the establishment of various grass-roots institutions), whereas others (such as the construction of dams) were more laborious, the evaluation rates the programme as moderately efficient, with a score of 4. 

Evaluation of Programme Partners

IFAD was the programme's initiating agency and thus responsible for programme design. Although there were some design weaknesses – such as the underestimation of construction and rehabilitation costs of irrigation schemes – IFAD carried out planning and design in a largely participatory manner and provided rapid implementation support, whenever needed. Moreover, the programme self-assessment, undertaken as an input for this evaluation, found that overall support from IFAD was timely and satisfactory. In sum, the evaluation considers IFAD's performance as satisfactory, with a rating of 5.

UNOPS – the cooperating institution – provided fully satisfactory supervision through annual missions; undertook loan administration, including disbursements, in an efficient manner; and provided implementation support as required. The performance of UNOPS is rated satisfactory, with a rating of 5.

Through its food-for-work activities, the World Food Programme has supported the construction of market access roads to an extent that allowed the construction of more roads than planned (an extra 77 km or 31 per cent) and has also provided food items worth US$883,366 for the excavation of 300 km of irrigation canals. However, initially during implementation, beneficiaries in some schemes confused "food for work" with food aid. Shortage of food supplies and uncertainties about provision of non-food items were also experienced, but these problems were resolved over time. The performance of the World Food Programme is considered moderately satisfactory, with a rating of 4.

Irish Aid provided substantial financing for the training of water users' associations, savings and credit cooperatives and women's economic groups. The funds were strictly earmarked for training of women, which caused some delays during early implementation until women's groups had been formed under the programme. Broadly speaking, according to the evaluation, the performance of Irish Aid is satisfactory, with a rating of 5.

The Government carried out its role of programme partner to the satisfaction of all parties. For example, it allocated additional counterpart funds than initially envisaged under the programme for irrigation scheme development. Sound programme management has also been ensured through the effective performance of the programme coordination unit, which includes an efficient monitoring and evaluation system. At the district level, performance has been dependent on support from the programme (e.g. in terms of training, additional staff and financing). It appears that performance in relation to the implementation of individual schemes has been less satisfactory since the gradual withdrawal of programme support. That is, district programme coordination units financed under the programme have already been phased out and their responsibilities have been taken over by the line departments at the district level, which have not allocated any specific resources for the operation and maintenance of the programme schemes. This raises concerns because several of these schemes are not yet fully operational, and water users' associations still need support and coaching to ensure proper scheme implementation. Funds from the national Agricultural Support Development Programme are expected to fill the gap; however, this has not yet materialized. Despite this issue, the Government's overall performance is considered satisfactory, with a rating of 5, given its overall positive approach to the programme.

Community organizations. Community-level institutions – especially the water users' associations, and savings and credit cooperatives – have performed their core functions. However, there is some concern about the level of resources invested by the communities in operation and maintenance, which could jeopardize scheme sustainability. The overall performance of the community organizations is satisfactory, with a rating of 5.

Programme impacts

Rural Poverty Reduction

The programme made a moderately successful impact on rural poverty in the country, earning a score of 4. The impact on social capital and empowerment has been positive and, in general, the communities have a greater role in development planning and implementation. In addition, increases in agricultural production and productivity, and in food security were observed in schemes where irrigation water was permanently available. Better extension services also contributed towards achieving food security. Likewise, the construction and rehabilitation of rural roads facilitated market access for produce and improved the timeliness of input delivery. Transportation costs were also lowered. That said, the impact of tree-planting on the environment has been low, and where land was cleared for cultivation, biodiversity has been lost. Increased irrigation had a somewhat negative impact on health, as the frequency of malaria and bilharzia grew when schemes became operational.

Sustainability

According to the evaluation, the programme is potentially sustainable, with a rating of 4. At the national level, there is clear evidence of political sustainability given the thorough attention that the Government has devoted to developing irrigation and water-harvesting systems, which is also reflected in its National Irrigation Policy. At the district level, while policy statements support irrigation development, these have yet to be translated into concrete commitments in budgets to ensure that extension agents can continue to provide advice to the community on a wide range of issues, after the phasing out of the programme. At the grass-roots level, the sense of programme ownership is strong, which is essential for sustainability, even though beneficiaries need to invest greater resources in the operation and maintenance of the schemes and the rural infrastructure developed.

Innovation Promotion

The performance of the programme has been satisfactory in promoting innovations that can be replicated and scaled up, and therefore the programme receives a rating of 5. For example, the promotion of participatory irrigation planning approaches ensured that incremental water was available by diverting water from rivers to irrigation fields in order to supplement traditional modes of water-harvesting, and also allowed members of water users' associations to be involved in the tender process for contractors. This and other aspects of the programme were innovative within the irrigation subsector in the United Republic of Tanzania, and given the broadly positive results, are considered by the Government and key partners as features that should be replicated by future irrigation programmes.

The following table shows the ratings of the programme for performance, impact and overarching factors and performance of partners.

Ratings of the Participatory Irrigation Development Programme

 

Programme evaluation ratings a/

Programme performance

 

Relevance

6

Effectiveness

4

Efficiency

4

Impact (overall)

4

Physical and financial assets

4

Human assets

4

Social capital and empowerment

4

Food security

4

Environment and natural resources

4

Institutions and policies

4

Overarching factors

 

Sustainability

4

Innovation

5

Performance of partners

 

IFAD

5

Government

5

Co-operating Institution: UNOPS

5

Co-financiers:  WFP

4

Irish Aid

5

Communities

5

a/  IFAD uses a scale of 1 to 6, where 1 represents the lower score and 6 the highest. 

Conclusions and recommendations

Conclusions

Through the implementation of 56 schemes, the programme has reached more than 25 000 beneficiaries, which is 59 per cent more than expected at appraisal. This also includes reaching a greater number of women than originally targeted, although the programme was unable to involve the poorest to the extent envisaged. Generally speaking, the performance of the programme is impressive because, for example, most of the outputs have been achieved or even exceeded. However, more reliable provision of water has not been fully achieved within some schemes.

On another issue, although the demand-driven, participatory approach lengthens the time for productive investments to yield results, the programme has demonstrated that this approach is effective in managing water in small-scale irrigation schemes and can enhance ownership and sustainability.

There is evidence that programme interventions have improved the knowledge and skills of beneficiaries, and have had positive impact on physical assets, food security, human assets and empowerment. Some challenges were: the absence of the necessary information to allow for optimum planning and design; lengthy tendering procedures; lack of local contractors; and underestimation of scheme construction cost. Drought has reduced the effectiveness of some of the completed schemes.

The evaluation considers overall programme performance to be moderately successful, assigning it a rating of 4 on the 6-point scale (where 6 is the highest score). Significantly, the programme scores were higher than the average scores in the 2005 ARRI report for relevance and innovation promotion, while they were the same as the ARRI report scores for all impact domains and for sustainability.

The implementation of the Participatory Irrigation Development Programme yielded several lessons. These form the basis of the recommendations below, which are categorized into policy and institutional issues, capacity-building and technological information. First, the Government faces a formidable challenge in striving to reduce poverty in the country, especially in semi-arid and other marginal areas that are characterized by low and unreliable rainfall, seasonal rivers and unpredictable water resources. Although irrigation and/or water-harvesting are among the development options of such regions, it is critical to ensure that the right choices are made, based on an assessment of the technical feasibility of the scheme. Second, owing to the limited availability of water and its various uses in any given area, a holistic approach to developing water resource management is necessary in order to avoid over-optimistic irrigation development plans that may not be sustainable. Further, the demand-driven approach to managing water for irrigation through water users' associations is effective and sustainable. However, these associations need continued technical support from the districts.

One of the challenges of the programme was to reach the target proportion of people living below the poverty line. This was problematic because these people either doubted the programme's benefits or could not afford to participate in the required construction work. As a consequence, in order to enhance participation the programme had to accept some delay in the decision by potential members to join, and to manage a trade off between (i) promoting ownership by requesting participants to contribute money and labour; and (ii) risking the exclusion of some of the poorest among the target group, especially women.

There is also a need to expand irrigated farming substantially by enhancing water-use efficiency (which is often as low as 30 per cent) through better management of irrigation or water-harvesting systems and through increased productivity. This could be achieved by using the same amount of water to irrigate a larger area, by cultivating more than one crop per year and/or, by growing high-value crops with low water requirements.

Participatory approaches, although time-consuming, proved effective in enhancing inclusion of the disadvantaged, but they were not documented. The evaluation also revealed that the participatory approach, the development of water users' associations and the training received in general contributed to building the social capital of the community, which could be useful for other development purposes.

The need for capacity-building at various levels cannot be overemphasized. Since the district authorities are expected to implement the irrigation policy within the new decentralized framework, the staff at this level must acquire the necessary skills in participatory planning and implementation of irrigation schemes, irrigation engineering, scheme construction, institution-building, agronomy and scheme management. In addition, resources need to be mobilized to support the requisite training. More training is needed in the sector of irrigated farming and, similarly, the private sector lacks the technical and financial capacity to cope with the contracted works.

Finally, the programme lacked necessary data, particularly on hydrology, that should form the basis for decision-making.

Recommendations

Policy and institutional issues

The National Irrigation Policy currently being developed should emphasize the need to analyse different irrigation/water-harvesting technologies, and crop/agronomy and livestock options in the context of each potential scheme area. In this way, a "one fits all" solution can be avoided and schemes can function more effectively.

Catchment approach. The Ministry of Agriculture, Food and Cooperatives and other line ministries should continue to develop the policy framework and ensure that the necessary institutions at the national, regional and local levels are in place so that a catchment approach14 may be followed in water resource management.

Supporting water users' associations. Greater institutional support should be provided to these associations to help them to mature in their crucial role in irrigation management and perform this role effectively. A study of how water users' associations function should be carried out and the results translated into operational tools for use by the associations.

Targeting the rural poor. It should be established at the design stage of any irrigation project or scheme that the overall aim is to target the rural poor, while giving due consideration to the economic efficiency of the schemes.

Water use efficiency. The Ministry of Agriculture, Food and Cooperatives should take measures at the policy level to ensure more efficient use of water. This could take the form of guidelines for optimum use of irrigation water.

Participatory approaches involving district councils and communities should be adopted as the standard methodology for planning, designing and implementing all future irrigation and water-harvesting programmes.

Exploiting already improved social capital. In instances where strong social capital – such as women's groups – has been built by the programme but where irrigation systems have been less successful, efforts should be made to use that social capital for other development purposes in the community.

Capacity-building

At the district level. The human and financial capacity at the district level should be enhanced. In particular, more financial resources should be allocated to irrigation development, and training programmes in various fields should be organized for the district staff, for example in following participatory approaches to development; and training water users' associations in good management practices and in the operation and maintenance of irrigation schemes. This increased capacity is essential for the planned decentralized implementation of the National Irrigation Policy and for attaining the targets with regard to the expansion of irrigated agriculture.

In irrigated farming. The availability of training in irrigated farming is a prerequisite for the successful implementation of the National Irrigation Policy. The basic functions of the specialized training institution, the Kilimanjaro Agricultural Training Centre, should be maintained through the provision of the appropriate financial and human resources.

Private-sector contractors. A dialogue should be initiated to identify training programmes for private contractors. Such programmes should also ensure that private contractors are well versed in tendering and procurement rules, regulations and procedures.

Technological information

Data collection programme. A data collection programme should be implemented in the water-harvesting diversion schemes that have already been constructed. This would facilitate the implementation of similar activities in the future. Aspects to be documented should include river flows, flood flows, and rainfall volume and intensity.

Compiling information regarding irrigation and water-harvesting techniques. Information about the range of available irrigation/water-harvesting technologies should be compiled, drawing on the knowledge and experience that have been gained on this subject in the United Republic of Tanzania.

 


1/ The programme revised the original logical framework as follows: Purpose: Crop productivity through expansion and improvement of farmer-initiated and well-managed small-scale irrigation schemes sustainably increased; Outputs: (1) Water management systems in the programme areas improved, (2) Services for agricultural development improved, (3) Market access roads in the programme areas constructed/improved, (4) Capacity of local institutions (farmers, districts, private sector) improved, participation, equity and sustainability fostered, and coordinated programme activities in place.

2/ This later became 13 districts as Shinyanga Rural was split into Shinyanga and Kishapu Districts during the period of implementation.

3/The disbursements from IFAD's loan are currently around 98 per cent.

4/ This included making an assessment of the programme according to internationally recognized evaluation criteria, namely: (i) programme performance, including relevance, effectiveness and efficiency; (ii) impact on rural poverty; and (iii) performance of partners involved in the programme, including IFAD, the cooperating institution, government institutions and others.

5/ As per OE's project evaluation methodology, on the six-point scale, 6 represents the best score. For example, in assessing project relevance, the scale would read as follows: 6 (highly relevant), 5 (relevant), 4 (partly relevant), 3 (partly irrelevant), 2 (irrelevant), 1 (highly irrelevant).

6/ The rapid assessment study collected household data on food security, incomes, and opinions on programme relevance and effectiveness from beneficiary and non-beneficiary communities.

7/ Informants were from the key ministries, donor agencies, local authorities and also included community leaders.

8/ The mission was composed of Mr Ole Olsen, team leader; Mr Moshe Finkel, Irrigation Engineer; Mr Charles Lwanga-Ntale, Social Scientist; and Ms Sylvia Schweitzer, IFAD Evaluation Officer. The lead evaluator was Ms Victoria Matovu, OE.

9/ The CLP comprised representatives from: the ministries responsible for poverty eradication; the Ministry of Agriculture, Food and Cooperatives; regional and district authorities, irrigation and agricultural extension services; the World Food Programme; Irish Aid; and research institutions and universities. The CLP also included representatives of UNOPS, Nairobi; the Eastern and Southern Africa Division, IFAD; in addition to stakeholders in irrigation development such as the Danish International Development Assistance and the Japan International Cooperation Agency.

10/ The criteria for selecting districts included the number of sites and the irrigation potential in the district and the contribution by the district of staff for programme implementation. Scheme selection was also informed by technical, social, economic and agricultural aspects and the situation in terms of management capacity, ease of implementation, land distribution and the environment. At the community level, resource-poor farmers especially women and woman-headed households were the main target groups. Target group composition had to meet the following requirements: (a) beneficiaries should not hold more than 2 hectares of cultivable land; (b) 75 per cent of the beneficiaries should be below the poverty line; and (c) at least 30 per cent of total beneficiaries and 50 per cent of the irrigation management committee members should be women.

11/ It was proposed during programme appraisal that beneficiaries would be only resource-poor farmers owning less than two hectares. As a result, land redistribution was imposed to cater for previous inequalities in access to land among the beneficiaries.

12/ In order to promote ownership, beneficiaries were required by the programme to show their participation by contributing labour, local materials or some funds to the programme.

13/ This is a mode of promoting knowledge among adults, whereby farmers are taught through experimenting with crops and observing crops on the farm or in the field.

14/ This approach aims to ensure that the water resources of a river are managed and used with due consideration for other users in the catchment area.

 

LANGUAGES: English

Special Country Programme – Phase II (2007)

Niger  
September 2007

Completion Evaluation

Introduction

Evaluation objectives and procedure. The completion evaluation of the Niger Special Country Programme – Phase II (PSN-II) was conducted by the IFAD Office of Evaluation two years after the programme's closing. It aimed, in the first place, at ensuring accountability of IFAD and its partners on results obtained by co-financed operations and, in the second place, to contribute to learning by project partners, in order to improve implementation of ongoing projects and programmes and the design of future interventions in the Niger and the region. Three aspects have been evaluated: (i) programme performance (relevance, effectiveness and efficiency); (ii) programme impact on rural poverty in various domains as well as sustainability, innovations and up-scaling; and (iii) the performance of IFAD and its partners. Following a preparatory mission in July 2006 and preliminary work to gather information at the programme sites, the main evaluation mission took place from 19 September to 17 October 2006. Three sources of information were used: interviews with programme participants, partners, former personnel and beneficiaries; a bibliographic study; and observations in the field. At the mission's end, a meeting was held at the Ministry of Agricultural Development and an aide-mémoire was presented to the Ministers of Agricultural Development and Livestock Resources, as well as to the Commissioner for Rural Development of the Ministry of Finance and Economy.

Country context. Niger is one of the world's poorest countries. Although agriculture employs some 87 per cent of the population and contributes 40 per cent of GNP, it is highly vulnerable to natural phenomena. Much of the country is desert. Arable land and pastoral lands cover just 3.5 per cent and 9 per cent of its territory respectively, concentrated in the Sahelian southern strip. Niger is regularly exposed to food insecurity and must resort to imports and international aid, as shown in the most recent crisis in 2005. Approximately two thirds of the population live below the poverty line, and a full one third below the extreme poverty line. This situation is attributable to a number of factors: climate constraints, sandy soil with limited potential, strong demographic pressures leading to imbalanced resource use, limited access to inputs and equipment that would favour intensification, and serious inadequacies in basic services and infrastructure. Women and pastoralists are particularly vulnerable, although the 2005 crisis pointed out high poverty levels in dense agricultural areas as well, among farmers having lost their land.

IFAD strategy in the Niger. The 2006 Country Strategic Opportunities Paper (COSOP) calls for reducing rural poverty in accordance with the Poverty Reduction Strategy Paper (PRSP) under three major thrusts: (1) reduce vulnerability and help rural households achieve food security; (2) improve incomes and market access for target groups; and (3) improve basic social services for vulnerable populations.

The project. PSN-II covers a large part of the Niger's territory, in four major agro-ecological areas: the agricultural area irrigated by the Niger River to the west, the agricultural area irrigated by the Komadougou to the east, the rainfed farming area to the west and centre (Ouallam, Loga, Illéla), and pastoral lands (Tchintabaraden, Abalak and Tchirozérine). The target group includes 900 irrigators (300 of them women), 74 000 farmers in 130 village terroirs1

(260 villages) and 30 000 pastoralists in 40 pastoral terroirs, representing 4 000 families.

Project objectives. The overall objective of PSN-II was to sustainably restore conditions for balanced development based on the optimal use of all natural resources. Specific objectives were: (i) to contribute to achieving food and income security by increasing agricultural production; (ii) to help restore and maintain the productive potential of agricultural and pastoral ecosystems through promotion of soil and water conservation and agroforestry activities; (iii) to help establish conditions for self-managed socio-economic development by promoting farmers' and pastoralists' organizations; (iv) to encourage partnership between community-based organizations and the private sector, and participation by women and youths in the decision-making process of their communities; and (v) to assist in laying the ground for self-sustaining development through the promotion of mutual savings and credit funds, in partnership with the formal banking system.

Programme components. These correspond to the four operational objectives: (a) increasing agricultural production and ensuring sustainable production systems in small-scale irrigation areas (Tillabéri and Diffa); (b) improving food security and helping to increase the incomes of small-scale producers through a terroir management approach (Illéla, Loga, Ouallam and Diffa); (c) improving living conditions for populations in pastoral and agro-pastoral areas (Tchintabaraden, Abalak and Tchirozérine); and (d) enabling women, youths and small-scale producers, once trained, to access working credit to raise their living standards. In addition to these four components, there is an institutional support and coordination component.

Project performance

Design. The component objectives are in line with Niger's policies (terroir management, professionalization of microfinance and irrigation development) and consistent with IFAD's country strategy. However, the programme's national and multisectoral ambitions, little discussed at the design stage, led to two major constraints on PSN-II's effectiveness and efficiency: the project's geographical dispersion as a result of circumstances in the wake of the crisis of the 1990s which favoured a comprehensive programme on a national scale, and dispersion of intervention themes encompassing very different issues and challenges: irrigated farming, rainfed farming and pastoral areas. Given the programme's ambitions and structural weaknesses within Niger's state apparatus, the operating resources provided for in design were insufficient. Also, the lack of a logical framework from the outset was a major handicap to coordination, understanding, steering and monitoring and evaluation.

Partnerships. With faire-faire (outsourcing) as the underlying principle, the project operated mainly through contracting and supervision of recognized service providers. Some activities had to be delegated to various public crop, livestock, rural engineering and agricultural research departments. But the programme design provided for important human resources and management capacity to be maintained within the field units. There was no effective delegation of responsibility to operators, and little visible complementarity between field units and operators in operational terms. Given such multiple participants in diverse contexts with heterogeneous capacity and experience, truly exceptional coordination, training and supervision was needed in the programme management unit – an unrealistic expectation in view of the country context at the time.

Funding. As of end-2004, IFAD loan disbursements stood at SDR 7.95 million (381-NE) and SDR 1.6 million (SRS 46-NE), resulting in disbursement rates of 99.02 per cent and 23.7 per cent. For PSN-II as a whole, funding reached 86.4 per cent of projections at the time of the programme's closing. At the Government's request, Annex 2 to Loan Agreement No. 381-NE was amended twice (November 2001 and August 2003) to reallocate expense categories. As a result of its high geographic dispersion and cumbersome coordination arrangements, the programme far exceeded the expenditures provided for in the loan agreements with respect to project operation, technical assistance and per diem allowances. Spending for rural finance, training, land reclamation and soil conservation works as well as rural infrastructure remained well below projections, although some ground was gained following the mid-term review in connection with physical investments.

Programme coordination and management through the programme management unit and seven field units performed poorly because of external constraints (suspension of the loans, communication problems owing to dispersion, cumbersome arrangements, etc.) as well as internal ones (lack of a logical framework and a functional monitoring and evaluation system, dissension among managers, weak administrative personnel, limited skills development for personnel and contractual operators, centralization of contract management, etc.). Although corrective measures were proposed by successive supervision missions, it was only after the Mid-term Review – which took place late – that the programme was restructured in early 2003, in order to lower operating costs.

Implementation of the small-scale irrigation component. A participatory agricultural appraisal took place first of all to determine constraints and guidelines for training and research by perimeter. Rehabilitation studies were conducted on the 28 sites selected. Rehabilitation work was undertaken on 20 collective irrigated perimeters (CIPs) but greatly delayed by slow administrative procedures and financing delays. In the Komadougou river valley, unresolved land disputes prevented rehabilitation work from commencing. Motorized pumps were repaired for one third of the CIPs. Agricultural extension activities on 15 useful technical topics reached 300 farmers in Diffa and 500 to 600 in the Niger river valley. Promotion activities for producers' organizations reached a larger number of groups than planned. Finally, promising agronomic research took place on a small scale in partnership with the National Institute for Agricultural Research (INRAN), based on tests and evaluations with producers, but little effort was made to disseminate the results.

Implementation of terroir management components in farming and pastoral areas. An investment was made at the outset in designing terroir management methodology. The sociological land-based approach proposed by international technical assistance was appreciated by staff for its modernity but was soon abandoned as overly cumbersome. Much work was done on information and terroir selection, particularly in the pastoral area, prior to undertaking participatory appraisals and local planning. In farming areas, 60 terroir development plans were prepared out of 170 planned, while in the pastoral area only 10 such plans were produced, out of 40 planned. Fewer than half the plans were implemented, for financial and organizational reasons. Among physical accomplishments, soil and water conservation and land reclamation were most significant, while infrastructure remained well below projections. Training reached a total of 21 000 people, or 4 per cent of the target.

Implementation of the rural finance systems component. Three specialized credit facilities were put in place with different objectives, principles and operators for each area. Under facility I, whereby SONIBANK credits were intermediated by a local NGO in the river area and farming terroirs, close to CFAF253 million in credits were granted, of which 85 per cent as short-term credit (4 907 beneficiaries) and the remainder as medium-term credit (355 beneficiaries). The volume of credit granted was lower than expected, mainly because of delays in guarantee fund availability and overly prudent behaviour on the part of SONIBANK. Under Facility II, whereby credit was provided through a local mutual savings and credit fund in the Diffa area, CFAF83 million in credits were granted, of which 92 per cent as short-term credit (711 beneficiaries) and the remainder as medium-term credit (81 beneficiaries). Slow decision-making processes were a constraint. In the pastoral area, where facility III was to promote the creation of local savings and credit funds, small credits (CFAF5 000 to 40 000) on very short terms (one to two months) were made to just four such funds. In the absence of rigorous monitoring, the total volume of credit granted by these local funds is not known, but CFAF3,38 million was available to them, including CFAF0,48 in the form of savings by their members. This very low level is attributable mainly to the fact that guarantee funds were made available late and were insufficient.

Achievement of outcomes. The project's management problems had very serious consequences (an insurmountable delay in investments, a loss of trust by partners and the demobilization of equipment), and a considerable adverse effect on the achievement of objectives. Expectations for adapting the terroir management methodology were met only in part. The approach developed raises many questions: cumbersome processes, weak project feasibility, slow decision-making for implementation, little accountability by local authorities. Despite poor capitalization of research, tests in the irrigated area did yield some results for onion, pepper and tobacco crops. There is no question that PSN-II met expectations best with respect to agricultural extension in the irrigated farming areas, where adoption rates reached 80 per cent for more than half the techniques taught. Nevertheless, farmer support neglected several factors with a considerable impact on the development of land for irrigation. These include household production systems, market access and secure land tenure for farmers. Building collective capacity remained limited in the absence of a longer-term vision and consideration of key issues in sub sector development and legal recognition. Soil and water conservation and land reclamation activities clearly enabled arable and sylvo-pastoral land to be recovered. The fattening of animals for market, considered a profitable activity, was assigned priority for credit under facility I, but repayments were modest since there was little follow-up after the service provider contract ended (92 per cent for short-term credit and 88 per cent for medium-term credit). As to credit under facility II, 83 per cent was provided to purchase inputs, and the mutual fund in the Diffa area successfully recovered the full amount through rigorous and regular follow-up.

Summary: programme relevance, effectiveness and efficiency. PSN-II's major orientations and design principles were relevant, but its proposals often lacked realism and operationality, particularly with respect to context and actual capacity. Serious management and organizational problems, in combination with geographically and thematically disperse arrangements, led inevitably to a low level of effectiveness, with only a small part of objectives actually achieved. Despite the reorientation that took place following the mid-term review at the end of 2002, operating costs at 230 per cent of projections with investments of less than 50 per cent in most cases clearly illustrate PSN-II's limited effectiveness.

Performance by partners

Performance by IFAD. Upon completion of the evaluation, IFAD's performance appears moderately unsatisfactory. Although IFAD made an effort to resolve the critical situations encountered, it was not successful in negotiating a realistic programme in terms of implementation conditions, and its contribution to support, monitoring and supervision was too limited given the particular programme and country challenges. The Mid-term review took place too late and the Project Completion report adds little to the Government's one. IFAD did however draw important strategic lessons for its future programme in the Niger.

Performance by the Government of Niger appears moderately unsatisfactory as well. Even taking into account the very difficult socio-economic and political conditions prevailing during the PSN-II implementation period, the loan repayment defaults and lack of decision on project management arrangements led to many problems. The Government lead supervision missions were rigorous and effective but too infrequent.

Performance by UNOPS is considered moderately satisfactory. Synergies were lacking with the programme management unit and partners, that might have avoided the suspension of land reclamation activities for several six-month periods. Although the missions conformed to the mandate, they lacked adequate resources in proportion to the programme's scale and diversity, and were too infrequent and too short to come up with recommendations beyond simply general management issues. The problems caused by the change in officers responsible for supervision and the mismatch between the goal of building capacity among participants and centralized decision-making procedures were noted as well.

Performance of service providers. Performance of NGOs under contract for implementing the terroir management approach and of service providers for rural credit and in the irrigated areas is considered moderately satisfactory. They were able to mobilize quality personnel and carry out their mission relatively well, despite the breaks in cash flow and delays in the provision of funding. They have shown that they were able to capitalize on the PSN-II experience.

Programme impact

Overall impact. Compared to the programme's aspirations and the substantial human and financial resources used, in particular the US$20 million in loans, PSN-II had a limited impact. Although this was one of the country's largest projects, its impact was clearly diminished by the serious problems encountered: a low level of effectiveness (breaks in financing and operating problems), problems with design quality and implementation in many instances, the lack of a sound strategy to reduce vulnerability among target groups, and an insensitive approach to providing support in the absence of a strategy to empower the organizations concerned.

Targeting of measures to reduce poverty and inequality. The proposed targeting strategy was justified in terms of choice of intervention areas. However, it was not sufficiently well-defined to bring about activities benefiting the most disadvantaged and none of the components proposed a targeting method. In the absence of a sound strategic and methodological approach to reduce vulnerability among the poorest, terroir management activities were too superficial to have an effect on highly disadvantaged groups. The same is true of small-scale irrigation. Nor did the credit facilities meet the needs of the most disadvantaged, including women and young people.

Environment and common resource base. PSN-II has improved access to irrigation water for farmers on perimeters. On plots having benefited from soil and water conservation and land reclamation measures, the impact is apparent in biodiversity, reforestation, erosion control and the expansion of treatment. It is highly regrettable that the impact on communal resource management systems is negligible. The programme's impact is considered moderately unsatisfactory overall in terms of the environment and the management of natural resources.

Agricultural productivity. Here the impact is greater thanks to agricultural extension and soil and water conservation combined with land reclamation measures. Farming on CIPs has increased greatly as a result, although it is gradually being abandoned because of land tenure insecurity and lack of capacity to repair motorized pumps. Introducing erosion control measures has increased crop yields by 30 per cent for farmers with land.

Food security. The programme's impact in this regard is considered moderately unsatisfactory. Irrigated crops are self-consumed in part, but the insecurity of land tenure makes them an unreliable source. In Diffa, CIPs are not deemed to contribute much more to producers with plots outside the collective perimeter. Still, soil and water conservation and land reclamation efforts have had a decisive impact on the agricultural development of land. Cereal availability has increased for those households with land, although the cereal banks are mainly ineffective. Returns on microcredit have generally been reinvested in household nutrition.

Social capital and empowerment. Impact in this area is judged moderately unsatisfactory, despite the concern to involve women and young people. Skills among water user groups remain weak in infrastructure management, maintenance and repair. Access to inputs and marketing was not mastered during the programme. In pastoral areas social networks around the investments were strengthened by consultations and exchanges in connection with PSN-II. A change in behaviour favouring women was noted as a result of the creation of savings and credit funds in these areas.

Financial assets. Generally speaking, insufficient institutional support was provided to build capacity among credit operators that would have enabled them to pursue lending activities with poor and vulnerable populations in the programme area.

Sustainability and ownership. Overall, these are unsatisfactory in the absence of any strategy to ensure the permanence of the organizations and services set up, in irrigation as well as terroir management and micro credit. The project was unable to take advantage of opportunities to transfer their functions and capacity to private or public operators or to involve permanent organizations (communes, land tenure committees, farmers' organizations) to follow up on the organizations created.

Innovation and up-scaling. Contrary to expectation, PSN-II was not very innovative. Promoting and disseminating local innovation was never a concern for project participants. Nor were the few programme innovations capitalized upon or disseminated.

Conclusions and recommendations

Overall assessment. PSN-II was an ambitious programme. All the partners involved had high hopes, and today there is a sense of great disappointment and bitterness among them. IFAD has however drawn important lessons from the experience, as reflected in the new COSOP and the IRDAR project. The overall assessment of PSN-II is moderately unsatisfactory. A comparison of PSN-II's scores with average scores for IFAD-financed projects evaluated in 2005 shows similar trends (relevance being the most satisfactory and sustainability the least), but overall PSN-II is assessed one or two points below the average. Particularly weak compared to the 2005 average are effectiveness, efficiency and innovation. The table hereunder presents ratings given by the evaluation to the project for the different evaluation criteria.

Performance ratings 2 for the different evaluation criteria

Evaluation criteria Rating
Project performance Relevance 4
Effectiveness 2
Efficiency 2
Performance of partners FIDA IFAD 3
Government 3
UNOPS 4
Service providers 4
Impacts Environment & common resource base 3
Agricultural productivity 4
Food security 3
Social capital and empowerment 3
Financial assets 2
Sustainability and ownership 2
Innovation and up-scaling 2
Overall assessment of the project 3

Ratings range from 1 = highly unsatisfactory to 6 = highly satisfactory        
Source: PSN-II Completion Evaluation

Recommendations. Three types of recommendations are proposed, based on lessons learned from the PSN-II experience. Two strategic recommendations relate to targeting and project feasibility, and capitalizing on experiences. Three recommendations relate to operational issues, i.e. essential conditions for effective outsourcing, ways of ensuring true accountability and capacity-building in project management and supervision. The last three recommendations have to do with three thematic issues that remain current in the Niger: the challenges of developing small-scale irrigation, key thrusts in developing pastoral areas, and building capacity among microfinance institutions.


1/ There is no precise translation for the French word terroir, which indicates a socio-culturally, geographically limited land area valorised by a relatively stable community. It is often easier do identify the terroir of a settled community compared to the sometimes very vast terroirs of nomad or semi-nomad groups. Several communities may charge parts of a terroir with each other during certain periods of the year.

2/The rating scale ranges from 1 to 6 as follows: highly successful (6), successful (5), moderately successful (4), moderately unsuccessful (3), unsuccessful (2), highly unsuccessful (1).

 

LANGUAGES: English, French

Cordillera Highland Agricultural Resource Management Project

Philippines  
July 2007

The Cordillera Highland Agricultural Resource Management (CHARM) project design was highly relevant to the needs of the targeted communities. The substantial support for indigenous processes and practices was not only appropriate to the community, but contributed to national policies and practices related to indigenous land and cultural integrity. The combination of sustainable agriculture development and Natural Resource Management reflected the specific conditions and needs amongst poor communities in the Region.

Overall, project performance of CHARM was satisfactory in the achievement of physical targets and attainment of objectives. The rural infrastructure activities resulted in increased yields and reduced input and marketing costs in most instances, but on-going maintenance is an issue. Reforestation activities provided opportunities for short-term local employment.

Agriculture development activities were not sufficiently field-oriented, so the potential in scope and outputs was lower than expected. Notably, rural finance achievements did not attain the expected results. Similarly, Peoples Organizations and Local Government Unit training did not achieve the expected results due to topics not being relevant and multiple training being accessed by community leaders, rather than spread across the community.

LANGUAGES: English