Completion evaluation - Executive summary

Introduction

Evaluation objectives and process. The completion evaluation of the Upper Mandraré Basin Development Project – Phase II (PHBM II) is intended in the first place to fulfil IFAD's obligation to report on the impact and results achieved by operations financed by its members and partners. In the second place, it should help project partners' learning process, with a view to the preparation and implementation of other rural development interventions in Madagascar. The evaluation focuses on five main aspects: (1) the performance of the project (relevance, effectiveness and efficiency); (2) the impact of the project on rural poverty; (3) the sustainability of project results; (4) innovations, their replication and scaling up; and (5) the performance of IFAD and its partners. Following a preparatory mission in May 2008, the main mission took place from 9 June to 4 July 2008. It based its work on three sources of information: a review of available documentation; interviews with project stakeholders and partners; and field observation in 15 villages within 6 of the 11 communes covered by the project. A wrap-up meeting with the various partners was organized at the Ministry of Agriculture, Livestock and Fisheries (MALF) on 4 July 2008, allowing discussion on the mission's aide-memoire.

Country context. Madagascar is an island country situated 400 km off the mainland of East Africa, with an area of 587.000 km² and a population of 18.1 million (2007), 73 per cent living in rural areas and nearly 44 per cent under the age of 15. Population growth is still high (2.8 per cent per year). The per capita gross national product is US$375 (2007) and the average economic growth rate is 5 per cent (2004–2007). According to the Human Development Index of the United Nations Development Programme, Madagascar was ranked 143rd out of 177 countries in 2007. The average life expectancy of Malagasies was 54.7 years in 2005. Agriculture provides employment for 78 per cent of the working population and generates 28 per cent of the gross domestic product (2007). The proportion of the population living below the poverty line was 66.3 per cent in 2007, as against 69.6 per cent in 2001. In 2005, poverty still affected 73.5 per cent of the rural – mostly agricultural – population, as against 77.2 per cent in 2001. The low level of domestic agricultural production has been identified as the main cause of malnutrition and poverty.

IFAD's strategy in Madagascar. Between 1979 and 2000, IFAD's activities focused on increasing food crop production and improving food self-sufficiency, especially in rice and meat. The IFAD Country Strategic Opportunities Paper (COSOP) prepared in 2000 focussed on improving the rural poor's access to technology on the basis of producers' demand, boosting farm capacities - particularly by improving irrigation infrastructures, organizing producers into users' associations, and promoting animal or mechanical traction. It also anticipated support and capacity-building for local institutions and the State's decentralized services. The most recent COSOP (2007–2012), approved in 2006, is based on three strategic thrusts: (1) improving risk management (especially risks linked to farm production and insecurity of land tenure) and minimizing the rural poor's vulnerability by increasing their access to resources and services; (2) improving the income of the rural poor by diversifying agricultural activities and promoting rural entrepreneurship; and (3) increasing the professional capacities of small producers and their organizations.

The Upper Mandraré Basin Development Project was formulated in the wake of the food crisis of 1991 that severely affected the south of the country after several years of drought. The first phase (PHBM I from 1996 to 2000) focused on four communes (3,233 km²) with 39,800 inhabitants in 1996. The amount of the IFAD loan was US$6.9 million. Implementation of the PHBM I helped to remedy the situation in the project area by improving access (102 km of roads and tracks), installing and rehabilitating 2,306 ha of irrigation schemes, improving health infrastructures (four basic health centres built), improving livestock health conditions (23 vaccination runs) and taking the first steps to organize local inhabitants (133 farmers' organizations established). The Interim Evaluation of the PHBM I by the Office of Evaluation in 2000 revealed that the project zone was still marked by severe inaccessibility and periodic localized food insecurity because of major climatic variability, an economic and social environment that was still not conducive to sustainable development, endemic rural insecurity connected with cattle thefts, and very poorly developed services and socio-economic infrastructures. The PHBM II (2001–2008) was formulated and appraised in the course of 2000, with an estimated cost of US$23 million. It covered nine rural communes (8 600 km²), increased to eleven in 2003 after two communes were split in two, and included the four communes already covered by the first phase. It targeted the whole population of the intervention zone, with a total of 96,000 people (17,400 households), especially women, young people and landless farmers.

Objectives and components. The general objective of the project was to contribute to a reduction in rural poverty, especially through: (1) a diversification and increase in the rural population's income; (2) an improvement in the food security of rural households; and (3) a contribution to the rehabilitation and sustainable management of natural resources. The PHBM II has five components: (a) Local capacity-building (7.7 per cent of the total cost); (b) Support for local initiatives (46.6 per cent); (c) Support to financial services (4.6 per cent); (d) Opening up of the project area (28.2 per cent); and (e) Coordination, management and monitoring-and-evaluation (12.9 per cent). The project's line ministry was MALF and the United Nations Office for Project Services (UNOPS) was IFAD's Cooperating Institution in charge of loan administration and project supervision for most of the project implementation period.

Project performance

With regard to the boosting of local capacities, the results obtained basically concern: (i) implementation of commune-level local participatory planning, which led to the formulation of 135 village community development plans, consolidated into 11 communal development plans; (ii) the training and installation of 11 communal development facilitators and 132 village-level facilitators; (iii) the recruitment and training of more than 1,300 local experts to provide social and economic services for communities in various sectors of activity; (iv) the establishment of a communal development committee (CDC) in each commune and the boosting of its capacities, in order to provide a space for discussion and a representative consultative body that can act as an interface between the commune and local inhabitants; (v) the promotion of nearly 1,500 rural organizations with a wide range of economic and social mandates; (vi) literacy training of 6,600 adults (49 per cent women); and (vii) the promotion of rural communications (radio station), nutrition and community health, as development tools supplementing other capacity-building activities.

The support for local initiatives expressed in the village community development plans and communal development plans has been consolidated, notably through the financing of nearly 1,130 productive sub-projects concerning agriculture, livestock and the environment; the boosting of agricultural and livestock infrastructures (2,250 ha of irrigation schemes installed or rehabilitated, more than 6,000 agricultural tools subsidized, 46 vaccination runs installed etc.) and the development of economic and social infrastructures (15 km of community tracks, 21 capital works and 36 water points installed, 9 schools and 5 health centres built). In addition, these local initiatives have been backed up with cross-cutting support activities (introduction and extension of technical packages on nearly 20 varieties of cassava, rice, maize and sorghum; support for marketing, supplies and agricultural services; and the promotion of environmental activities).

With regard to opening-up the project area the results have basically concerned the rehabilitation of 140 km of provincial roads (79 km of these in partnership with the Sectoral Transport Programme) and the rehabilitation of 64 km of roads between communes. Regarding support to financial services, the project established a savings and credit union and nine branches in the chief towns of the various communes.

Relevance. The project design had a number of strengths that contributed greatly to the achievement of the results described above. First, the project adopted a participatory approach to the planning and management of local development, which was harmonized and well coordinated between the communal and village levels and boosted by complementary capacity-building activities (literacy training; information, education and communication [IEC] concerning health, nutrition and environmental education; the organization and training of producers; and rural communications). Then, the project mobilized sufficient resources (human, financial and technical) to provide real support for rural inhabitants' local initiatives as expressed in the village community development plans and the communal development plans. Finally, a good overall consistency between the project's objectives and the Government and IFAD strategies was maintained throughout its execution, thanks to the various adjustments and refocusing of activities supported by the supervision and mid-term review missions.

The design of the PHBM II also contained weaknesses that would limit the scale of impact in certain domains and the sustainability of the project itself. Firstly, the social infrastructure component (health, education and drinking water) was too small, despite the fact that its vital importance was stressed by the Interim Evaluation in 2000. There was a lack of reflection and real effort during project preparation to establish a solid partnership that could have made up for this weakness. Secondly, project design failed to take the decentralized technical services into account as particularly valid strategic institutional partners in project implementation and as priority targets for capacity-building. Thirdly, it neglected the communal level as a valid centre point for the planning and management of local development, instead of focusing the participatory approach solely on the village level. With these various considerations, project relevance is thus deemed moderately satisfactory.

Effectiveness. In terms of results, the project has generally performed well inasmuch as it has already reached, one month after completion, an overall financial execution rate of 96.4 per cent (in local currency) and an overall physical execution rate of 122 per cent. The IFAD loan was, at that time, disbursed for 97.07 per cent.  In terms of the achievement of specific objectives, the project has shown varying degrees of effectiveness, depending on the objective in question. Thus:

• It has been very effective with regard to the objective of boosting local capacities for the planning and management of economic and social development, as is seen especially in: the good level of ownership assumed by the CDCs with regard to the participatory planning process, consultation, the search for partnerships and the management of local development; the firm rooting of CDCs within communal structures; the boosting of the collective and individual capacities of local communities through the establishment and training of farmers' organizations and IEC concerning health, nutrition, literacy training for adults; and the real involvement of local experts trained in the supply of local services for communities. However, the evaluation highlights, on the down side, the sluggishness of the village development committees and also of the associations and committees responsible for the use and maintenance of certain socio-economic infrastructures (health centres, watering points, tracks).
• It has also been effective with regard to the objective of support for local initiatives, as is seen, on the one hand, in the increase in irrigated areas and the greater intensification on irrigated land thanks to a more secure water supply, and, on the other, in the improvement in farmers' agricultural equipment and the extension of new agricultural and livestock techniques through the promotion of nearly 1,130 sub-projects. The main weakness lies in the low level of support with regard to social infrastructures and access roads because project design underestimated the importance of these components. Moreover, most of the infrastructures established are not used and maintained as well as they might be because of the inappropriate approaches and procedures adopted for their installation, the lack of specific training for the committees charged with their management and the continuing inaccessibility of many villages, hampering full use of the infrastructures.
• The project had a fairly limited effectiveness with regard to the objective of developing sustainable agricultural systems. Some strong points were observed especially concerning the development of irrigated rice cultivation and market gardening (horticulture) on the banks of water courses, with considerable use of manure, the running of an integrated agro-forestry trial on five sites and attempts – with varying degrees of success – to diversify and boost rainfed agriculture and livestock rearing (cassava, goats, cattle health). However, many weak points were observed as well. First, the development of irrigated rice cultivation is precarious due to the only partial water control at the irrigation scheme level¹ and the individual plot level, making the irrigated areas vulnerable during the off-season and in years with poor rainfall. The development of gardening is also limited due to the location of gardening plots, most often along the banks of small streams and rivers with intermittent flow and reduced potential in terms of irrigable land, and the predominance of manual irrigation with watering cans. Then, diversification of rainfed crops was limited (with a predominance of cassava), and extension and integration of soil protection and rehabilitation measures into the rainfed cropping systems were absent, although such techniques could minimize the risks associated with fluctuations in rainfall. Furthermore, the livestock rearing systems promoted by the project remain vulnerable and are poorly integrated with cropping systems. Finally, natural resource management activities were of an ad hoc and demonstrative nature in the absence of an overall vision regarding the development and sustainable management of natural resources.
• Satisfactory effectiveness was noted with regard to the objective of optimizing production by opening up the project area, improving access to markets and reducing post-harvest losses. Although road and access track rehabilitation works did not succeeded in fully opening up the area, they have clearly led to an improvement in the circulation of goods and people in the project area. They have thus contributed to: better access to communal markets, certain production zones and major consumer centres outside the project area; a marked reduction in transport times; an appreciable increase in the number of agricultural produce collectors and transporters; an increase in the amount of produce on local markets; and revitalization of these markets. Moreover, the remarkable proliferation of paddy hullers (more than 30) has led to a considerable reduction in post-harvest losses. These various aspects have led to a major reduction in transaction costs and thus to greater value of agricultural produce. However, there are still some weaknesses, particularly with regard to the incomplete and still precarious opening up of the project area² and continuing difficulties in the supply of inputs and the marketing of garden produce.
• With regard to the objective of improving access to local financial services, a moderately satisfactory effectiveness was observed. The establishment of the FIVOY savings and credit union and its nine branches was late (2004), but since then it has made access to financial services much easier for local inhabitants, as the results of the network show: a penetration rate of 15 per cent (compared with an anticipated rate of 5 to 10 per cent); 5,213 beneficiaries reached (compared with the anticipated 4,500) of which 43 per cent were women; 3,455 union members of which 26 per cent women etc. Moreover, the most vulnerable groups (women and farmers without land or cattle) have been effectively reached by the network, thanks to a varied range of products, some of which require only the moral guarantee of a support group such as Credit with Education, specifically adapted to the development of women's income-generating activities, and the Associative Working Capital Credit intended to finance the activities of small groups of producers. However, certain weaknesses can be seen, especially with regard to: a poor capacity to mobilize savings (less than 30 per cent of total deposits allocated); the very slow progress toward financial autonomy (now expected for 2011 instead of 2008); and the very poor coverage of villages outside the main towns of the various communes.

In conclusion, given the positive elements mentioned above and considering the initial situation in the project zone, the overall effectiveness of the project is deemed satisfactory.

Efficiency. The economic internal rate of return of the project's productive investments calculated by the evaluation was at least 27 per cent and the timeframe for capital recovery was estimated at less than seven years, so that the investments made over the period 2001–2007 had already been recovered by 2008. This exceptional economic performance is essentially a result of the low unit costs of works on irrigation schemes. Moreover, the attainment of a 122 per cent rate of physical execution when only 96 per cent of the budget was used is a further indication of the project's efficiency. However, the efficiency of the rural financial services promoted by the project was modest, due to unfavourable conditions in the project zone and the staff's poor technical and management skills. To sum up, the overall efficiency of the PHBM II is judged satisfactory.

Rural poverty impact

The PHBM II has had a generally satisfactory impact on rural poverty. However, the scale of the impact varies across the different impact domains.

Household income and assets. The impact noted at the household level is very satisfactory. Monitoring by the Rural Observatories Network in the area showed that between 2002 and 2005 the average per capita cash income increased by 74 per cent. This increase affected all social categories, but particularly households without land or cattle (with a 178 per cent increase) and small farmers with less than 0.30 ha of irrigated land (with an increase of 160 per cent), who formed the project's special target group. This increase in income has had a major effect on the state of housing and led to an improvement in household and agricultural equipment.

Food security and agricultural productivity. The impact in this domain has also been very satisfactory, thanks particularly to the rise in rice and cassava production, with increases of 92 and 118 per cent respectively between 2001 and 2007, and also to the promotion of onion and garlic production (620 and 160 tonnes respectively) intended mainly for sale. This was possible thanks to the application of a value-chain approach focusing on three commodities (irrigated rice, garlic and onion) benefiting from a surge in demand from the rapidly developing city of Fort Dauphin and the global price increase of cereals. However, livestock yields saw very little improvement, and diversification and improvement in rainfed crops have been limited by periodic rainfall deficits. Household food security has improved significantly, seen in the very marked increase in the proportion of self-sufficient households, which rose from 36 per cent in 2001 to 78 per cent in 2008.

Human and social capital and empowerment. The project has also had a very satisfactory impact on the boosting of rural inhabitants' individual and collective capacities through literacy training, improved communications, the organization of grass-roots groups and associations, and training. The participatory approach has also allowed a new culture of consultation and self-confidence to be established at all levels, together with greater social and economic integration of traditionally marginalized categories (especially poor farmers and women). Furthermore, women's social and economic status has seen a marked improvement. Women are now much more involved in the new dynamics of social and economic development, forming more than 40 per cent of the members of grass-roots organizations, 67 per cent of sub-project recipients and 50 per cent of credit service beneficiaries. However, while social development is clearly visible in some spheres, it is still hampered particularly by the extent of the needs that have not yet been met with regard to drinking water, health and education facilities and services. Moreover, the situation of young people has not seen any marked improvement, for lack of specific targeting mechanisms.

Natural resources and the environment. The project has started to have some noticeable effects in this domain through the growth of awareness and a certain environmental discipline, both individually and collectively, thanks to the awareness-raising and educational activities undertaken and the demonstrations carried out. However, the ad hoc and demonstrative nature of these activities, combined with the fact that not enough time has yet passed, means that so far no widespread impact can be observed. Moreover, some negative effects and impact could be caused in the medium and long term especially because of the failure to incorporate soil protection and rehabilitation practices into the rainfed cropping systems practised on hillsides, and the lack of a vision and good practices for the rational, sustainable management of soil and water resources on irrigated areas.

Institutions and services. The project has enabled a radical transformation of the local institutional environment to take place by boosting the capacities of communal structures (CDCs, communal councils), revitalizing these structures and setting up an agricultural service centre, a local microfinance network and an inter-communal land tenure office. On the other hand, its contribution has been very limited in the sphere of consolidating and revitalizing decentralized technical services.

Sustainability and innovations

Sustainability. Insufficient time has yet passed to make a proper assessment of this aspect. However, some favourable indications exist, concerning especially the ready assumption of ownership of local development planning and management on the part of the CDCs and communal councils, the enthusiasm of the majority of the farmers' organizations established, the development of a new awareness that is more favourable to the protection and rational management of natural resources, and the emergence of new local services. On the other hand, some negative aspects constitute real threats to the sustainability of project gains. These consist particularly in: the incomplete and still precarious opening up of the zone; the lagging behind of social development due to the insufficiency of educational, health and drinking water infrastructures and services; the precariousness of the improvement in agricultural production due to climatic fluctuations and the absence of rational development and management practices for soil and water resources; the poor efficiency of microfinance services; the low capacity of decentralized technical services to take over the project because of the poor opportunities for capacity-building offered to them by the project; and the fragility of the newly created service institutions (agricultural service centre and inter-communal land tenure office) because of their novelty.

Innovations. The project has enabled the introduction of a number of innovations, particularly concerning the large-scale implementation of a participatory approach to planning and management of local development; the introduction of the value chain approach and its integration into the participatory approach; the incorporation, starting at the design stage, of an exit strategy, and its implementation throughout the project through a programme for supporting the transfer of assets and knowledge; the on-farm trial of a new concept of integrated agro-forestry; the introduction of new plant and animal production techniques and crop diversification. However, the project missed a number of promising opportunities for innovation, especially the introduction of strategies and practices that are better suited to the specific climatic conditions of the project area in order to manage and develop soil and water resources, and the promotion of local sub-project management by   communes or farmer associations.

Performance of partners

The Government and its various agencies have performed satisfactorily overall. The Ministry of Agriculture, Livestock and Fisheries has assumed effective ownership of the project and shown a good capacity for piloting and supervision. It has provided substantial support to the project through its central and regional services, and given it sufficient autonomy. The Ministry of Finance and the Budget has provided counterpart funds as foreseen. Other ministerial departments have also provided proper support through their sectoral services and programmes (infrastructures, health, water supply, the environment), despite the prior absence of any solid partnerships. Regional authorities and communes have been actively involved in project implementation, placing it at the centre of their concerns. However, decentralized technical services (at regional and district levels) have not been sufficiently involved by the project. The project management unit has performed particularly well, mobilizing a large number of partners not initially anticipated, taking many valid initiatives that advanced the relevance and effectiveness of the project, and ensuring good planning and coordination of activities. Although the project's monitoring and evaluation system came into operation very late, it has been exemplary. However, the project office has shown little responsiveness to the challenges of sustainable natural resource management (water and soil fertility in particular) and the adaptation of rainfed agriculture to climatic fluctuations. It has also experienced some difficulties in financial management of the project. Service providers have given a moderately satisfactory performance on the whole, with variations depending on the provider. Thus, the specialized technical operators and the generalist operator have generally performed well, supplying qualified staff and working in harmony with the project team. However, the performance of consulting firms and businesses is judged moderately unsatisfactory because of a number of technical errors and the lack of innovation in the case of the former, and insufficient equipment, training and managerial capacities in that of the latter. IFAD was given an overall moderately satisfactory performance. The initial project design had several shortcomings, with regard particularly to the insufficient consideration of a number of important recommendations made by the Interim Evaluation, the failure to establish a solid partnership and the exclusive focusing of the participatory approach on the village level. However, IFAD showed sufficient flexibility and responsiveness to be able to rectify some of these shortcomings at the time of the first mid-term review, even introducing some fresh adjustments to take account of changes in the strategies of the Government and IFAD itself. UNOPS performed satisfactorily by regularly carrying out supervision missions with the same teams, resulting in the growing effectiveness of its services. Moreover, its recommendations often went beyond simple verification of the status of technical and financial execution of the project, to touch on deeper technical, methodological and economic aspects, thus helping to improve the project's effectiveness.

Conclusions and recommendations

Overall assessment. The assessment of the PHBM II is on the whole satisfactory. The strong points noted are: a good efficiency and effectiveness in achieving the objectives of boosting local capacities, supporting local initiatives and optimizing agricultural production; a major impact on household income, agricultural production and food security; and a good performance of Government agencies, especially the project management unit. The weak points lie basically in: the lesser effectiveness in promoting sustainable production systems; the lower impact on natural resources and the environment; and the uncertain sustainability of certain achievements.
Ratings Attributed to the PHBM II and its Partners by the completion evaluation

 

 Source: Completion evaluation of the PHBM II.

Recommendations. The completion evaluation of the PHBM II makes four recommendations. The first three concern the important strategic lessons that can be drawn from the experience of the project and should be taken into account in future rural development projects in Madagascar or elsewhere, while the fourth is intended to capitalize on the gains of the PHBM II.

Recommendation 1. To promote the balanced, sustainable development of Madagascar's rural zones by harmonizing and further integrating the interventions of integrated participatory local development projects with those of national sectoral programmes and regional development programmes on the basis of clear, firm partnership commitments.

Recommendation 2. To promote the diversification, integration and competitiveness of agricultural activities by promoting a diversified and integrated value chain approach to agricultural development, while placing women and young people at the centre of such development.

Recommendation 3. To promote the flexibility of project design and implementation so that implementation modalities can be modified and fine-tuned according to the changing context, thus allowing a good level of performance (relevance, effectiveness and efficiency) to be achieved and maintained, and a greater assumption of ownership of projects by those actually carrying them out.

Recommendation 4. To rapidly consolidate the dynamics of social and economic development already set in motion by the PHBM (I and II) in its intervention area in order to make the project gains permanent and capitalize on them.

---

1/ Fed by off-take diversion dams, offering no possibilities for water storage and control, and lacking adequate water control works or water distribution arrangements that could ensure a fair distribution of water among irrigators.

2/ The existence of a problematic un-rehabilitated 20-km section on provincial road 107, the destruction of an important ford at PK8 on provincial road 107 at the southern border of the zone, the existence of a 24-km un-rehabilitated section at the end of provincial road 117, which could have opened up the west coast zone, and the lack of upkeep of the roads and tracks already rehabilitated or constructed.

3/ Rating is carried out on a scale of 1 to 6 (6 = highly satisfactory; 5 = satisfactory; 4 = moderately satisfactory; 3 = moderately unsatisfactory; 2 = unsatisfactory; 1 = highly unsatisfactory)