Ex-post evaluation

Introduction

IFAD's Monitoring and Evaluation Division conducted an ex-post evaluation of the Oaxaca Integrated Rural Development Project in October, 1991. The evaluation's purpose was to assess the project's impact on target beneficiaries, as well as to draw lessons from this experience on how to improve rural development assistance efforts applicable both to future project design and implementation in Mexico, and to a wider context.

Background and socio-economic context

The project area is one of the poorest in the State of Oaxaca, which in turn is the poorest state of Mexico. Oaxaca has largest concentration of indigenous populations. With over 12 languages, a poorly developed infrastructure, and only 10% of the arid land cultivable, Oaxaca represents a particularly difficult context for project implementation.

Although the project was designed by IFAD at the end of the 1970's during Mexico's economic boom fuelled by the increase in petroleum prices, the project implementation period (1981-1988) coincided with one of the worst and most prolonged economic crisis suffered by Mexico in the last five decades. From a period of growth with relative price stability, the Mexican economy entered a period of unprecedented hyperinflation, which tripled from 30% in 1981 to 518 in 1982, with a 40% devaluation over the same period. At the same time Mexico's terms of trade deteriorated from a level of 185.4 in 1982 to 87.4 in 1988 (1987=100), as world interest rates rose steeply.

The effects of hyperinflation led to a considerable increase in local currency, while the Government policy of public expenditure austerity reduced the availability of counterpart funding. Project disbursements were negatively affected by the contraction of public spending, both in terms of decreased amounts and increased delays in releasing funds. Funds were not released at the federal level for periods of up to seven months, halting all project activity underway.

Project description

The Oaxaca Rural Development Project was initiated by IFAD in May, 1980 as a five year project (subsequently extended by three years) with IFAD providing USD 22 million of the total estimated project costs of USD 57.2 million. IFAD's loan terms were for a 15 year loan, at an annual interest rate of 8% and a three year grace period. Remaining funds in the amount of USD 35.2 million were to be provided by the Government of Mexico. The World Bank was IFAD's cooperating institution.

The project represents IFAD's first initiative in Mexico, and was intended as a follow-on to the earlier World Bank financed PIDER project. Although PIDER was a nation-wide effort, the Oaxaca project was limited initially to the rural poor of the Chatina region (in the districts of Juquila and Sola de Vega), and in Miahuatlan and Pochutla. As of 1984, the project extended into the districts of Putla, Yautepec and Jamiltepec.

Project objectives and components. The Oaxaca project aimed to reach the smaller and less privileged farmers in remote areas who were not adequately reached by PIDER. The principal objective was to increase the standards of living of the poor population of the inland and coastal zones through economic and social improvements. The project design assumptions were that the indigenous population was strongly motivated to remain in the area, and the under-exploited resource base would be able to support the population on a sustainable basis. The project intended to:

i) Substantially increase food crop and livestock production;

ii) generate employment and income opportunities through agri-cultural diversification;

iii) increase fish production and consumption, and increase poor fishermen's income;

iv) improve living and working conditions of the target population through improvements in health, infrastructure, service provision;

v) protect the natural resource base through soil conservation.

Project objectives would be met through the following components:

i) agricultural production services (increased and improved extension services, land titling, credit provisions, institutional strengthening of INI (National Institute for Indigenous People);

ii) agricultural development (pasture improvement and livestock development, fruit and coffee production, small-scale irrigation programs, soil conservation;

iii) forestry development (aerial photography, nursery establishment, reforestation);

iv) fisheries development (training and infrastructure development, fish marketing assistance);

v) community development, farmers' participation and project management (promotion of community organizations, beneficiary involvement and participation in planning and execution, project management, monitoring, evaluation, coordination, etc.);

vi) improvement of infrastructure (rural road rehabilitation and construction, basic community health services through water supply provisions).

Target Beneficiaries. The project was expected to benefit 15 000 small farmer families, or 84 000 of the poorest people, mostly indigenous, with an average per capita income at the time of appraisal of USD 60 for farmers, and USD 100 for fishermen.

Project Implementation. The project was implemented by a consortium of 15 federal and state agencies, under the coordination of the Ministry of Programming and Budgeting (SPP). As an attempt at decentralization, major responsibility for planning, execution and supervision was placed at the State level, with disbursements made at the federal level. A Coordinating Unit was established within each of the two project regions to coordinate between the SPP delegation in Oaxaca and project beneficiaries.

Project implementation

The project's implementation strategy characteristics were as follows:

A concentration of highly diversified activities in a few limited areas, and fragmented activities dispersed over a large number of communities;

Inopportune and fully subsidized provision of inputs, with insufficient and inadequate complementary technical assistance;

A combination of federal and state institutions supporting various and at times the same activities, with progressive responsibility given to the state agencies;

Provision of services considered priorities by the target communities, even if outside the defined realm of project activities, as a means to penetrate the project area. An example is the construction of basketball courts, or municipal buildings.

Project Implementation Mechanisms

Resource Approval and Allocation Process. The delay in funds approval and disbursement was one of, if not the key, factor negatively affecting the project's implementation. Funds were often disbursed at inopportune moments which did not correspond to the agricultural cycle. Inputs were not delivered in the critical periods in the agricultural cycle. In addition, the executing agencies lacked competent personnel with the ability to develop the work plans with adequate leeway to go through the procedures and channels for the timely release of funds.

The project had a late start, with only 7% of funds disbursed in the first two years. Activity increased in 1983. By 1985, the initial close of the project, only two thirds of total IFAD funds had been disbursed.

Beneficiary Participation in Project Implementation. Although on a limited scale, the project introduced a series of participative mechanisms, such as ‘participative planning forums' in 1985; collaboration with the Regional Council of Marginalized Peoples which represented 16 communities in the project area; planning of project activities in consultation with elected representatives of the target population; such as the municipal authorities, and with elected representatives of indigenous groups.

Supervision. The multiple strata of supervision complicated project implementation, without resolving problems identified, given the clear lack of accountability for performance to institutions. The key problems repeatedly identified were the delays in fund disbursement, and the lack of coordination among the executing agencies. The various layers of supervision were provided by the project's Coordination Unit; 15 state and federal level executing institutions; and the World Bank. In some instances, such as in the case of road construction, there was direct beneficiary supervision.

Despite the high degree of supervision, recommendations made by the 14 World Bank and IFAD supervision missions over the project life often were not relayed to the executing agencies or to the Coordination Unit to be acted upon.

Monitoring and evaluation. Although the project appeared to have a large M&E apparatus, with two M&E units; one each for the Chatina and Miahuatlan/Pochutla regions, this component was not adequately implemented and utilized for project management purposes. Despite the large amount of resources apparently dedicated to M&E and the large structure for this function (nine professionals, administrative staff and two vehicles) in reality these resources were thinly spread over many other activities as well. No computers were available for data synthesis.

The M&E units processed information received from the executing agencies and the supervision memorandums. Monitoring by executing agency, program component, and community was presented in semestral reports with data on physical advances and, in some cases, financial expenditures. The Coordination Unit proposed recommendations on the basis of monitoring reports to the various executing agencies which were at times considered and acted upon.

Key indicators established at the time of appraisal were not utilized to measure project performance. The M&E unit did not collect the type of data necessary for this team to have measured the project's impact on beneficiaries. While ‘output' data on project activities was maintained, such as ‘kilometers of roads constructed,' indicators were measuring the impact of these activities on project beneficiaries were neither established nor measured. External factors affecting project performance such as product prices, input prices, climate effects, etc. were not tracked or analyzed through the M&E system.

Project Coordination. The complex project implementation structure and the lack of coordination of project components repeatedly cited by the supervision missions as a critical problem affecting the project. The project design established a Coordination Unit linked to the State Government as a mechanism to support the Government's decentralization efforts. Tensions between the federal and state governments exacerbated by the funds disbursement problem negatively affected the project's implementation.

The Coordination Unit was able to better coordinate activities in the State of Oaxaca with the 1984 creation of the regional subcommittees of the State Planning Committee for the Development of Oaxaca. The Project Coordinator was nominated as Technical Secretary in the subcommittees for Sierra Sur and Costa. This permitted integrated planning of the diverse state coordination meetings. The new coordinator since 1987 was also designed the State government delegate in Miahuatlan, which included areas not in the project zone.

The complexity of the project design called for intense coordination, without adequately considering the existing administrative and institutional constraints limiting the coordination Units' maneuverability.

Project results

Economic Analysis. The project's internal rate of return (IRR), calculated using 1979 deflated prices as the base year to facilitate comparison with the appraisal IRR conducted in 1979 was 25%, as compared to the appraisal's projected IRR of 18%. This can be attributed in large part to higher prices for project produced goods, and lower labour and input costs than estimated at the appraisal stage.

Nutritional Impact. The Duval method to assess nutritional impact determined that in all areas of the project, ranging from low to high intervention, the population's nutritional status improved over the implementation period. Greater impact was achieved in high intervention over low intervention areas. Positive impact in this area was also achieved through DIF's interventions in nutritional and hygiene education.

Productive Diversification. One of the project's explicit objectives was to diversify the employment and income-generating activities through fruit, coffee, honey, and other types of cultivation. While fruit cultivation had a limited impact, given the untimely supply of trees, inadequate technical supervision, and poor coordination among executing agencies, the limited success has had a demonstration effect. The honey production activities were not successful, again, in large part, due to the lack of technical assistance accompanying the provision of inputs. The coffee component had very positive results, leading to expanded surfaces cultivated, production, municipalities and producers involved and yields. This was in large part due to the efforts of IMMECAFE, the National Coffee Corporation, in developing an appropriate technical package.

The project led to agricultural diversification, since the increase in the area dedicated to coffee production did not lead to a decrease in food crop (primarily maize) production.

Increased Access and Reduced Costs. The construction of 639 kilometers of rural roads has reduced transportation costs, and the price of goods, while increasing access to markets and urban centres.

Slowdown of Migration. The project appears to have slowed down the patterns of out-migration, through having created economic opportunities in the project area. Population in the four districts where the project had a major presence; Juquila, Sola de Vega, Miahuatlan, and Pochutla increased in the 1980s in a significantly higher proportion than the State of Oaxaca as a whole. The increase in Oaxaca between 1980 and 1990 was 28%, while in the four districts mentioned it was 35, 38, 34 and 53 percent respectively over the same period.

Small Enterprise Development. Although not initially planned, the project assisted in the development of various sustainable microentrepreneurial activities, in diverse sectors, such as wood, textiles, and agro-processing. The majority of these activities employed primarily women.

Sustainable Irrigation Development. Two of the five irrigation works have created irrigator's committees, which collect user fees to cover operating, maintenance and water costs.

Institutional Development. The project constructed two rural development centres (CADERs) in Sola de Vega, and Santa Reyes Nopala, the poorest and most indigenous areas respectively in Oaxaca. Both centres were operating at the time of this evaluation as offices for the various executing agencies, serving as a focal point for the integration and coordination of State development efforts.

Demonstration Effects. The project has had a positive demonstration effect in several areas. Fruit cultivation although not realized during the project, is now being realized. The project has led to a sustained effort by government institutions in the area where they were not previously active, and forced attention to meeting the needs of indigenous peoples.

Limited Impact Due to Scale of Activitites. The fragemented delivery of project interventions limited the magnitude of the potential impact. Although the project appears to have bettered conditions, the dispersion and minuteness of the interventions did not lead to overall noticeable changes, with the exception of the coffee producers.

Lack of Deliverables Reaching the Target Groups: In many cases, the project's deliverables never reached the intended beneficiaries. The fisheries component provides such an example. The project intended to give some 70 boats and 100 outboard engines on credit to individual and cooperative fishermen. Of the reportedly 22 engines given, not one was functioning by early 1987. No credit recovery was enforced by the Dirección de Desarrollo Pesquero del Estado. The two of four planned refrigeration units were underutilized, one for lack of catch and one for being out of order since 1985, according to the February 1987 mission supervision report. A visit to coastal fishery sites by this mission found that reported deliveries had never been made.

Conclusions and lessons from the experience

Conclusions

The project was designed at the end of the 1970s when Mexico was enjoying an economic boom fuelled by increased petroleum prices, but was implemented from 1981 to 1988, a period of one of the worst economic crisis faced by Mexico in the last five decades. As can be expected, project implementation and expected results differed significantly from its design.

Despite limitations in the release and availability of counterpart funding due to the stringent policies applied by the government during the crisis, the project had a positive impact in several areas, as indicated in the previous section. These include: surpassing established goals for production, yields, and incomes of coffee producers, resolution of agrarian conflicts, construction of rural roads and bridges in previously inaccessible areas, beneficiary management of irrigation works, the development of appropriate technologies for basic grains production, the establishment of CADERs in rural marginalized areas, economic diversification, use of aerial photography for the planning of integrated development of natural resources and forestry, soil conservation efforts with the planing of maguey, and development of non-farm activities.

The project was negatively affected by the delay in the release of funds, which did not coincide with the agricultural cycle. Construction of physical works were severely hampered as well, with work in progress left until the next release of funds. A lack of technical assistance accompanying the delivery of inputs negatively affected the project, as in the case of livestock, bees, fruit trees, etc.

There were 15 federal and state agencies involved in implementing the project, with a Coordination Unit of the state government to coordinate and orchestrate activities among all. On the other hand the SPP state Delegation played a key role in releasing funds, thus limiting the role of the Coordination Unit. Despite these constraints, the Coordination Unit was better able to coordinate activities through active participation on regional planning subcommittees.

The project demonstrates the complexity of working with indigenous groups in terms of language, cultural and access problems. It raises the issue of the need to train or include indigenous people in the project design, as IFAD is doing under the Guaymi project in Panama.

Although incorporated at the design phase, the project did not use the services of anthropologists in an area with highly diverse ethnic populations. Since the project began in the Chatina area, it became known as the ‘Chatina project," creating false expectations among the Chatino ethnic group that it was to serve only them, whereas they were the predominant group in only one of the several sub-areas of the total project area. The project also worked with Zapotec, Mixe and Mestizo ethnic groups. INI was assigned a major role as the representative of the various indigenous groups in the area, and as the mediator between them and the various government institutions. In practice it did not perform this function well.

Although the monitoring and evaluation units collected useful information, key indicators were not applied to measure project impact on beneficiaries. The project overemphasized the delivery of inputs in status reports and annual reports, rather than the actual application and results of application or utilization of inputs delivered.

The ex-post IRR of 25% surpassed appraisal projections of 18%. The nutritional impact was positive, and higher in areas of high project intervention than in areas of low intervention.

The project had a positive effect on strengthening federal and state organizations' personnel in implementing rural development projects, and in working with indigenous populations. In addition, non-government community organizations were strengthened through their participation in project activities.

Activities were too dispersed and small to have an impact of any magnitude in any one area (with a few exceptions), given the fragmented approach to providing services.

Lessons and recommendations

In order to speed the process of funds release, the fiscal and agricultural cycles have to be made more compatible through existing or new methods. Assistance could be provided to train staff in the preparation of work plans to expedite the process required for fund release. Another mechanism would be for IFAD to operate a ‘special funds' account which could by-pass the lengthy government procedures.

Project managerial, informational and financial transparency is necessary for effective implementation. This implies establishing clear lines of communication and institutional responsibility for project performance by components. A system of managerial accountability needs to be developed early on in the project life, with overall supervision and decision-making responsibility given to a principal institution or unit, with the leverage to exercise its authority. The number of executing agencies involved should be minimized, in order to increase efficiency, and reduce cumbersome and costly coordination needs. A ‘too many cooks' structure and lack of clear implementation accountability mired project implementation.

A diagnosis of the existing institutional capacities to implement a project of this size and complexity should be conducted prior to project initiation. An institutional strengthening component may be the necessary first step to lay the groundwork for project implementation.

If the target population includes indigenous peoples, the project design phase should consider cultural aspects of the target population which have important implications for rural development project design and implementation; such as division of labour, household structure, production and cultivation methods, appropriate technical packages, range of economic activities, etc. Financial resources for monitoring project performance on this population through case studies etc. should be committed. Such projects should employ extension workers who are fluent in the language of the populations they are to assist. It is critical to consult with and involve representatives of the target population in all decisions pertaining to the project.

Although the project made some strides towards increased beneficiary participation, the objective of participation is for beneficiaries to assume ‘ownership' of the development process. While this occurred in some cases (irrigation), in general it did not. The project did not make use of existing forms of communal participation, such as the prevalent ‘tequio' system of community members providing voluntary labour for communal projects. Instead, ‘participation' was encouraged through paying wages to beneficiaries to construct project-financed infrastructure.

The project provided all services gratuitously, thus debilitating the ‘beneficiary ownership' process, and not institutionalizing the process or making it even partially self-sustaining. Rural development projects such as this one should strive to achieve at least partial financial self-sustainability, in order to ensure sustained benefits beyond the project's funded life. Cost-recovery can be achieved through charging user fees, partial or no subsidization of inputs and credit, and fees for technical assistance. Such a demand-driven approach also makes project services more responsive to the target group's needs.

The sequencing of activities is crucial to successful project implementation. Too many activities were conducted simultaneously. Sequencing of activities, although planned, did not occur. A ‘pilot test' approach where a host of activities are concentrated in a limited area, with sufficient time for a ‘gestation period' should be used, where tested results can then be replicated in other project areas.

Women were not adequately considered in the project design and specific efforts were not made to identify their needs. Women's development constraints should be considered at the design phase, to incorporate if necessary, specific measures to address their specific constraints. Future projects should provide training in agro-processing, and other more profitable activities than in the traditional low market potential areas that the project focused on, such as handicraft production.

A future project in the area should provide explicit support for the development of microenterprises. This could be done through support of sustainable intermediary institutions such as ARIPO (the Oaxaca Agency for Artisan Promotion), which has become an important market link between artisans and exporters of their products. The creation of rural non-farm activities (RUNFAs) is of particular importance to Oaxaca given the limited agricultural base, with the total cultivable area representing less than 10% of the total area. A partial solution to stem the rural-urban flow would be to expand the base of economic activity in the area, in light of the limited agricultural potential.

The appropriate plantation of maguey and nopal is a valid protection against soil erosion in arid areas. This method was employed by the project. These plants also provide raw materials for a host of other products. Maguey can be used for producing fibers, honey, syrups, and pharmaceutical steroids, thus stimulating the development of backward and forward linkages.

Reforestation efforts linked with productive wood-based activities have a greater probability of success than as an effort in of itself.

Given the multiplicity and duplicity of activities by various government agencies in the project area, beneficiaries were unable to distinguish the IFAD project from others. The low profile of IFAD in the project has implications for the evaluation of such projects, in that it impedes being able to distinguish project activities and beneficiaries from others, and constrains knowing what to attribute benefits or changes to. IFAD projects should be identifiable through plaques and signs, to increase visibility, for both the project and IFAD. The requirement to ‘publicize' the projects could be stipulated by IFAD in the loan agreement.

IFAD should promote an exchange of experiences in its efforts both regionally and worldwide, since many of the constraints faced, solutions proposed and lessons learned share common features, and have wider applicability than to a specific project or country.

In conclusion, while the Oaxaca project has been criticized as being largely ineffective, the ex-post evaluation mission was able to identify several project activities which had positive sustained impact three years after project funding ended. The project has provided lessons on how to generate sustained benefits in other similar rural development efforts, as well as shed light on what pitfalls to avoid under similar circumstances.