Ex-post evaluation
Introduction
IFAD’s Monitoring and Evaluation
Division conducted an ex-post evaluation of the Oaxaca Integrated
Rural Development Project in October, 1991. The evaluation’s
purpose was to assess the project’s impact on target beneficiaries,
as well as to draw lessons from this experience on how to improve
rural development assistance efforts applicable both to future project
design and implementation in Mexico, and to a wider context.
Background and socio-economic context
The project area is one of the poorest in the State of Oaxaca, which
in turn is the poorest state of Mexico. Oaxaca has largest concentration
of indigenous populations. With over 12 languages, a poorly developed
infrastructure, and only 10% of the arid land cultivable, Oaxaca represents
a particularly difficult context for project implementation.
Although the project was designed by IFAD at the end of the 1970’s
during Mexico’s economic boom fuelled by the increase in petroleum
prices, the project implementation period (1981-1988) coincided with one
of the worst and most prolonged economic crisis suffered by Mexico in
the last five decades. From a period of growth with relative price stability,
the Mexican economy entered a period of unprecedented hyperinflation,
which tripled from 30% in 1981 to 518 in 1982, with a 40% devaluation
over the same period. At the same time Mexico’s terms of trade deteriorated
from a level of 185.4 in 1982 to 87.4 in 1988 (1987=100), as world interest
rates rose steeply.
The effects of hyperinflation led to a considerable increase in local
currency, while the Government policy of public expenditure austerity
reduced the availability of counterpart funding. Project disbursements
were negatively affected by the contraction of public spending, both in
terms of decreased amounts and increased delays in releasing funds. Funds
were not released at the federal level for periods of up to seven months,
halting all project activity underway.
Project description
The Oaxaca Rural Development Project was initiated by IFAD in May, 1980
as a five year project (subsequently extended by three years) with IFAD
providing USD 22 million of the total estimated project costs
of USD 57.2 million. IFAD’s loan terms were for a 15 year loan,
at an annual interest rate of 8% and a three year grace period. Remaining
funds in the amount of USD 35.2 million were to be provided by the
Government of Mexico. The World Bank was IFAD’s cooperating institution.
The project represents IFAD’s first initiative in Mexico, and was
intended as a follow-on to the earlier World Bank financed PIDER project.
Although PIDER was a nation-wide effort, the Oaxaca project was limited
initially to the rural poor of the Chatina region (in the districts of
Juquila and Sola de Vega), and in Miahuatlan and Pochutla. As of 1984,
the project extended into the districts of Putla, Yautepec and Jamiltepec.
Project objectives and components. The Oaxaca project aimed to
reach the smaller and less privileged farmers in remote areas who were
not adequately reached by PIDER. The principal objective was to increase
the standards of living of the poor population of the inland and coastal
zones through economic and social improvements. The project design assumptions
were that the indigenous population was strongly motivated to remain in
the area, and the under-exploited resource base would be able to support
the population on a sustainable basis. The project intended to:
i) Substantially increase food crop and livestock production;
ii) generate employment and income opportunities through agri-cultural
diversification;
iii) increase fish production and consumption, and increase poor fishermen’s
income;
iv) improve living and working conditions of the target population
through improvements in health, infrastructure, service provision;
v) protect the natural resource base through soil conservation.
Project objectives would be met through the following components:
i) agricultural production services (increased and improved
extension services, land titling, credit provisions, institutional strengthening
of INI (National Institute for Indigenous People);
ii) agricultural development (pasture improvement and livestock
development, fruit and coffee production, small-scale irrigation programs,
soil conservation;
iii) forestry development (aerial photography, nursery establishment,
reforestation);
iv) fisheries development (training and infrastructure development,
fish marketing assistance);
v) community development, farmers’ participation and project
management (promotion of community organizations, beneficiary involvement
and participation in planning and execution, project management, monitoring,
evaluation, coordination, etc.);
vi) improvement of infrastructure (rural road rehabilitation
and construction, basic community health services through water supply
provisions).
Target Beneficiaries. The project was expected to benefit 15 000
small farmer families, or 84 000 of the poorest people, mostly indigenous,
with an average per capita income at the time of appraisal of USD 60
for farmers, and USD 100 for fishermen.
Project Implementation. The project was implemented by a consortium
of 15 federal and state agencies, under the coordination of the Ministry
of Programming and Budgeting (SPP). As an attempt at decentralization,
major responsibility for planning, execution and supervision was placed
at the State level, with disbursements made at the federal level. A Coordinating
Unit was established within each of the two project regions to coordinate
between the SPP delegation in Oaxaca and project beneficiaries.
Project implementation
The project’s implementation strategy characteristics were as follows:
A concentration of highly diversified activities in a few limited areas,
and fragmented activities dispersed over a large number of communities;
Inopportune and fully subsidized provision of inputs, with insufficient
and inadequate complementary technical assistance;
A combination of federal and state institutions supporting various
and at times the same activities, with progressive responsibility given
to the state agencies;
Provision of services considered priorities by the target communities,
even if outside the defined realm of project activities, as a means
to penetrate the project area. An example is the construction of basketball
courts, or municipal buildings.
Project Implementation Mechanisms
Resource Approval and Allocation Process. The delay in funds approval
and disbursement was one of, if not the key, factor negatively
affecting the project’s implementation. Funds were often disbursed
at inopportune moments which did not correspond to the agricultural cycle.
Inputs were not delivered in the critical periods in the agricultural
cycle. In addition, the executing agencies lacked competent personnel
with the ability to develop the work plans with adequate leeway to go
through the procedures and channels for the timely release of funds.
The project had a late start, with only 7% of funds disbursed in the
first two years. Activity increased in 1983. By 1985, the initial close
of the project, only two thirds of total IFAD funds had been disbursed.
Beneficiary Participation in Project Implementation. Although
on a limited scale, the project introduced a series of participative mechanisms,
such as ‘participative planning forums’ in 1985; collaboration
with the Regional Council of Marginalized Peoples which represented 16
communities in the project area; planning of project activities in consultation
with elected representatives of the target population; such as the municipal
authorities, and with elected representatives of indigenous groups.
Supervision. The multiple strata of supervision complicated project
implementation, without resolving problems identified, given the clear
lack of accountability for performance to institutions. The key problems
repeatedly identified were the delays in fund disbursement, and the lack
of coordination among the executing agencies. The various layers of supervision
were provided by the project’s Coordination Unit; 15 state and federal
level executing institutions; and the World Bank. In some instances, such
as in the case of road construction, there was direct beneficiary supervision.
Despite the high degree of supervision, recommendations made by the 14
World Bank and IFAD supervision missions over the project life often were
not relayed to the executing agencies or to the Coordination Unit to be
acted upon.
Monitoring and evaluation. Although the project appeared to have
a large M&E apparatus, with two M&E units; one each for the Chatina
and Miahuatlan/Pochutla regions, this component was not adequately implemented
and utilized for project management purposes. Despite the large amount
of resources apparently dedicated to M&E and the large structure for
this function (nine professionals, administrative staff and two vehicles)
in reality these resources were thinly spread over many other activities
as well. No computers were available for data synthesis.
The M&E units processed information received from the executing agencies
and the supervision memorandums. Monitoring by executing agency, program
component, and community was presented in semestral reports with data
on physical advances and, in some cases, financial expenditures. The Coordination
Unit proposed recommendations on the basis of monitoring reports to the
various executing agencies which were at times considered and acted upon.
Key indicators established at the time of appraisal were not utilized
to measure project performance. The M&E unit did not collect the type
of data necessary for this team to have measured the project’s impact
on beneficiaries. While ‘output’ data on project activities
was maintained, such as ‘kilometers of roads constructed,’ indicators
were measuring the impact of these activities on project beneficiaries
were neither established nor measured. External factors affecting project
performance such as product prices, input prices, climate effects, etc.
were not tracked or analyzed through the M&E system.
Project Coordination. The complex project implementation structure
and the lack of coordination of project components repeatedly cited by
the supervision missions as a critical problem affecting the project.
The project design established a Coordination Unit linked to the State
Government as a mechanism to support the Government’s decentralization
efforts. Tensions between the federal and state governments exacerbated
by the funds disbursement problem negatively affected the project’s
implementation.
The Coordination Unit was able to better coordinate activities in the
State of Oaxaca with the 1984 creation of the regional subcommittees of
the State Planning Committee for the Development of Oaxaca. The Project
Coordinator was nominated as Technical Secretary in the subcommittees
for Sierra Sur and Costa. This permitted integrated planning of the diverse
state coordination meetings. The new coordinator since 1987 was also designed
the State government delegate in Miahuatlan, which included areas not
in the project zone.
The complexity of the project design called for intense coordination,
without adequately considering the existing administrative and institutional
constraints limiting the coordination Units’ maneuverability.
Project results
Economic Analysis. The project’s internal rate of return
(IRR), calculated using 1979 deflated prices as the base year to facilitate
comparison with the appraisal IRR conducted in 1979 was 25%, as compared
to the appraisal’s projected IRR of 18%. This can be attributed in
large part to higher prices for project produced goods, and lower labour
and input costs than estimated at the appraisal stage.
Nutritional Impact. The Duval method to assess nutritional impact
determined that in all areas of the project, ranging from low to high
intervention, the population’s nutritional status improved over the
implementation period. Greater impact was achieved in high intervention
over low intervention areas. Positive impact in this area was also achieved
through DIF’s interventions in nutritional and hygiene education.
Productive Diversification. One of the project’s explicit
objectives was to diversify the employment and income-generating activities
through fruit, coffee, honey, and other types of cultivation. While fruit
cultivation had a limited impact, given the untimely supply of trees,
inadequate technical supervision, and poor coordination among executing
agencies, the limited success has had a demonstration effect. The honey
production activities were not successful, again, in large part, due to
the lack of technical assistance accompanying the provision of inputs.
The coffee component had very positive results, leading to expanded surfaces
cultivated, production, municipalities and producers involved and yields.
This was in large part due to the efforts of IMMECAFE, the National Coffee
Corporation, in developing an appropriate technical package.
The project led to agricultural diversification, since the increase in
the area dedicated to coffee production did not lead to a decrease in
food crop (primarily maize) production.
Increased Access and Reduced Costs. The construction of 639 kilometers
of rural roads has reduced transportation costs, and the price of goods,
while increasing access to markets and urban centres.
Slowdown of Migration. The project appears to have slowed down
the patterns of out-migration, through having created economic opportunities
in the project area. Population in the four districts where the project
had a major presence; Juquila, Sola de Vega, Miahuatlan, and Pochutla
increased in the 1980s in a significantly higher proportion than the State
of Oaxaca as a whole. The increase in Oaxaca between 1980 and 1990 was
28%, while in the four districts mentioned it was 35, 38, 34 and 53 percent
respectively over the same period.
Small Enterprise Development. Although not initially planned,
the project assisted in the development of various sustainable microentrepreneurial
activities, in diverse sectors, such as wood, textiles, and agro-processing.
The majority of these activities employed primarily women.
Sustainable Irrigation Development. Two of the five irrigation
works have created irrigator’s committees, which collect user fees
to cover operating, maintenance and water costs.
Institutional Development. The project constructed two rural development
centres (CADERs) in Sola de Vega, and Santa Reyes Nopala, the poorest
and most indigenous areas respectively in Oaxaca. Both centres were operating
at the time of this evaluation as offices for the various executing agencies,
serving as a focal point for the integration and coordination of State
development efforts.
Demonstration Effects. The project has had a positive demonstration
effect in several areas. Fruit cultivation although not realized during
the project, is now being realized. The project has led to a sustained
effort by government institutions in the area where they were not previously
active, and forced attention to meeting the needs of indigenous peoples.
Limited Impact Due to Scale of Activitites. The fragemented delivery
of project interventions limited the magnitude of the potential impact.
Although the project appears to have bettered conditions, the dispersion
and minuteness of the interventions did not lead to overall noticeable
changes, with the exception of the coffee producers.
Lack of Deliverables Reaching the Target Groups: In many cases,
the project’s deliverables never reached the intended beneficiaries.
The fisheries component provides such an example. The project intended
to give some 70 boats and 100 outboard engines on credit to individual
and cooperative fishermen. Of the reportedly 22 engines given, not one
was functioning by early 1987. No credit recovery was enforced by the
Dirección de Desarrollo Pesquero del Estado. The two of four planned refrigeration
units were underutilized, one for lack of catch and one for being out
of order since 1985, according to the February 1987 mission supervision
report. A visit to coastal fishery sites by this mission found that reported
deliveries had never been made.
Conclusions and lessons from the experience
Conclusions
The project was designed at the end of the 1970s when Mexico was enjoying
an economic boom fuelled by increased petroleum prices, but was implemented
from 1981 to 1988, a period of one of the worst economic crisis faced
by Mexico in the last five decades. As can be expected, project implementation
and expected results differed significantly from its design.
Despite limitations in the release and availability of counterpart funding
due to the stringent policies applied by the government during the crisis,
the project had a positive impact in several areas, as indicated in the
previous section. These include: surpassing established goals for production,
yields, and incomes of coffee producers, resolution of agrarian conflicts,
construction of rural roads and bridges in previously inaccessible areas,
beneficiary management of irrigation works, the development of appropriate
technologies for basic grains production, the establishment of CADERs
in rural marginalized areas, economic diversification, use of aerial photography
for the planning of integrated development of natural resources and forestry,
soil conservation efforts with the planing of maguey, and development
of non-farm activities.
The project was negatively affected by the delay in the release of funds,
which did not coincide with the agricultural cycle. Construction of physical
works were severely hampered as well, with work in progress left until
the next release of funds. A lack of technical assistance accompanying
the delivery of inputs negatively affected the project, as in the case
of livestock, bees, fruit trees, etc.
There were 15 federal and state agencies involved in implementing the
project, with a Coordination Unit of the state government to coordinate
and orchestrate activities among all. On the other hand the SPP state
Delegation played a key role in releasing funds, thus limiting the role
of the Coordination Unit. Despite these constraints, the Coordination
Unit was better able to coordinate activities through active participation
on regional planning subcommittees.
The project demonstrates the complexity of working with indigenous groups
in terms of language, cultural and access problems. It raises the issue
of the need to train or include indigenous people in the project design,
as IFAD is doing under the Guaymi project in Panama.
Although incorporated at the design phase, the project did not use the
services of anthropologists in an area with highly diverse ethnic populations.
Since the project began in the Chatina area, it became known as the ‘Chatina
project," creating false expectations among the Chatino ethnic group
that it was to serve only them, whereas they were the predominant group
in only one of the several sub-areas of the total project area. The project
also worked with Zapotec, Mixe and Mestizo ethnic groups. INI was assigned
a major role as the representative of the various indigenous groups in
the area, and as the mediator between them and the various government
institutions. In practice it did not perform this function well.
Although the monitoring and evaluation units collected useful information,
key indicators were not applied to measure project impact on beneficiaries.
The project overemphasized the delivery of inputs in status reports and
annual reports, rather than the actual application and results of application
or utilization of inputs delivered.
The ex-post IRR of 25% surpassed appraisal projections of 18%. The nutritional
impact was positive, and higher in areas of high project intervention
than in areas of low intervention.
The project had a positive effect on strengthening federal and state
organizations’ personnel in implementing rural development projects,
and in working with indigenous populations. In addition, non-government
community organizations were strengthened through their participation
in project activities.
Activities were too dispersed and small to have an impact of any magnitude
in any one area (with a few exceptions), given the fragmented approach
to providing services.
Lessons and recommendations
In order to speed the process of funds release, the fiscal and agricultural
cycles have to be made more compatible through existing or new methods.
Assistance could be provided to train staff in the preparation of work
plans to expedite the process required for fund release. Another mechanism
would be for IFAD to operate a ‘special funds’ account which
could by-pass the lengthy government procedures.
Project managerial, informational and financial transparency is necessary
for effective implementation. This implies establishing clear lines of
communication and institutional responsibility for project performance
by components. A system of managerial accountability needs to be developed
early on in the project life, with overall supervision and decision-making
responsibility given to a principal institution or unit, with the leverage
to exercise its authority. The number of executing agencies involved should
be minimized, in order to increase efficiency, and reduce cumbersome and
costly coordination needs. A ‘too many cooks’ structure and
lack of clear implementation accountability mired project implementation.
A diagnosis of the existing institutional capacities to implement a project
of this size and complexity should be conducted prior to project initiation.
An institutional strengthening component may be the necessary first step
to lay the groundwork for project implementation.
If the target population includes indigenous peoples, the project design
phase should consider cultural aspects of the target population which
have important implications for rural development project design and implementation;
such as division of labour, household structure, production and cultivation
methods, appropriate technical packages, range of economic activities,
etc. Financial resources for monitoring project performance on this population
through case studies etc. should be committed. Such projects should employ
extension workers who are fluent in the language of the populations they
are to assist. It is critical to consult with and involve representatives
of the target population in all decisions pertaining to the project.
Although the project made some strides towards increased beneficiary
participation, the objective of participation is for beneficiaries to
assume ‘ownership’ of the development process. While this occurred
in some cases (irrigation), in general it did not. The project did not
make use of existing forms of communal participation, such as the prevalent
‘tequio’ system of community members providing voluntary labour
for communal projects. Instead, ‘participation’ was encouraged
through paying wages to beneficiaries to construct project-financed infrastructure.
The project provided all services gratuitously, thus debilitating the
‘beneficiary ownership’ process, and not institutionalizing
the process or making it even partially self-sustaining. Rural development
projects such as this one should strive to achieve at least partial financial
self-sustainability, in order to ensure sustained benefits beyond the
project’s funded life. Cost-recovery can be achieved through charging
user fees, partial or no subsidization of inputs and credit, and fees
for technical assistance. Such a demand-driven approach also makes project
services more responsive to the target group’s needs.
The sequencing of activities is crucial to successful project implementation.
Too many activities were conducted simultaneously. Sequencing of activities,
although planned, did not occur. A ‘pilot test’ approach where
a host of activities are concentrated in a limited area, with sufficient
time for a ‘gestation period’ should be used, where tested results
can then be replicated in other project areas.
Women were not adequately considered in the project design and specific
efforts were not made to identify their needs. Women’s development
constraints should be considered at the design phase, to incorporate if
necessary, specific measures to address their specific constraints.
Future projects should provide training in agro-processing, and other
more profitable activities than in the traditional low market potential
areas that the project focused on, such as handicraft production.
A future project in the area should provide explicit support for the
development of microenterprises. This could be done through support of
sustainable intermediary institutions such as ARIPO (the Oaxaca Agency
for Artisan Promotion), which has become an important market link between
artisans and exporters of their products. The creation of rural non-farm
activities (RUNFAs) is of particular importance to Oaxaca given the limited
agricultural base, with the total cultivable area representing less than
10% of the total area. A partial solution to stem the rural-urban flow
would be to expand the base of economic activity in the area, in light
of the limited agricultural potential.
The appropriate plantation of maguey and nopal is a valid protection
against soil erosion in arid areas. This method was employed by the project.
These plants also provide raw materials for a host of other products.
Maguey can be used for producing fibers, honey, syrups, and pharmaceutical
steroids, thus stimulating the development of backward and forward linkages.
Reforestation efforts linked with productive wood-based activities have
a greater probability of success than as an effort in of itself.
Given the multiplicity and duplicity of activities by various government
agencies in the project area, beneficiaries were unable to distinguish
the IFAD project from others. The low profile of IFAD in the project has
implications for the evaluation of such projects, in that it impedes being
able to distinguish project activities and beneficiaries from others,
and constrains knowing what to attribute benefits or changes to. IFAD
projects should be identifiable through plaques and signs, to increase
visibility, for both the project and IFAD. The requirement to ‘publicize’
the projects could be stipulated by IFAD in the loan agreement.
IFAD should promote an exchange of experiences in its efforts both regionally
and worldwide, since many of the constraints faced, solutions proposed
and lessons learned share common features, and have wider applicability
than to a specific project or country.
In conclusion, while the Oaxaca project has been criticized as being
largely ineffective, the ex-post evaluation mission was able to identify
several project activities which had positive sustained impact three years
after project funding ended. The project has provided lessons on how to
generate sustained benefits in other similar rural development efforts,
as well as shed light on what pitfalls to avoid under similar circumstances.
