Completion evaluation

Introduction

Background. After Mongolia joined IFAD in 1994, discussions between the Government of Mongolia (GOM) and the Fund led to the formulation of the first IFAD-financed project in the country, the Arhangai Rural Poverty Alleviation Project (ARPAP).  An IFAD loan of SDR 3.45 million (approximately US$5.04 million) was approved by IFAD's Executive Board in April 1996 and the project was declared effective in the same year.  Between 1996 and 1998 the project was implemented in the Arhangai province; after the Mid-Term Review (MTR) in 1998, the project scope and implementation was extended to the Huvsgul province. Following an extension of six months, the project was completed in December 2003, while the repayment of the livestock loan will continue till 2012.

The completion evaluation mission visited the country between 1 and 30 September 2005. The objectives of the evaluation mission were to: (a) assess and document the results, impacts and effectiveness of the project; and (b) draw lessons learned applicable to the design of future projects.  The mission held working meetings in Ulaanbaatar with authorities from the Ministry of Food and Agriculture (MOFA), the Ministry of Finance (MOF), and various stakeholders, including representatives of other development agencies operating in Mongolia.  A total of 20 sums within the project area were visited by the mission members and discussions were held with a cross-section of stakeholders, including beneficiaries.  A household survey covering representatives of 180 households was conducted during the mission by the Centre for Policy Research.

Changing conditions during the lifetime of ARPAP. At the time of project design, Mongolia was a country in transition following the withdrawal of Soviet subsidies. The urban economy was in decline, and for the first half of the 1990s, this was reflected in a flow of population back to the countryside. However, the economy of Mongolia changed radically during project implementation, following the liberalisation of the mining sector, the growth in export markets for livestock products and the back-migration of rural populations to the towns.

Project features. Given the limited experience in project management of GOM officials in the mid-1990s, the project design incorporated the following features:

• moderate number of beneficiaries with an initial phase in one aimag (province) to assess local absorptive potential and strengthen institutional capacity;
• simplified components and interventions to address the needs of the rural poor in a gradual manner;
• beneficiary involvement in defining the composition and number of the herds, and strong awareness of the terms and conditions for repayment;
• location chosen in terms of ecological zones and pastoral system, so that the intervention could be replicated in other areas;
• land tenure reforms to guarantee access to pasture for poor and women-headed households and to include elaborated policies/regulations for detailed guidance for implementation; and
• human resources development through appropriate training of management personnel and individual herders.

The design drew on IFAD experience and took into consideration the environmental/rangeland degradation risks and herders' preparedness for risk management.

The framework for IFAD assistance comprised:

• Targeting the poorer segment of herding communities and the less advantaged area with special attention to herding households with few animals and those headed by women;
• Promotion of income-generating activities (IGA), especially for poor households outside the livestock production system;
• Restoring animal health services to reverse the rising trend in mortality rates of both adult and newborn stock; and
• Provision of support for improving livestock marketing and removal of remaining government controls to redress the deteriorating terms of trade, facilitate access to essential inputs and broaden the market opportunities for poor households.

The project had four components:

• Livestock distribution
• Vegetable production
• Project implementation and institutional support
• Technical assistance (TA), studies and training

Project area. Arhangai aimag was chosen because it is almost entirely rural and heavily dependent on livestock production, with a high number of poor households, and communication with Ulaanbaatar. The target group was to consist of a mix of residents of aimag/sum centres and rural households. Initial beneficiaries were 6 600 poor and very poor households, of which 2 000 were to be herding households and very poor households, possibly with some animals. The proposed extension to a second aimag added 4 600 poor and very poor households, bringing the total to 12 000 households.

Livestock distribution. The main activity of the project was an in-kind livestock credit scheme, with beneficiaries who were full-time and experienced herders owning a maximum of 20 bod (traditional large ruminant unit) of animals. The maximum loan size was to be a minimum of five and a maximum of ten bod, with an average of seven bod.  The borrowers were to repay the equivalent of two mature offspring for each breeding animal with a repayment period of six years, including a one year grace period for sheep, and nine years including a three years grace period for cattle and yak.

Vegetable production and income generation. Credit was provided to households officially classified as poor and very poor to purchase input packages for vegetable gardens and IGA. The loan size for vegetable production was US$30 without provision of fencing and US$60 with fencing. The estimated number of beneficiaries was 1 800 households.

Implementation. At the national level, overall responsibility for project implementation, supervision, monitoring and liaison with IFAD and other government agencies rested with MOFA.  At the aimag level, a Project Implementation Unit (PIU) was responsible for coordination and implementation of the project activities. A Project Coordination Committee under the aimag Governor was responsible for the review of progress, approval of annual work plans and budgets and coordination.

Subsidiary Loan Agreement. The original Subsidiary Loan Agreement (SLA) signed between MOF, MOFA and Arhangai Aimag in 1997 stipulated that all funds drawn from the IFAD loan were repayable by the Aimag (contrary to the provision of the IFAD/GOM Loan Agreement whereby only funds required for the Livestock Distribution and Restocking Credit would be relent to the Aimag). The terms of the SLA were denominated in US dollars, thereby passing the foreign exchange risk to the Aimag. At the recommendation of the MTR in 1998, amended SLAs with Arhangai and Huvsgul were finally signed in May 2001.

Summary of implementation results   

Quantitative targets set by the Appraisal Report and revised by the MTR were exceeded for livestock restocking (72 per cent more animals were redistributed to 41 per cent more herders). A total of 3 494 households (1 433 in Arhangai and 2 061 in Huvsgul) received restocking loans. An average of 9.5 bods per household were distributed at an average cost of T 597 000 per household, against an appraisal cost of T 625 000.

Loan recovery in Arhangai was 100 per cent during the first two years, but declined over the lifespan of ARPAP, particularly following heavy losses of animals due to the dzuud (a freezing over of the snow in spring which makes it impossible for grazing animals to penetrate to the grass below) of 1999, 2000 and 2001, and the insufficient compensation paid to herders by the insurance scheme.  In Arhangai, only 28.6 per cent of those who took loans in 1997 are still repaying regularly, while in Huvsgul, 55.4 per cent of those beneficiaries who took loans in 1997 are still repaying on a regular basis.  Vegetable production activities included provision of loans to 4 276 beneficiaries through supply of seeds, tools, herbicides, etc. Repayment rates were 85 per cent in Arhangai and 71.8 per cent in Huvsgul.

Project performance

The initial project design and objectives were relevant both to GOM policy and to IFAD's mandate in that:

• -the credit component was intended to target the poorest households, which in the Mongolian context were those with few or no livestock holdings;
• credit in kind was intended to protect beneficiaries from price fluctuations in livestock prices in that if beneficiaries had to repay in cash and prices fell, they would be in a debt trap;
• credit for female-headed households was emphasised in the design; and
• research on issues of pasture sustainability, marketing and haymaking were all included in the design.

Relevance. IFAD sought to identify a model to replicate in other aimags and to assist poor herder households make full use of the potential advantages of a market-oriented economy, while ensuring adequate social and economic support services.  The project aimed also to reduce localised overgrazing around sum/aimag centres. The overall relevance rating of the project, in the light of the evolving economic and political conditions at the time of appraisal is 5.

Effectiveness.  An increase in the annual income and assets of poor and very poor herders was achieved only to a limited extent, since in Arhangai a majority of beneficiary households reported a decrease in herd size. In Huvsgul, most beneficiaries reported an increase. It should be noted that the project was operating within the context of the reforms introduced by the Asian Development Bank (ADB) Agriculture Sector Loan, whereby the role of the state to support herders in emergency situations like the dzuud of 1999, 2000 and 2001 was minimal, and emergency fodder reserves were extremely low.  However, it is also questionable whether the beneficiaries were ‘poor and very poor herders'; as the majority of beneficiaries had over 15 bods, which is close to a viable herd.

Regarding improved nutrition, reduced food expenditure and/or increased incomes of other poor and very poor households, the pattern in Arhangai and Huvsgul shows that production of vegetables and potatoes rose markedly during the period of implementation of the project, and then waned after project closure. Although the component appeared to be targeted correctly, the evidence suggests that it was not sustainable and has had little or no long-term impact on the incomes of poor households.

Concerning the developing of a replicable model of livestock distribution and vegetable production to implement in other aimags, the project approach did not take fully into account the IFAD experience with restocking projects over a long period.  These restocking projects have shown that this approach has not been very successful for a wide variety of reasons. Moreover, through discussions held by the evaluation team with donors in the country, it was evident that while restocking provided an important short-term safety net in rural Mongolia in the late 1990s, GOM and donors have now come to appreciate that it also raises serious concerns about long-term sustainability.  The vegetable production component performed higher than the restocking one; yet, a number of constraints to the effective implementation of the component were noted, such as: (i) limited marketing at soum level; (ii) shortage of good quality seeds; and (iii) water scarcity.  In light of the above considerations, the effectiveness of the project is rated 3.

Efficiency.  If the project were considered purely in terms of credit delivered against cost, the ARPAP was very efficient. Not only were more loans delivered than projected, the PIU also only spent 80 per cent of the budget, due to favorable foreign exchange terms. This was also possible due to the elimination of all TA and research components. In addition, animals for distribution were bought at fair prices on the market and inputs were delivered in a timely manner. In both cases, the beneficiaries interviewed had no complaints about the performance of the project. However, the issue is whether focusing on credit to the exclusion of all other types of public good (such as pasture research) is a desirable way to manage a project.  The efficiency of the project is rated 4.

Performance of IFAD.  IFAD underestimated the difficulties of operating in Mongolia.  Contextual difficulties were compounded by lack of country knowledge and country analysis capabilities.  The project design was based on IFAD experience from similar projects, and took into consideration the environmental/rangeland degradation risks and herders' preparedness for risk management.  There were, however, major design weaknesses, most notably: (a) inadequate awareness of the impact of dzuud; (b) an inadequate strategy for responding to other sources of animal loss; and (c) a failure to consider the viability of insurance. In addition, as some components were never implemented and as credit repayment has been operated in a manner not in line with IFAD standards, but unknown to the Fund until approaching time of project closure, IFAD's support to implementation was weak.  The rating is 3.

Performance of Government.  The GOM has continued to make poverty reduction a high priority and has generally been supportive of the project at the central government level. But its policies have not always had this effect on the ground. A lack of investment in rural areas, a concentration of mining as the major source of government income and a failure to consider the long-term consequences of cashmere production have increased inequality between the urban and rural areas.  The GOM appears to have complied with its agreements with IFAD on a financial level if not in terms of implementation. However, the signing of the SLA effectively undercut the intentions of the original loan agreement. Even when changes were agreed by GOM in principle (such as not passing on foreign exchange risks to the aimag governments, as under the original SLA) these were only slowly carried out in practice. The rating is 3.

Performance of the cooperating institution.  UNOPS carried out 11 supervision missions during the life of the project. Though the quality of supervision varied, the supervision missions identified and made recommendations on all relevant policy and operational issues, although with sometimes inconsistent and even contradictory concerns.  A number of important issues, such as revision of the SLA, implementation of support service activities and others, repeatedly highlighted in supervision reports, remained unresolved for long periods. The rating is 3.

Project impacts

The original target groups for ARPAP were ‘poor and very poor' households, split between herders and those with too few animals to build viable herds. This was to be achieved by setting limits to the bod holdings of potential beneficiary households. Households eligible for vegetable production and income-generating loans were those defined by the Local Development Fund as ‘poor and very poor'. The pressure to ensure repayment, a consequence of the terms of the SLA, encouraged the PIU to increase the level of livestock holdings required for eligibility.  The household survey, conducted at the time of the evaluation, shows that loans were given to 25 per cent of beneficiaries who already had more than 20 bods. If this were scaled up for all beneficiary households, it would suggest that 1 069 households were given loans to which they were not exactly entitled.  Regarding vegetable loan beneficiaries, no data on the economic status of vegetable loan beneficiaries is available in the PCR or from the PIU, but interviews conducted by the evaluation team all suggest that the beneficiaries were appropriate.

Physical and financial assets.  In Arhangai, 54 per cent of households that received animals from 1997 to 2003 reported a decrease in herd size, and only 46 per cent an increase in herd size by 2004. In Huvsgul, 30 per cent of households reported a decrease and 70 per cent an increase in herd size.  Although the main cause of these losses was the dzuud, its negative impact could have been reduced by the activities proposed at design (provision of hay-making machines, promotion of fodder production, etc.).  It is also true that GOM policy in this same period was to divest itself of centralised hay production and distribution, to cease collective wolf hunts and reduce government veterinary services. In such a policy climate, the losses were inevitable, although project design did not take sufficient account of this. Vegetable loans were more successful than livestock loans in improving access to material possessions, but in all cases, the percentage of households benefiting was never more than 50 per cent.  The rating on Physical and Financial Assets is 3.

Human assets.  The project was to implement skills' development training for vegetable producers, aimag and sum staff and for the PIU. In addition, herders were to be informed about repayment procedures, to be kept abreast of livestock prices and veterinary issues through regular radio programmes. The training of herders had limited impact on their livelihood, as training focused mainly on familiarizing beneficiaries with livestock loan procedures and selection criteria instead of skills training on livestock production and pasture management.  Also, although funds were allocated to mobilize consultants for technical advice, no expenditures were made for this purpose. Extension was rather weak: radio programmes were mostly used for disseminating information about the achievement of the project instead of TA for livestock and vegetable production.  The rating is 2.

Food security. If the project had achieved the projected results, the impact on food security would have been considerable. Poor herders would have viable herds, poor sum centre households would have both greater access to vegetables and a potential source of additional income to supplement the protein in their diets. The impact of ARPAP on food security overall has been confined to households where herd growth enabled loan repayment and on those where vegetable production is being sustained. The rating on this impact domain is 3.

The direct environmental impact of ARPAP has been minimal; the marked environmental changes since 1995 result from larger national policies and economic trends. The need to establish a range monitoring system capable of measuring range productivity and its changes over time and under increasing grazing pressures was not fully recognized by the PIU. Although these issues were highlighted in the MTR and in subsequent supervision reports by UNOPS, no action was taken until late in the life of the project. The rating is 2.

The impact on institutions and policies of ARPAP was low. The absence of monitoring, the lack of a practical research element and failure to make use of institutions such as the High Mountain Research Centre in Ikh Tamir (HMRC) gave it no authority to intervene in the policy arena. The problems of the repayment system have impacted most notably on female-headed households. All the households that lost their herds in the recurrent dzuud, are still being actively pursued to repay their debts.  The rating is 3.

Sustainability. Concerning restocking, it is arguable that it could have worked under the environmental conditions present in Mongolia. The frequency of dzuud was known prior to appraisal and it could reasonably be assumed that one or two would occur within the lifetime of the project. The safety nets provided in the Soviet era had all been eliminated by 1995, increasing the potential for animal losses on a greater scale. Given that the direction the Mongolian economy would take was difficult to predict, the project should have been treated as experimental and been more willing to respond rapidly to feedback from the ground with significant design changes. Regarding vegetable production, this was successful in some sums, but in others the growing season is too short for reliable yields. In addition, the increasing number of plagues of grasshoppers that are a consequence of pasture degradation are also attacking vegetable gardens, making the long-term sustainability of this intervention somewhat doubtful.  The rating for sustainability is 2.

The following table shows the ratings of the programme for performance, impact and overarching factors and performance of partners:

Ratings of the Arhangai Rural Poverty Alleviation Project

1/  IFAD uses a scale of 1 to 6, where 1 represents the lower score and 6 the highest.

Conclusions and recommendations

Overall, the impact of the project has been limited, and in a few cases, negative. Some households were assisted to develop a viable herd, but almost as many were saddled with serious long-term debt for which they are still being pursued. This burden has particularly fallen on female-headed households and is the cause of significant personal distress. The vegetable credit is probably the most successful element of the project; and although vegetable growing declined after the project closed, there has been a visible change of culture in sum centres and almost certainly improved nutrition for children. The project design, based on IFAD experience from similar projects, took into consideration the environmental/range land degradation risks and herders' preparedness for risk management. There were, however, major design weaknesses, notably inadequate awareness of the impact of dzuud, an inadequate strategy for responding to other sources of animal loss and a failure to consider the viability of insurance.

Recommendations

Debt relief and livestock loan repayment. A failure to consider the consequences of dzuuds serious enough to eliminate entire family livestock holdings left many households with insurmountable repayment problems. The financial burden on aimags created by the SLA has meant that the poorest households are still being harried for debts, and it is unrealistic to expect them to repay. This burden falls particularly on widows and single mothers because other beneficiaries with more resources have simply decamped.   IFAD and the GOM must thus find an institutional solution to resolve this issue.

Risk preparedness. By 2004, 40 per cent of project beneficiaries had less livestock than when they took the loan. The principal reason was the incidence of dzuud, a climatic phenomenon whose intervals of occurrence was already known prior to project appraisal.  Further dzuuds will occur and losses may well be greater as animal nutrition deteriorates due to degraded pastures.  Effective risk management must include the establishment of safety net mechanisms along with pasture management to avoid localized overgrazing.

Supervision. The project failed to execute many ‘soft' components of the project, such as socio-economic data collection and rangeland monitoring. Both UNOPS and IFAD should have acted to ensure compliance with the formulated recommendations; failure to do so has had adverse consequences for beneficiaries at the later stages of project implementation.  In future, IFAD and its Cooperating Institution need to ensure compliance with the project design and with recommendations of supervision missions.

Rangeland assessment. The rangelands in Arhangai and Huvsgul have undergone considerable damage since the implementation of the project in 1996, as evidenced by the major switch in the pastoral economy towards goat production. This occurred especially after the 2001 dzuud, an occurrence that has been verified by all households interviewed, and supported by the official statistics. Priority should be given to rangeland assessment and be actively and effectively pursued by all parties concerned in the ongoing IFAD Rural Poverty Reduction Programme in Mongolia.  In addition, IFAD should explore and capitalize further on knowledge concerning the technical aspects of extensive pastoralism in low-temperature regions.