Completion evaluation

Introduction

Evaluation objectives and procedures. The Office of Evaluation (OE) of IFAD undertook a Completion Evaluation of the Dir Area Support Project (DASP) from 18 June to 9 July 2007 in order to ensure accountability and to offer an overview of good practices and lessons learned for all stakeholders. This Completion Evaluation was carried out ten years after the DASP loan became effective. It is in line with the IFAD evaluation policy and adopts the latest methodology for project evaluations developed by OE. The latter focuses on three dimensions: (i) the performance of the project measured in terms of relevance, effectiveness and efficiency; (ii) the rural poverty reduction impact of the project measured against impact indicators that are grouped according to impact domains; and (iii) the performance of IFAD and its partners. The evaluation has also assessed the contribution of the project to IFAD’s involvement in policy dialogue, the building of partnerships, knowledge management and the promotion of innovations for replication and up-scaling. In addition, the evaluation has considered the sustainability of project outcomes and impacts and gender equity as a cross-cutting issue.

The evaluation was cognisant of the fact that the project, after having been extended twice, is still ongoing. The DASP evaluation is expected to generate important inputs for the Pakistan Country Programme Evaluation which is also undertaken by OE in 2007. The Completion Evaluation took place from 18 June to 9 July 2007, with a wrap-up meeting in Peshawar held on 9 July 2007 to which all major stakeholders were invited.

The economy, population and poverty issues. Pakistan showed high economic growth in recent years (6.6 per cent in 2006). By contributing 22.3 per cent to Pakistan’s GDP, the agricultural sector was the second largest sector in 2006. Official estimates put the population of Pakistan at 156.9 m in the same year, with over 65 per cent of the country’s population living in rural areas. In 2005, Pakistan still ranked among the poorest quarter of the world’s countries with a per capita GDP based on purchasing power parity (PPP) of only international US$2 per day. Poverty in Pakistan is still predominantly a rural phenomenon with both the headcount rate of poverty as well as the poverty gap being higher in rural than in urban areas. The high economic growth rates and a significant increase in pro-poor public expenditure have had a positive impact on poverty reduction with consumption poverty (measured by the national poverty line) falling from 34.5 per cent in 2001 to 23.9 per cent in 2005. A vast majority of the poor are small landowners, sharecropping tenants, and landless labourers.

Causes for rural poverty are related to numerous development constraints, which include: inefficiencies in rural service delivery, inadequate infrastructure such as irrigation, drainage, rural roads, farm commodity markets, warehouses and cold storage facilities; lack of compliance with tenancy laws; limited rural financial markets; and a lack of incentives for the private sector to set up new agribusinesses and enterprises.

IFAD’s strategy in Pakistan. The Country Strategic Opportunities Paper (COSOP) of 2003 puts the main emphasis on: (i) marginal and resource-poor geographical areas; (ii) an institutional framework permitting target group influence and project interventions in support of poverty alleviation activities; (iii) the development and dissemination of technology suitable to be adopted by poor people; (iv) complementarity of rural infrastructure development in area based projects; and (v) the need for employment generation.

The project. The project area covers the central and western part of the districts Upper Dir and Lower Dir, which are located in the north-west of the North Western Frontier Province (NWFP). At the time of project identification, the seven tehsils in the project area contained about 700 000 persons which equalled 60 per cent of the total district population. The overall goal of the project is to reduce poverty in the project area by increasing farm and off-farm incomes. The project’s objectives as mentioned in the Appraisal Report are five-fold. It aims to (i) boost agricultural production and incomes of populations living in poverty; (ii) establish and strengthen community organisations as the institutions through which technical and social services can be provided to IFAD’s target group on a sustainable basis; (iii) improve the status of women by targeting them for special attention in a culturally acceptable manner, including the provision of support to income generating activities; (iv) improve market access hence marketing to/from rural communities through the improvement of provincial and feeder roads; and (v) increase opportunities for on-farm and off-farm employment in the project area.

The project loan was approved on 11 September 1996 and signed on 21 November 1996. It became effective on 15 April 1997.

However, project implementation started in 1999 only because staff recruitment proved to be difficult in a relatively remote area. Total project costs were estimated at US$25.37 million. The project is financed by an IFAD loan of US$16.5 million which covers about 65 per cent of costs including contingencies. The Government of Pakistan’s (GOP) contribution amounts to US$5.95 million while the share of the beneficiaries amount to US$2.55 million or 10 per cent of total project costs. By January 2007, 79.9 per cent of the loan had been disbursed. The DASP completion has been extended twice. While the original loan closing was 30 September 2004, it has been initially extended to 31 December 2006, and eventually to 31 December 2008. The DASP is not an exception in this regard as all projects in the Pakistan portfolio have been extended.

Project performance

Design features. When the DASP was formulated, IFAD had a written country strategy for Pakistan. When comparing the design of the DASP with the then valid strategy, it appears that both match to a considerable extent. This assessment does not fundamentally change when the DASP design is revisited ex-post with regards to the more recent COSOP of 2003. Painstaking surveys and research in a hitherto unknown environment devoid of comprehensive statistics had to be undertaken in order to culminate in the Formulation and Appraisal Reports, which provide evidence of this effort. However, the logical framework is not completely consistent per se and partly contradictory to the goal and objectives stated in the text, the indicators defined are somewhat arbitrary and not entirely relevant, and many assumptions refer to internal factors, under the control of the project, rather than to genuine external factors. Another design shortcoming of the DASP is that it failed anticipating foreseeable implementation weaknesses, and that it therefore did not put enough emphasis on adequate implementation support.

The DASP made a design shift in the short period between project formulation and the Appraisal Document (March and October 1996, respectively). While in the Formulation Report the approach strategy was based on an appropriate NGO for the selection and the support of village and women’ organisations (VOs and WOs, respectively), in the Appraisal Report these key functions were foreseen to be implemented via the Project Management Unit (PMU). This shift is noteworthy because the Operational Strategy Committee (OSC) of IFAD placed explicit emphasis on arrangements for NGO participation, in particular on the respective roles and mechanisms of collaboration between the NGO, the planned village organisations and the traditional village decision making systems. In hindsight, the shift to direct PMU implementation proved to be beneficial in view of the partly resentful anti-NGO campaign that emerged around the year 2002 and is still pervasively felt today.

Community and women’s development. The expected outputs of the Community and Women’s Development component are: (i) to establish VOs and WOs, (ii) to organize interventions of line departments such as extension, training, irrigation improvement etc. so that the VOs and WOs are the primary means of achieving community development in general, and women’s development in particular; and (iii) to provide for the sustainability of the VOs and WOs after project completion. As of 30 June 2007 - one year to project completion on 30 June 2008 – the DASP has formed a total of 700 VOs and 265 WOs, against original targets of 400 and 100, respectively. The force behind the Community Based Organisations (CBOs) formed is not only their individual membership (20 084 men in VOs, and 4 821 women in WOs), but potentially also their accumulated savings of VOs of PKR 10.02 m, with an average of PKR 500 per VO member, and savings of WOs amounting to PKR 1.38 m or an average close to PKR 300 per WO member.

Such savings are presently only a potential for meaningful development activities as the amount per capita is very small, especially in WOs. As it appears, the rapid and successful build-up of VOs and WOs prevented the project from focusing on their consolidation. They are still fragile constructs and require further systematic support to enable them to function as launching pads for continued development progress in their communities.

Agricultural development. This component is the one with the highest number of line agencies involved. It includes three sub-components: (i) crops development encompassing adaptive research, demonstrations and extension in topics such as food grains, fruits, vegetables, soil and water conservation and social forestry, (ii) livestock development consisting of artificial insemination facilities and services, and training and extension in animal health and husbandry, and (iii) irrigation development focusing on the upgrading and rehabilitation of small-scale irrigation schemes.

In crop development, all training modules implemented by the line departments, in favour of male and female Community Extension Workers, seed producers and input dealers have overshot the targets, benefiting some 4 600 persons. Demonstration plots were laid out on close to 500 ha, and food grain seed was multiplied by farmer from basic high quality stock, that is now sufficient for covering 70-100 per cent of the cereal area of Lower and Upper Dir. On the negative side, the adaptive research trials conducted under the DASP were made in isolation from the Department of Agricultural Extension (DOAE), resulting in a dead end before reaching extension maturity. Another serious shortcoming is that the DOAE did not keep systematic records of demonstration plots and technology adoption patterns. These are substantial flaws in a modern extension set-up. On the other hand, soil and water conservation and social forestry outputs fare comparatively better. The various soil and water conservation measures are carried out with a cost sharing of 50 per cent by the community organisations and have so far claimed 460 ha of new rainfed area for 2 500 households. In social forestry, it is noteworthy that 250 female nursery operators have produced 900 000 saplings serving more than 3 000 ha of newly afforested area.

Livestock development activities have generated to date 6 000 improved female cattle and buffalo offspring through artificial insemination, albeit at conception rates that are still far from being satisfactory (30-55 per cent). Some 900 Community Livestock Workers – of which close to 400 are female – have been trained. They are now offering their services in their communities on a fee basis. Hundreds of vaccination and de-worming campaigns, as well as animal health and feeding demonstrations were carried out. However, and not unlike to agricultural extension, no systematic records on demonstration details, inputs, outputs and final adoptions are available.

The irrigation sub-component has helped rehabilitating or upgrading 5 500 ha, as per target, and resulted in 750 ha newly claimed land. Cropping intensities went up by 100 per cent on average, in comparison with the pre-project situation. In total, more than 6 000 households are beneficiaries of the irrigation sub-component, meaning that each household has on average increased its irrigated command area by 0.125 ha. This is significant for smallholders that own less than one ha of total arable land.

Provincial and feeder roads. The project initially provided for 45 km of provincial roads. It was revised in June 2004 to include improvement and black topping of 100 km of roads in Lower Dir area and the construction of 60 km of shingle roads in Upper Dir. Provincial roads were reduced from 45 to 41 km and completed in 2002. This road section shows signs of massive damage. There is on-site evidence that inappropriate design of box culverts and deficient protection and drainage structures, especially in slide prone areas, are the cause of these problems. The files at the Works and Services Department are devoid of most state-of-the-art technical documents that are normally required to build roads. It appears that there was a blind replication of some standard designs without giving consideration to local conditions. In the feeder road sub-component, contractor services are provided by the VOs, which are awarded the work based on average tender rates. The land for the roads is provided by the communities as their contribution. Implementation quality compares well with similar roads constructed by contractors. It is likely that the 300 km of planned feeder roads will be completed by June 2008.

Employment generation. The employment generation component of DASP includes three sub-components, i.e. (i) micro-enterprise promotion, (ii) promotion of income generating activities and (iii) technical training and apprenticeships of unemployed youth. The micro-enterprise promotion targeted either prospective or existing micro-enterprises and focused on enterprise development, business and technical skills related training modules, covering a total of 875 persons or 98 per cent of the target. The promotion of income generating activities differentiates between male and female clients. In terms of persons trained, 61 per cent is female, benefiting in total 2 000 persons. The technical training and apprenticeship sub-component is the one with the most quantitative client outreach. Over 3 600 persons underwent formal training with durations between 140 to 520 days. Close to one third of trainees are women, and a particularly visible highlight is the female medical technicians training involving 125 persons. Thus, this component has generated highly tangible outputs by responding to real demand for new employment opportunities.

Relevance, effectiveness and efficiency. Regarding relevance and considering the pace of VO and WO formation, the effective commitment and multiple activities carried out, it is obvious that the DASP has found visible and vivid response to the array of changes proposed to the CBOs, which can be perceived as a good proxy for relevance. The component mix has drawn on experience from other projects and corresponds to established strategies. It addresses causes of rural poverty and is consistent with the COSOP and the Poverty Reduction Strategy Paper (PRSP) of the Government of Pakistan. Regarding effectiveness, it is fair to conclude that the key project outcomes have been attained on the whole. Agricultural and livestock production have indeed been boosted, and the DASP has been very effective in employment generation. Income increase, the only indicator related to improved status of women in the logical framework, may have reached 1 500 women at least. Market access due to the construction of provincial and feeder roads has been improved undoubtedly for more than 50 000 households.

In terms of efficiency, project outputs have been generated at comparatively low cost against the achievement of physical targets, which may come close to 100 per cent or above in most of the components by June 2008. In foreign currency equivalents, and strictly related to quantitative outputs versus costs, the project has achieved more with the same funds over a project life extended by close to 60 per cent. Good efficiency (and transparency) indicators are the costs calculated for training day and person by the Employment Generation Unit of PMU, which should become good practice for all training activities of the project.

On the other hand, flawed monitoring and impact assessment methodologies make it impossible to verify, in precise terms, whether the internal economic rate of return (IERR), calculated at appraisal, has been achieved.

Performance of IFAD and its partners. The performance rating conducted for IFAD, GONWFP, UNOPS as the cooperating institution, the GONWFP staff of PMU and the participating village organisations reveals that all project partners have performed at less than a satisfactory rate, with the exception of the PMU staff and the target population, which fare comparatively - and slightly - better. The main problem related to performance appears to be the lack of attention to implementation support, which would have attenuated deficient construction and methodological quality issues. With a project cost outlay of US$25 m, more focused assistance would have been a good investment in terms of quality control and durability of project outcomes and impacts.

Impacts

Targeting and outreach. The project has followed the targeting criteria proposed by the Appraisal Report, and these criteria were also applied before reaching a selection decision. The mission’s visits to both a dozen VOs and WOs suggest that not all households are poor according to the landholding threshold of one ha or less of the Appraisal Report. When considering that the number of farm households with land holdings under one ha has practically doubled in the project area between 1990 and 2005, mostly by hereditary partition, there is a natural propensity of having progressively more landholdings that fit into the poverty definition of the DASP. Due to this fact, and also acknowledging the project’s commonsense and drive for socially acceptable inclusiveness, the poor have been reached in an effective way with the project reaching out to 24 800 VO and WO households against a target of 14 500.

Rural poverty reduction impacts. Taking together the nine defined impact domains of rural poverty reduction, the mission concludes that the project indeed is contributing to alleviate rural poverty in practically all impact domains that were relevant to the project as per design and component mix chosen. Although remittances from abroad appear to remain important contributors to household income, the outcomes and impacts of the DASP are significant. Many of the observed poverty reduction impacts can be directly attributed to the project. Two villages that chose not to collaborate with the DASP – despite being eligible on the basis of their respective poverty profiles – are today devoid of basis infrastructure and not served by line agencies.

Another observation angle related to project impact was offered by drawings prepared by VOs and WOs, depicting their respective villages at three different points in time: (i) before the start of the DASP, (ii) at present and (iii) in some years from now. The collected drawings represent perceived change and future aspirations in a pattern fairly congruent with dimensions of poverty in general, and with the project approach and component mix in particular.

Sustainability and ownership. The principal driver of sustainability is the strong sense of ownership that becomes evident when analysing the performance of VOs and WOs. In their own words, there is no going back to the time before the DASP when their communities were unorganised and left alone. The fact that most interventions require own contributions in cash and kind, ranging from 20 per cent to 50 per cent, is another factor of sustainability. The sense of ownership is particularly strong among WO members. The 4 800 organized women at present achieved this impressive result with many personal sacrifices. However, the VOs are still far from a status of maturity that would allow them to go ahead unaccompanied. Especially in terms of management of their financial assets and of credit allocation, both the VOs and WOs have to make more progress. Non-farm employment created, thanks to the DASP, has a good perspective of sustainability. The rapid growth of the clusters, including hundreds of enterprises and thousands of jobs generated, is a driver of its own for further development, which does not critically depend on an intensive coaching after completion of the DASP. Regarding on-farm employment and its potential of generating corresponding income, much will depend on the future intensity of support by line agencies. The mission observed at various instances that the sustainability of the roads constructed under the project, particularly provincial roads, are a cause for concern. Apart from deficient quality at completion, maintenance is presently not assured.

Innovation, replicability and scaling-up. The skilful adaptation of the CBO-based approach strategy to a difficult context can be considered as innovative. However, an issue of concern is the perceived lack of consideration for intangibles. One typical example is the apparently successful implementation of 3 000 demonstration plots and events for crop and livestock development. These demonstrations would have been an outstanding opportunity to generate and manage knowledge if only the responsible extension agents had recorded the basic parameters of the demonstration plot and events, such as locations, dates, description of farmer and improved technologies, and comparative yields obtained. There are virtually no records available from the concerned line department in this respect. The roads component also could have been a source of good practice for design and construction in difficult mountain environments. With the generalised absence of design data and environmental impact consideration, there is no opportunity to replicate and up-scale the art of road construction and upgrading in favour of similar projects.

Conclusions and recommendations

Conclusions. Table 1 below provides an overview of DASP’s performance. The ratings used  for this purposes are based on the following six-point scale: 6 = highly satisfactory; 5 = satisfactory; 4 = moderately satisfactory; 3 = moderately unsatisfactory; 2 = unsatisfactory; 1 = highly unsatisfactory. The project fares relatively well with regards to relevance, effectiveness, efficiency and rural poverty reduction impacts while the performance of partners, sustainability, innovation, replicability and scaling-up are moderately unsatisfactory. Hence, the project is rated as moderately satisfactory despite a difficult context, but it failed to address foreseeable and basically correctable flaws.

Rating summary

Evaluation Criteria	Ratings
Relevance	4
Effectiveness	5
Efficiency	4
Project Performance */	4.33
Impact	5
Sustainability	3
Innovation, Replication and Scaling-Up	3
Overall Project Achievement **/	4
Performance of IFAD	3
Performance of CI	3
Performance of Government	3

Source: IFAD evaluation mission 2007
*/    Calculated as the average of relevance, effectiveness and efficiency.
**/  Determined based on the ratings for relevance, effectiveness, efficiency,
impact, sustainability, innovation, replication and up-scaling.

Outreach can be considered satisfactory due to the higher than anticipated number of participating households, i.e. 14 500 at appraisal versus 20 000 and 4 800 VO and WO members to date. The distribution of benefits is widespread, because of the multiple interventions that affect many households that are not necessarily VO and WO members, such as market access, social forestry and watershed management, soil and water conservation and the various irrigation schemes. Gender impact may seem modest by general standards, but considering the still prevailing conservative outlook of Pukthun society, the impact achieved is respectable. In general terms, the DASP has contributed in attaining IFAD’s strategic objectives, notwithstanding some reservations made mainly by EVEREST

Summing up the conclusions, the completion evaluation of the DASP teaches the following lessons:

• It is possible to design and implement multi-sectoral and multi-stakeholder area support projects in difficult environments and that such projects can be effective, like the DASP.
• However, the complexity of the design and the implementation modalities, including monitoring and evaluation, were not met by adequate technical and methodological support by IFAD, the CI and GONWFP. Many implementation and quality flaws could have been avoided by more awareness and pro-active interventions in this respect. The major lesson learnt is that conceptually ambitious and complex undertakings cannot be left to business as usual; they require a form of focused coaching.
• The fundamental working hypothesis that development must be owned by the people who are affected by change has been confirmed. The overwhelming response of the rural population to organisational forms that were not part of their traditional life provides evidence of the correctness of this working hypothesis. In only ten years, the Jirgas have been transformed from rather exclusive councils of village elders into rejuvenated and broad-based VOs. At the same time, women were allowed to step out of their invisibility and to form WOs. Both are remarkable achievements in a conservative rural society.
• Consequently, VOs and WOs have become launching pads for multiple development activities, which would not have been possible without their existence.
• Due to the relative void of intangibles generated by the project, policy dialogue was almost impossible. IFAD should have put more effort in policy dialogue, which however requires better knowledge management and documentation of good practice in the first place.
• For IFAD, this means that its ambition of being a centre of excellence for rural poverty alleviation may not have been lived up to sufficiently in relation with the DASP. The fact that IFAD is hardly perceived as the leading steward of rural development and poverty alleviation in Pakistan may point to a deficiency going beyond the DASP.

Recommendations. The first set of recommendations refers to the remaining project life, and is addressed to GONWFP and the PMU. The main purpose is to assure an adequate consolidation of all components and address correctable weaknesses, such as establishing a sound monitoring basis for project completion.

The second set of recommendations aims at facilitating an adequate coaching of the project by future supervision and project completion missions. In this framework, the mission recommends, in a third set of recommendations, that IFAD and GONWFP engage in consultations on the post-project threshold levels of financial inputs into the two districts and measures that improve line department performance.

Finally, the mission recommends to IFAD and GOP to explore new project completion modalities for complex and multi-stakeholder projects in order to mitigate donor withdrawal symptoms. In this context, the set-up of a Phasing-out Facility is proposed.