Interim Evaluation Report

Background

The project area consists of two quite diverse districts, particularly in terms of access, population pressure and ethnic groupings. The districts lie south of the provincial capital Kundiawa to which only the Gumine District is linked by road. Gumine is characterized in some area by population pressures, increasing land degradation, a deficit (at times) in food production. Karimui is generally at a lower altitude, climatically warmer (and thereby liable to malaria), underpopulated and accessible only by air or foot. Both districts are characterized by limited income opportunities and inadequate health, agriculture and education services.

There are some 36 700 people living in Gumine District and 12 100 people in Karimui District. Most are rural villagers who are dependent on subsistence agriculture but the majority have some modest involvement in cash cropping. Sweet potato is the main staple food in both districts and is the most important crop grown by villagers, while in Karimui sago is also of importance. Land use intensity in Gumine District varies from low to high. Land use in Karimui District is much less intensive than in Gumine District and soil fertility, in the latter district, is still maintained by forest fallows of long duration (10 to 20 years). Coffee is the main cash crop in both districts with the majority of the population receiving some benefit from this crop while cardamom is an important cash crop in the Karimui District.

Project desing and objectives

Target group

According to the Staff Appraisal Report (SAR) about 5 680 households (52% of the total households) would benefit through participating in increased and better cash cropping programmes (as a result of improved extension efforts), while an estimated 2 400 households would, by adopting an improved crop rotation model, improve their nutritional status. The road upgrading programme would provide better year-round access to facilities to the 37 000 people in the Gumine district while 960 adults and pre-school children would benefit from library education in Karimui. An estimated 1 400 households would benefit through upgrading of roads to all-weather standard and through project promoted commercial vegetable production.

Institutions would benefit through an upgrading of their capacity to plan and implement development projects both at the provincial and district levels.

Objectives and components

The project objectives were to increase the incomes and employment of the population; improve their health and nutritional status; promote the role of women within society; and strengthen the institutional framework for provincial planning and execution of district-based investments in rural development. The basic assumption was that the target population will improve its living conditions and income if it has access to essential social and development services. More equal distribution of economic benefits and of government services among different areas was at the core of project justification.

The project would seek to achieve its objectives through investments in agricultural activities, health and nutrition services, education and training, rural infrastructure, management support and media information services. The project would concentrate on improvement and expansion of existing activities in recognition of the limited implementation capacity of the Government of Papua New Guinea (GOPNG) agencies and local organizations. Project activities were to be phased over a six-year period to ensure the workload was compatible with implementation capacity. The project was to include the following components:

Agricultural Development

• Strengthening agricultural extension services
• Commercial vegetable production and marketing

Nutrition and Health Development

• Improvement of nutrition and health services
• Water supply to Karimui Station

Rural Infrastructure Improvement

• Road improvement and rehabilitation
• Footbridge construction
• Airstrip improvement at Karimui

Education and Training

• Literacy programmes
• Promotion of education for women
• Construction of a staff development and training centre
• Agricultural training
• Health training

Management Support and Medical Services

• Planning and management support
• Medical information service unit
• Monitoring and evaluation

Taking into account past experience with integrated Rural Development Projects in PNG, it was decided that this project will be implemented through Government structures, i.e., by the Department of Simbu using the South Simbu Management Team (SSMT). These arrangements were meant to ensure the sustainability of project achievements after its completion.

Expected Results and Assumptions

The SAR indicated that seed multiplication gardens, coffee nurseries, cardamon nurseries, pyrethrum nurseries, cardamom driers, demonstration gardens and a poultry distribution centre were to be established. In Gumine about 1 400 households were to cultivate vegetables for sale over an area of about 42 ha.

Incremental production, as a result of project interventions, was estimated as follows:

• the total expected annual incremental production of food crops would be 600 mt by Year 5 and 1 760 mt by Year 10. Commercial vegetable production in Gumine is expected to reach 250 mt p.a. by year 5. Incremental livestock production derived from the project would amount to 55 000 chicken eggs, 3 000 duck eggs, and 1 200 kg of duck meat p.a. within five years; and
• at full production it is expected that incremental coffee production would be about 135 mt p.a. of parchment coffee. The 90 ha of cardamom planted would produce about 32 mt p.a. of dried cardamom.

Infrastructure building, particularly through upgrading and extending the facilities of the Division of Primary Industry (DPI), was a major activity.

Evaluation

Starting from the findings of a Mid-term Review (MTR, 1989), the aim of the Interim Evaluation was to make a further assessment of the performance, efficiency and relevance of the project in the context of the objectives stated in the SAR. In so doing, the mission's intention was to draw lessons from the project's experience and make recommendations relevant to future IFAD operations in the northern districts of the Simbu Province with particular reference to the replicability of the model developed in South Simbu.

As the project has been reviewed regularly, a substantial amount of information was available particularly on the physical implementation of its infrastructural components. The mission's approach was to review this information in order to rapidly assess the overall status of the project and then to devote maximum time to field investigations and to focus on key issues. A very candid internal evaluation of the project presented to the mission on its arrival at Kundiawa was of great assistance in assessing the project's status.

While the project's activities and outputs had been monitored regularly, little information on the effects and impact of the project was available. In order to gain information of effect and impact, the mission developed interview questionnaires for both farmers and project staff. Informal group interviews were also conducted, particularly with women groups. Twenty farmers, most of whom were living relatively close to rural extension centres were interviewed. While the sample was obviously insufficient for any quantitative analysis, the interviews allowed the mission to check assumptions and statements particularly regarding the relationship between the project and its general environment.

The mission also conducted 15 formal interviews with project staff involved in field activities (these represent 53% of the total agricultural field staff). Individual staff members were generally keen to participate as they appeared to view the exercise as an opportunity to fully express their views. Extensive informal interviews were held with senior district and provincial staff as well. The mission believes that those interviews provided a fairly accurate picture of the project (and of the agricultural production component in particular).

While the mission would have liked to develop a more substantial analysis of the project's impact it has concentrated on the people's perception of the relevance of the project and the efficiency of its delivery mechanisms. The MTR (1989) recognised that the lack of reliable data on beneficiaries as well as on their productivity and incomes made it difficult to provide meaningful information in this respect. Project design and implementation methods are also emphasized as they have been seen as possible models for similar developments elsewhere.

Project implementation context

The major investment expenditures have now been completed generally within the original estimates and approximately USD 3 million (IFAD and AIDAB Funds combined) are still available for future project expenditure. This is considered sufficient to meet all future funding requirements. Expenditure has not been constrained by any lack of project funds as such but by government budget allocations. In 1991, for instance, the project's request for Kina (K) 919 000 was reduced to K 500 000 under the governments budgetary policies.

Uneven performance characterised project activities. However, the overall performance of the SSRDP must be viewed in the context of difficulties imposed by government budget cuts (as the bulk of the project's funding was incorporated in the national budgets) in 1990 and 1991 and the re-organisation of the Department of Simbu which had substantial effects on project staffing. These occurrences curtailed some project activities substantially.

Project achievements

In relation to its stated objectives the project's main measurable achievement has been the strengthening of the provincial administrative structure and an improvement in the geographical distribution of government services. For a number of reasons (the effects of the de-centralisation process on staff, the relative shortages of funds due to budget cuts, less farmer involvement with a decrease in the free supply of inputs) the project has lost some of its initial momentum. The main conclusions by component are summarised in the following:

Infrastructure. With the exception of the water supply subcomponent (not yet implemented), the project has been very successful in fulfilling the SAR's infrastructure building objectives. The 26 km of road has been upgraded, eight footbridges built and the Karimui airstrip improved. As well, in virtually all cases, facilities for agricultural extension and health staff (including the upgrading or establishment of DPI base camps, kit houses for staff, marketing centres, nurseries and seed multiplication centres) were upgraded or built in accordance with SAR specifications.

Staffing. In May 1991, only 16 of the 23 and 9 of the 16 positions proposed for agriculture and health, respectively were filled. This situation is unsatisfactory but, again, must be seen in the context of budgetary restrictions and upheavals bought about by the decentralisation process. Until the departure of the Staff Development and Training Advisor (SDTA) staff training had been carried out well, while the Wara Simbu Training Centre (WSTC) which had been upgraded by the project had stimulated staff training as a whole in the province.

Agricultural Development. Although some of the data in the various project reports must be treated with caution the SAR target (which seems quite modest) for planting and rehabilitating coffee and cardamon appear to have been achieved. While some production increases attributable to rehabilitation could have been expected by now, most of the plantings are too young, at this stage, to have substantially affected production.

Efforts to bolster food crop output and improve nutrition by promoting new crop rotations have been unsuccessful largely because both farmers and extensionists appear unconvinced both by the need for this activity and the appropriateness of the technology offered.

Advisory services provided by DPI have generally not reached a large part of the target population. Some antagonism by farmers towards DPI exists and a change in attitude by many of the staff, together with intensive training in extension methodology, is needed if extension is to function effectively.

Health and Nutrition. The delivery of health services to the target population is largely related to the timely supply of medicines to Health Centres and Aid Posts, the inaccessibility of many areas and the presence of sufficient staff. Several "key indicators" supplied to the mission failed to show an appreciable increase in services delivered presumably because one of several factors affecting delivery of services have often been lacking.

Education and Training. According to the International Evaluation Report (May 1991) prepared by the project management team, there are 40 literacy schools established in the project area (which has only 6% of the province's population) against only 14 in the other districts of the Province. Over 1 000 children and 760 adults have attended literacy schools in the Karimui district and although this activity was not prescribed for the Gumine district the SSRDP has supported five schools in that district as well. With a limited project input targets have been well exceeded basically because the population sees the need for literacy education and because management has been effective.

The provision of training infrastructure together with the appointment to a three year period of a SDTA has allowed over 60 courses in a variety of fields to be carried out. Over 1 500 people (including more than 400 women) participated. Agricultural training has lost some momentum since the departure of the SDTA and the unfortunate transfer of this understudy.

Management Support. Long-term technical assistance proved to be crucial in the successful implementation of workable framework for project programming and budgeting. The consistency of the approach from project design to implementation has been remarkable thanks to the continued involvement of a small number of national and expatriate staff who were able to establish an effective team spirit and working relationship. However, the departure of the expert programme planner and a major reshuffling of staff due to decentralisation has decreased the planning capacity of the administrative structure significantly.

Project Effects - Immediate and Lasting

Monitoring and Evaluation. The project's monitoring system was closely tied to the planning and budgeting process, at all levels. While it contributed to the establishment of improved planning and implementation review procedures, it was generally not used for decision-making. Evaluation was limited to an MTR in 1989, a nutritional survey, finalised in 1989, which allowed a comparison with a similar survey conducted in 1980 and a ‘baseline survey' which was still in the process of finalisation at the time of the mission. While the survey contains some interesting information of cropping practices and land use, it is unlikely to be of any great help in assessing project impact in the future. Partly because the SAR's working definition of evaluation activities differed substantially from evaluation as a periodic, critical assessment of project relevance, performance and impact in the context of its stated objectives, there is little objective data on performance and impact.

Effects on Target Group. The conclusion of the MTR (1989) that awareness of the project activities among men (or at least those in contact with SSRDP) is widespread is confirmed. However, women are less aware of the project's extension work and its possible benefits. The project supported health facilities are extensively used while the literacy programmes have been received with enthusiasm. However, the mission's impression was that interest and participation in cash cropping activities were waning (if not receding) due to the following main reasons.

• The notion of the project and what can be reasonably expected from it are not yet fully understood. Therefore once free handouts are stopped, contact with extension staff and advice from project are not perceived as vital "inputs" for production. Free handouts had set a negative precedent that the project will have to try to undo.
• A communication gap exists between some staff and population. This is due to ethnic and cultural barriers and to the relative youthfulness of the staff. This is compounded by the fact that sometimes public servants do not stay enough in one area to become familiar with the people and their customs.
• Despite the recommendations of the MTR (1989), groups and in particular women's groups are neither encouraged nor assisted by the project. Their number is decreasing and their potential as extension foci is not being used.

Sufficient extension visits to farms are not being carried out by staff. This causes a negative perception of staff and the project by farmers and contributes to poor communication. While some extensionists state that visits to farms and houses are hampered by road conditions and unavailability of vehicles, and insufficient direction and encouragement from headquarters, there is also a psychological reluctance on the part of the staff vis-à-vis "on field extension and visits".

There is a strong indication that some of the farmers in contact with the project (vis. the procurement of free inputs) are in fact "new style big-men" who position themselves in such a way as to acquire project inputs. If this is so, it makes for difficulties in some activities of the project effectively reaching the target population under its current design and management.

Thus, in terms of achieving a widespread and mean coverage with regard to agricultural advice and providing access to inputs, the project has not achieved its objectives.

The project has improved physical access to some areas, provided a better basis for nutrition and health services to operate and assisted in improving literacy in some remote areas. While the project has provided health staff with better living conditions there is little evidence of an increased delivery of health services to the population. The effectiveness of health services is closely linked to staff numbers, availability of medicines and funding; one or several of these essential ingredients has often been lacking.

Largely because the approach to promoting the role of women was indirect and non-specific (and ignored the social context) the situation of women has generally not improved. Some specific benefits through literacy schools has occurred especially in the Karimui district.

Effect on Crop Production. By distributing coffee and cardamon planting material and promoting the rehabilitation of existing coffee stands the project has certainly had some effect on production /, however, the extent of the effect cannot be measured with some representative samplings of these plantings. The commercial vegetable production by the project is unsustainable in its present form. The envisaged increases in food production and additives to the species of crops grown were unrealistic in the context of the situation in Southern Simbu.

Effect on Institutions. Starting from a low level of institutional capacity, the project has succeeded in developing a workable framework for district-based investment programming and budgeting. While the projects' physical facilities and infrastructure were implemented, further attention (including the use of technical assistance) will be needed if this achievement is to be sustained.

The achievements of institutional strengthening have been overshadowed by weaknesses in managerial skills which resulted in better planning not being adequately reflected in better operation of facilities and delivery mechanisms.

Recommendations

Recommendations Consistent with the Present Project Structure and Rationale

Institutional Strengthening. To improve the manner in which project management is implemented and to strengthen the province's institutions, the following recommendations are made.

• in addition to further staff training appropriate personnel management methods providing for career development plans which recognise performance and minimise both external influences and subjective considerations should be promoted;
• revolving funds and/or accounts should be established to facilitate the marketing of cash crops, the payment of patrol allowances and funds and the provision of finance for selected non-recurrent expenditure items;
• to overcome "decentralisation shock" and help the project regain momentum, short-term technical assistance should be used as necessary;
• a project liaison officer should be appointed to support the District Managers and their District Management Teams; and
• to evaluate project impact more project-specific surveys should be undertaken. Additionally, as project output data are incomplete, quarterly field reports (in particular) should be re-checked and output data revised.

Agriculture Development. Extension should undertake the following general actions now that its primary role of providing agricultural inputs is being phased out:

• re-examine its current thrust as far as policies are concerned and define them more clearly. It is anticipated that an approach entailing more involvement by extensionists at the village level would be adopted;
• develop and prioritise a structured training programme; and
• rationalise the staffing situation so that frequent staff transfers are minimised.

To increase the income of farmers the project should consider the following possibilities:

• devote more resources to rehabilitating coffee and increasing yield per unit area rather than planting new areas, particularly in the Gumine district;
• seek private operators involvement in agricultural inputs supply; and
• examine the feasibility of setting up a revolving fund, initially financed by IFAD, by which DPI or its agents can market cash crops in remote areas.

The proposed feasibility study on commercial vegetable production should be carried out as a priority. If it becomes apparent that the venture cannot be made commercially viable, the project should withdraw support immediately rather than further raise false expectations.

Health and Nutrition. The project should urge provincial authorities to recruit medical staff (including a volunteer nutrition worker for Karimui), provide better transport for health workers and address the issues of both shortages of vaccines and funds for Maternal and Child Health (MCH) clinics.

Training and Education. The following measures should be considered under the training and education component:

• more consideration should be given to self-funding for literacy programmes, including the possibility of the project establishing a crop marketing service to be run by the organisers of the Karimui programme;
• actual staff training needs should be developed in detail (in the case of DPI after a re-examination of its priorities) through technical assistance. Special assistance should be given to the training needs of women; and
• in order for the Media Unit to become more effective in serving community needs, it should be closely integrated with the Simbu Extension Support Programme (SESP).

Recommendations for Consideration in the Future

The assumption should not systematically be made that project implementation is to be carried out through government agencies (as was the case with SSRDP). The selection of implementation channels or delivery mechanisms should be based on an assessment of the comparative strengths and weaknesses of the various possible implementors with respect to each major action considered by the project. The end result of such an approach would most likely be an adjustment of project objectives to existing capabilities and the involvement of a larger number of operators. Consequently, institution capacity building should be proportionate to the contribution that public services can realistically be expected to make to the project objectives.

Because churches and some non-government organisations (NGOs) are will entrenched in many PNG communities they should, where feasible, be used as a link between the project and the population. This should lead to more involvement by the target groups as most non-government agencies have a better awareness of the population's specific needs and capabilities than public servants do.

Technical assistance should be used as necessary and it should not be terminated abruptly as was the case with SSRDP.

Extreme care should be taken before new technologies are included in projects. Even if this is the case, they should first be demonstrated then phased in.

In the PNG context, renewed attention should be given to the involvement of women and women's groups in project planning. The mission believes that women's groups and the issue of women's involvement was not considered seriously enough by the project implementors.

Lessons learned

The assumption that the public service, if given sufficient support through project instigated institutional building, would be able to deliver a wide range of services should not be made. For services which involve complicated interactions between the project and the target group, NGOs which are more intimately involved with the communities, should be used as one channel for project implementation. This approach should be given sufficient weight in project design particularly in situations (such as PNG) where churches and NGOs are well entrenched.

The introduction of new technologies without proper testing and without a thorough understanding of the target groups social and economic needs should never be contemplated. A realistic need assessment should be mandatory in project design and new technologies should be carefully tested and slowly introduced.

Project designers should not be reluctant to use and persevere with TA where a need is identified. TA should not be abruptly terminated particularly if adequate local expertise has not been developed. Project design should have sufficient flexibility to increase (or decrease) the amount of TA provided according to implementation experience.

Institution strengthening, particularly training, should not be treated as a one-off activity, but rather as a continuous intervention which is capable of responding to the project's changing needs. For training to be effective, be it at senior, field or farmer levels, training needs should be developed in detail and prioritised. TA should be used as necessary for training.

Project design and implementation should give sufficient weight to the issue of women's involvement both as beneficiaries and implementers of activities. In situations were women are, effectively, the most disadvantaged in both material and social senses (such as PNG) adequate attention to women's involvement in project design and implementation should be a condition for loan effectiveness and continued disbursement of project funds. Without such a proviso it is probable that the issue of women's involvement will not be adequately considered by project implementors.