Indonesia Country Programme Evaluation (2004) - IOE
Indonesia Country Programme Evaluation (2004)
Country programme evaluation process1
The Indonesia Country Programme Evaluation (CPE) was undertaken by the Office of Evaluation (OE) at the request of the Government of Indonesia to take stock of past experiences and contribute to the future directions of IFAD's strategy in Indonesia. The main objective of the CPE is to assess the results and impact of IFAD's (lending and non-lending) operations and provide building blocks for a new Indonesia Country Strategic Opportunities Paper (COSOP).
The CPE took place in 2003 and followed the general provisions contained in IFAD's Evaluation Policy. The evaluation uses three conventional information sources that allow evidence and results to be triangulated. The sources of evidence were: (i) a desk review of relevant documents; (ii) self-assessments by the management of four IFAD-supported projects and four corresponding beneficiary self-assessments facilitated by the University of Gaja Mada; and (iii) detailed field investigations and interactions with various stakeholders including visits to eight of the 12 projects in nine provinces and 21 districts where discussions were held, inter alia, with 95 project groups in 49 villages and with provincial, district and subdistrict level staff in the line agences concerned.
The evaluation includes assessments of all IFAD-financed projects in Indonesia using OE's Methodological Framework for Project Evaluation as well as the internationally established evaluation criteria of relevance, effectiveness, efficiency, outcome and sustainability. The draft of this report was discussed in a stakeholders' workshop in Indonesia on 14 Janaury 2004, where the Government and other participants expressed their appreciation and satisfaction with the findings and conclusions of the CPE. The Evaluation Committee of IFAD's Execuitve Board discussed the report for the first time on 20 February 2004. A more intensive debate on key issues raised by the evaluation followed at the national roundtable CPE workshop in Yogyakarta on 11-12 March 2004.
Development context
For 25 years prior to the financial crisis of 1997, Indonesia was a development success story. Rapid growth between 1970 and 1996 moved Indonesia from low to middle income status, social indicators improved and the incidence of poverty fell from 60% in 1970 to 11% in 1996. With the economic crisis in 1997, unemployment and poverty rose sharply, the latter reaching 27% nationally in 1999 and 51% in Eastern Indonesia. However, marked urban-rural disparities continue to exist with inherent differences in agricultural potential: rainfall and soil quality declines markedly from west to east and some remote eastern islands have very poor communications with islands to their west. Agriculture drives Indonesia's rural economy and provides over 60% of the country's jobs (two thirds of the popuation is rural). In the sixties, agriculture accounted for over 55% of gross domestic product (GDP) but that share had fallen to 17% by 2000. Agricultural growth rates from 1960 to 2000 averaged 3.7% per year. However, problems survive from before the financial crisis of 1996-98 including an unfavourable policy environment, slow technological progress, a lack of agribusiness development and deteriorating infrastructure.
In January 2001, central government powers and responsibilities were devolved to local government so as to bring government closer to the people, increase efficiency and strengthen accountability. This massive change, not yet fullly completed, has profound implications for all sectors. According to the World Bank, devolution is producing new grass-roots leadership and is bringing the Government closer to the people. But, challenges remain: the division of labour among levels of government remains unclear, the intergovernmental fiscal system is far from equal and mechanisms to bring external financing to the regions are not yet established. The old agricultural order has been turned on its head. District governors now allocate resources as they see fit. One result is that agricultural, financial and human resources are being reassigned to other departments with negative knock-on effects on farming and development projects. For the Ministry of Agriculture, and other departments, the adjustment to the direct loss of control over resources and the shift at the provincial and district levels from an executive to an advisory role is proving difficult.
IFAD strategy in Indonesia
Although IFAD provided its first loan to Indonesia in 1980, and undertook a general identification mission in 1982, it was not until 1988 that IFAD developed its first workable strategy for Indonesia. The strategy, written in the late eighties was heavily descriptive, resource endowement driven and insufficiently analytical. It did however for the first time lay particular emphasis on the rural poor and women. Given that Indonesia was self-sufficient in rice, the strategy saw the main challenge as raising the living standards of the rural poor by expanding rural employment through commodity and regional diversification. Overall, the strategy was relevant, well argued and an effective guide for IFAD for much of the next decade.
The Fund prepared its next strategy (the COSOP) in 1998. The COSOP focused on the Government's rural poverty reduction strategy, but resisted the temptation to adopt social safety nets, arguing that IFAD should continue to work on long-term solutions to chronic rural poverty. It was also opportunistic, seizing the chance presented by the new political climate to carry forward an agenda of empowering the rural poor. The COSOP included a clear definiton of the target group and appropriately defined the Fund's corresponding geographic and sectoral priorities.
The COSOP stressed the need for intensive policy dialogue with the Indonesian government and other international and national development partners on issues such as indigenous rights, transparency, decentralization, land rights, the role of non-governmental organizations (NGOs) and civil society. It promised to address issues such as inadequate intragovernment coordination and information-sharing, transparency and corruption, better collaboration with NGOs, and the lack of beneficiary input into design. Overall, the COSOP gives clear directions for IFAD that reflect heightened international attention to poverty reduction in the nineties and the view that many past agricultural interventions foundered because they were too technocratic and had not first built strong social and community foundations. However, the COSOP was not as consonant with the Government's central push for rural growth based in higher crop and livestock production and greater value added in the rural economy as it might have been. The CPE acknowledges the importance of social mobilization and building social capital as a key dimension of IFAD's work in Indonesia. However, it argues that the formation of social capital is a necessary but not sufficient condition for successful rural poverty reduction, and that the COSOP would have gained by taking a more balanced approach and by paying more attention to the economic empowerment of the rural poor through agriculture development.
Although the COSOP was developed before both the regional strategy for Asia and the Pacific and IFAD's strategic framework, it is to a large extent consistent with the central elements of both these strategies. However, there are important areas that were not given sufficient attention. For example, as mentioned above, the COSOP does not pay enough attention to activities that would increase agriculture production and productivity through improved access to productive natural resources and technology, even though these are core objectives of the regional strategy and strategic framework. Like the 1988 strategy, the COSOP underplays the development of market linkages, off-farm opportunities and agro-processing. The COSOP is however in tune with the regional strategy and strategic framework in emphasizing IFAD's catalytic impact, especially through policy dialogue and partnership-building.
Moreover, it would have been useful had the COSOP included a more crisply defined hierarchy of objectives, especially for the non-lending activities, and performance indicators to assist in monitoring the implementation of the strategy and ultimately measuring its results and outcomes. The COSOP would also have benefited from a thorough analysis of the inputs, processes and activities required to achieve its multiple objectives and outputs, as well as from a prioritization or time plan for the delivery of its outputs. The evaluation noted that although the Indonesia COSOP included aspects relating to lending and non-lending activities, the document was designed primarily as a vehicle for implementing an IFAD lending programme in the country. On the whole, IFAD should have paid greater attention to the policy and portfolio management aspects of its strategies.
To sum up, IFAD has pursued strategies broadly in line with government ambitions and international thinking of 20 years. It has neither led nor lagged and has remained broadly relevant to Indonesia's needs. The Fund's evolving strategies in Indonesia have found fairly full expression in the poverty, geographic and subsectoral focus of IFAD-supported projects. Specifically, the 1998 COSOP is satisfactory in terms of poverty targeting, emphasis on gender, community development and local institution-building. It is not in line, however, with the Government's push for rural growth based on higher production and the growing number of voices worldwide calling for more attention to be paid to agricultural growth.
Programme performance
Policy dialogue. The CPE found that IFAD has not materially engaged in policy dialogue with the Government of Indonesia, the United Nations, international financial institutions (IFIs), and other development agencies on issues of significance to rural development and poverty reduction in Indonesia. This lack of engagement represents an important missed opportunity for IFAD. It is not making, in an active and direct manner, the case among its peers for a major reduction in Indonesian poverty based on rapid rural and agriculture development. There are various issues that the CPE raises on this topic. For example, it is not clear whether IFAD has the ability to undertake detailed policy analysis that builds on its operational experiences, thus equipping staff with substantive and well-researched issues for dialogue. The Fund's small size, lack of country presence and modest lending programme (compared to other IFIs), may also constrain its ability to make itself heard among the larger players in the country and the Government. In short, IFAD may not have the capacity to gain credibility and ensure the requisite continuity in dialogue, which is fundamental if it is to contribute effectively to policy reform. Equally, changes in the role of cooperating institutions and other major IFAD partners in supporting IFAD policy dialogue efforts with the Government may be useful. For example, the Fund could enhance its strategy for partnership (with for example, cooperating institutions, cofinanciers, research institutions and selected NGOs) to make its policy dialogue more effective. In the context of its current strategy, IFAD's lack of attention to policy analysis and policy dialogue in Indonesia is a cause for concern, and in future should have high priority among the Fund's non-lending activities in the country.
Project effects. All 12 projects in Indonesia had broadly or partly relevant objectives, so taken as a whole the relevance of the portfolio is substantial. The projects were or are consistent with Indonesia's development priorities (at the time of design) and IFAD's country strategy. How far projects have been entirely relevant to the rural poor is harder to determine. In so far as they have been aimed at helping small farmers, the landless, women and the otherwise marginalized, they have clearly been relevant. To the extent that they have covered rainfed areas and other places of low potential or environmental stress, they have been relevant to the goal of reducing poverty and reducing inequity – at least regionally. But, the extent to which they address the needs of the poor, as defined by the poor themselves, is less certain. Only in the Post-Crisis Programme for Participatory Integrated Development in Rainfed Areas (PIDRA) and the East Kalimantan Local Communities Empowerment Programme (EKLCEP) have efforts been made at the outset and during implementation to listen to the people.
In terms of effectiveness, none of the projects were judged highly effective. Four are rated substantially effective in achieving all or most objectives (the Eastern Islands Smallholder Cashew Development Project (EISCDP), Income-Generating Project for Marginal Farmers and Landless (P4K) II, and the two early livestock projects). The cashew project was effective as it achieved its central goal of introducing a new technology among small and marginal farmers. Among projects with modest ratings, the East Java Rainfed Agriculture Project (EJRAP) used conservational technologies rejected by most farmers on grounds of cost and efficacy. The technical projects supported by IFAD were not very effective – the cashew project being a clear exception. This shortcoming is mainly attributable to poor project design. The technical projects did not establish strong links with Indonesia's agricultural research system to ensure that technologies were suitably adapted to where they were being used. The PIDRA and EKLCEP are likely to be substantially effective but are still in the initial stages of implementation. Moreover, the effectiveness of IFAD-supported projects in general has not been sufficiently driven by innovation. Much of the effectiveness of IFAD's projects can be traced to the successful formation of self-help groups as the central organizing device in nearly all operations. Placing the onus on the people themselves has been effective.
It was difficult to assess efficiency in almost all projects given the lack of usable empirical data concerning outcomes. The exception is the EISCDP where the internal economic rate of return was estimated at 16%. However, P4K III and EKLCEP have a good chance of producing substantial positive economic benefits. The internal economic rates of return of the EJRAP and Eastern Islands Smallholder Farming Systems and Livestock Development Project (PUTKATI) are likely to be modest. A less rigorous method of gaining an insight into efficiency is to compare project costs per household across projects, with similar projects funded by other development organizations. The comparative data of more recent projects suggest that IFAD projects are more cost-effective than similar Asian Development Bank (AsDB) projects, about the same as similar World Bank projects, but may not be as cost-effective as the famously successful Aga Khan Rural Support Programme in Pakistan.
Institutional development was not an objective in earlier IFAD operations. It has however been a clear objective in more recent IFAD interventions. There has been noticeable impact on institutional development at the grass-roots level, in particular by promoting the participation of NGOs in project activities and in contributing to the development of around 100 000 self-help groups. The CPE however points to the need to ensure that these groups are able to sustain themselves by creating ways for them to grow into more advanced institutions. Without such opportunities, the impact so far accomplished will be undermined. Little impact has been achieved on institutions at the national level, be it in terms of influencing institutional policies and mind-sets, or their capacities and priorities. Overall, achievements in institutional development are modest, due to limitations in project design and implementation, inadequate monitoring and evaluation (M&E) systems that do not contribute enough to learning and the continuing survival of top-down project management. The lack of institutional impact at the national level in Indonesia is also a direct result of IFAD's lack of engagement with the Government and other partners in policy exchange.
IFAD's early projects paid scant attention to gender issues. Even in the nineties, three out of the five projects launched did not have gender-related goals and thus had negligible impact on women. In the late nineties gender was incorporated successfully into IFAD operations. The EJRAP was the first to target women by forming women's groups and providing equipment and training: impact was small but significant. In the PIDRA, the impact on women is impressive. The attempt to address women and men in their own right is having positive results and should be strengthened. Yet, as programme activities expand, extension services may not be able to maintain the same intensity of community interaction; economic improvement is being sought by women and men, but the programme has yet to respond. Self-help groups of women or men and women are an effective instrument for enhaning the role of women. Although, increased income in the hands of women does have positive effects on their status, health and education (and that of their daughters) much depends on the intrahousehold distribution of power and income. Gender relations in Indonesia strongly favour men, so if women are to improve their status, more than the ability to earn extra income is required. In this regard, there is almost no evidence that IFAD-supported projects, even the most recent one, are taking a progressive, let alone an aggressive, posture. Lastly, there is a residual concern that implementation is still focused on achieving targets, such as the number of women's groups established and training sessions imparted, rather than on outcomes and the qualitative aspects of women's development.
Impact can be more accurately assessed if pre and post-project data regarding welfare levels are available. However, due to weak or non-existent baseline surveys and inadequate M&E systems, impact assessment has been a major challenge for the CPE. But by using the triangulation methodology, it has been possible to draw conclusions. Of the nine projects completed or nearly completed by mid-2003, four had positive income effects. Evidence of impact on the poor, beyond increased incomes, is limited partly because this was not explicitly sought in early projects. Only the PIDRA has so far made progress with this kind of impact. Even there, the most empowering effect on the poor so far has been learning how to save money and bridge crisis situations. The EKLCEP is also likely have impact on non-income aspects of poverty. The CPE argues that the search for success in non-income areas of rural poverty reduction has lead to insufficient attention to increasing the incomes of the poor. The field evidence points clearly to the need for a better balance between the income and non-income dimensions of poverty reduction in IFAD operations. Getting this balance right partly depends on better M&E systems, which have been weak throughout the portfolio, although in the last three years efforts (especially in the PIDRA) have been made to improve this function.
The portfolio does not score well on sustainability. Two completed projects are likely to be sustainable while in two others, this is highly unlikely. Of the four ongoing projects, only two are likely to be sustainable – P4K III and EKLCEP. In the PIDRA, eventual sustainability is judged unlikely due to the imbalance between the successful formation of social capital and the inadequate formation of economic capital in the form of sustained increases in income. Lack of sustainability can be attributed to failed or unsuitable technology, a lack of robustness in the social and institutional apparatus, poor physical implementation, and weak government and farmer institutions. Sustainability strategies are not embedded in project designs and project staff cannot explain how they will tackle the phase-out of project support or how benefits will be sustained beyond the project's lifetime. Group formation has provided a means of change (increased awareness, knowledge and capacity) for individuals, yet the sustainability of groups is not assured. The lack of sustainability in the portfolio must be regarded as its greatest weakness.
Project supervision. The CPE finds that both supervision intensity and continuity have declined in the last few years due to resource limitations and high turnover of staff in the cooperating institutions. Moreover, the proportion of cooperating institution staff and consultants on a given supervision mission participating in successive missions has declined. This has led, inter alia, to inadequate follow-up on previous supervision issues and recommendations. The cost of supervision is an issue. The United Nations Office for Project Services (UNOPS) informed the CPE that IFAD provides an annual per project cost of around USD 47 000, whereas its real costs are between USD 56 000 and 62 000 per project, depending on how overheads are handled. The CPE also underscores the need for the supervision function to evolve with the changing nature of IFAD operations. Most of the Fund's recent operations have a much higher community development aspect than earlier projects. Moreover, the fiduciary aspects of supervision need to be better balanced with the learning and performance enhancement dimensions of supervision.
The PIDRA is directly supervised by IFAD. There is support for the concept of direct supervision. Direct supervision has brought IFAD directly in touch with stakeholders, but it has encountered difficulties. So far it is seen as being unbalanced. It must be more supportive and participatory, and help projects overcome difficulties, inter alia, with procurement, and monitoring and evaluation processes.
Although direct supervision from Rome reduces the ‘distance' between IFAD and project stakeholders, it is not the same as supervision from a country office. In the latter situation, communication, depth of supervision and quality of follow-up are likely to be better. Finally, IFAD also undertakes implementation-support missions from Rome to provide inputs and advice to project partners in Indonesia. Coordination, synergies and feedback between the IFAD-led implementation support activities and the supervision exercises by cooperating institutions need to be improved.
Performance of partners. Overall government performance has been just satisfactory. In earlier projects, project implementation arrangements responded to the relatively simple project designs and were satisfactory. However, government performance in more recent projects where designs have called for stronger skills in social mobilization has faltered at all levels. At the local level, where decentralization is taking hold, government performance is mixed. Some district governments are strongly supportive of IFAD-supported projects while others are more concerned with control over resources and implementation authority. Finally, the Government of Indonesia needs to take sterner measures to combat corruption and enhance its partnership with NGOs and civil society at large.
The performance of NGOs has varied depending on the NGO involved and their institutional capacities and area of operation, but on the whole they have provided a new dimension to the partnership and made a useful contribution. In some instances, NGOs have been asked to undertake tasks beyond their abilities and scope of activities. There is need for IFAD to actively facilitate the building of a more equal partnership between government agencies and NGOs.
On balance, IFAD's performance needs significant improvement. On the positive side, IFAD developed well-focused strategies (both in 1988 and 1998) in Indonesia that were largely in line with IFAD's and the Government's priorities for rural poverty reduction, although the CPE argues that building social capital is overemphasized in the COSOP. IFAD has also built productive relations with the Ministry of Agriculture and NGOs. However, there are a number of areas where its performance needs measurable improvement, for example, in: policy dialogue and partnership-building, particularly with the international development community in Indonesia; in finding and deploying innovative solutions to poverty reduction; in stimulating higher agricultural production and productivity and better marketing opportunities; in project supervision; in combating corruption; in M&E; and in learning from past operations.
Corruption is widespread in Indonesia. The CPE was mandated to review generally the issue of corruption, although it did not undertake detailed financial investigations to assess transparency and identify possible financial irregularities. Nevertheless, tendering procedures were reviewed in each project. During the enquiries and at other times, there were frequent allusions to benefits leakage and shortcomings in procurement. Beneficiary self-assessments cited various cases of expected inputs being diverted and infrastructure being constructed at a lower level of quality than agreed. Collusion in procurement is evident and mostly ignored or even condoned by project management: one set of tender documents showed less than USD 500 difference between the three lowest bidders in a tender worth nearly USD 1.0 million; NGOs openly admit to agreeing who would bid and at what rate to ensure competition does not lower prices. IFAD does not have a policy on corruption and strong measures are needed to ensure project resources are not misappropriated.
Recommendations
IFAD needs to rethink its strategy in Indonesia. IFAD's comparative advantage does not lie in competing with the AsDB or the World Bank, but in being a progenitor of well-tested innovative approaches that can be scaled up by those with greater resources. IFAD's small size and flexibility should be exploited to instigate new models of rural development. By building on evidence from on the ground, IFAD could substantially increase and deepen its contribution to policy change and assume a position of knowledge and influence in councils such as the Consultative Group on Indonesia. IFAD's unique mandate is a powerful imperative for IFAD to take a leading role in showing how agriculture and rural development reduces poverty. For this to happen, IFAD needs to:
adjust its Indonesia country strategy to better balance the current focus on empowering the poor with efforts to raise farm and non-farm productivity. This will require, inter alia, stonger linkages with formal and non-formal agriculture research systems and promoting the development of markets and other aspects of market-linkages, such as rural infrastructure, market information and agro-processing; and
increase its staff and other inputs devoted to knowledge generation, advocacy and policy dialogue. In this regard, attention should be paid to generating evidence of what works, preferably of new things that work, to help carry the policy and advocacy dialogue forward. Moreover, it should use networking (both real and virtual) and experimentation on the ground as key instruments in knowledge generation.
To give effect, this shift in strategy requires at least three lines of action should be considered:
IFAD should establish and nourish strategic partnerships with: NGOs and community-based organizations working with the poor to find new and workable solutions to raising incomes and empowering all people; all levels of the administration to build capacity for effective poverty reduction; and other aid agencies to provide an audience and a market for new policies and ideas;
IFAD should provide greater support to its operations during implementation, better supervise and better monitor and evaluate its operations: improved supervision and implementation support is needed – possibly through in-country local staff. Such support must be knowledgeable about and effective in anti-corruption activities. Better quality M&E systems are essential if learning and knowledge are to be captured; and
IFAD should allocate adequate resources to implementing all objectives in its next COSOP for Indonesia. In addition, the COSOP should include a coherent hierarchy of objectives, for both lending and non-lending operations. It should contain performance indicators to monitor the implementation of the strategy, which will serve eventually to measure the performance and outcomes of the COSOP. The preparation of the COSOP should be based on a thorough analysis of the inputs, processes and activities required to achieve its objectives, as well as include a prioritization or a time plan for the delivery of its outputs.
1/ The Country Programme Evaluation mission was composed of Mr Roger Slade, Mission Leader/Development Economist; Mr Hans Dieter Seibel, Rural Finance; Ms Dorothy Lucks, Institutions/Decentralisation and Project Management; Mr Dimyati Nangju, Agriculture Expert; and Ms Suwan Yang, Sociologist. Mr Ashwani Muthoo, Senior Evaluation Officer, was the OE lead evaluator of this Country Programme Evaluation.