Evaluation of Project Performance and Impact
It should be noted at the beginning of this section that no attempt has
been made to evaluate the two projects – the first project and the twin
project – separately. While there are independent appraisal reports and
loan agreements, they have the same approach, the same components, the
same management arrangements, the same structure of project costs and,
since the launching of the twin project a year ago, they have been managed
essentially as one project by the same management team.
Strategy and Approach
The strategic foundation for the project is sound. It
is well thought out and well presented in the appraisal reports. It is
worth quoting some of the key statements, which are as relevant for the
second phase as they were for the design of the overall project:
· ‘The challenge in Umutara is to activate a process
of economic and human development consistent with efficient, sustainable,
and equitable mobilisation of human and natural resources and with social
reconciliation.’ This is the opening statement of the project strategy
and is eminently sound.
· ‘The project strategy pivots around three basic considerations: the
concept of administration and community partnership … and the principle
of joint financing of most activities by communities, government and civil
society …’; … mobilizing agents widely … to provide services … (to) be
made accountable to the end users for the services rendered; and … government
has a major role in regulation, a role to share in planning, … is the
supplier of last resort in delivery, and has a major role in financing,
…’
· ‘… rural communities participate in the decision making
and activity implementation and evaluation processes as major stakeholders,
and … they contribute a share of activity cost sufficient to prove their
commitment to acquire ownership.’
· Resources are made available to all service providers, … subject to
performance,’
· ‘It is important to avoid over-taxing the limited
implementation capacity of the emerging institutions and to test the practical
implication of the arrangements proposed before adopting them on a large
scale.’
Objectives. Project objectives are divided into
two – (i) institutional development and good governance objectives
and (ii) specific objectives. The first set of objectives is fine
provided that it is adapted to reflect decentralization and the district
role in promoting local development: establishment of an effective bottom-up
planning process, an efficient service provision process, a continuous
participatory service providers’ evaluation process, and emphasis on sustainability.
The subsequent listing of ten specific objectives indicates a lack of
focus. Nowhere in the section on ‘Project Objectives’ is there a statement
of the overall objective or development goal of the project, though the
‘dynamic logical framework’ uses the statement contained in the first
bullet point (above) as the development goal.
Project Approach. The general approach and
strategy for project implementation is centred on two main elements:
demand-driven participatory processes and project response through performance-based
contracts. The principle of using a participatory planning process to
identify needs and priorities and to match a series of deliverables to
meet those priorities has been employed successfully in many projects
and it is particularly relevant in Rwanda where the previous regime had
employed a centralized top-down system of administration that allowed
little interaction with the communities. The second element of the design
– performance-based contracting or ‘outsourcing’ – can be an effective
way to promote efficient and flexible use of limited resources but needs
to be adapted to reflect decentralization and district responsibilities.
The aim was to use a range of service providers to implement all project
interventions, with the project coordination unit intended to provide
the facilitation, monitoring, oversight and financial controlling function.
The designated service providers included both private sector entities
– primarily national and international non-governmental NGOs (referred
to as N-NGOs and I-NGOs) – and public sector entities such as the National
Agricultural Research Institute (ISAR), the national agricultural research
institution, and the extension department of the Ministry of Agriculture
(MINAGRI). While the principle was sound, its translation into practice
has been less successful. As discussed later, the project has been successful
in carrying out participatory planning in the communities and in assisting
the sectors and districts to develop identified needs and priorities into
district development plans. It has been less successful in matching the
investments available from the projects – ‘the deliverables’ – with the
plans. Similarly, the detailed implementation arrangements for management
of the service providers and in particular the working relationship between
the N-NGOs and I-NGOs has been problematic. This is discussed later under
project management and implementation arrangements. It should also be
noted that the very detailed specification of project investments – as
presented in the cost tables – goes counter to the participatory approach
and removes a considerable amount of the flexibility needed to respond
to demand and priorities expressed by the communities.
The scope of the project is one of the aspects
of project design that has elicited considerable debate. It is ambitious
in its breadth and scale of activities, covering: agricultural development,
livestock production and veterinary activities, forestry and natural resource
management, development of domestic and cattle water requirements, feeder
road construction, rural financial services, community capacity building
and women in development initiatives. Admittedly, these activities respond
to clear needs within the province and as the project was the only major
development initiative within the province (and still is), there was pressure
to include as wide a range of agricultural and infrastructure elements
as possible. Nevertheless, even with its principle of ‘outsourcing’, it
imposes a heavy burden on project management. What is clear from the evaluation
is that the strong emphasis on investment in domestic and cattle water
points and the sizeable allocation of project funds to water development
was correct and reflects the real and expressed need of the people for
improved access to water. Feeder road development is also clearly a pressing
need in most areas of the province, certainly in those areas that have
been taken over from the National Park where no roads existed in the past.
Thus, the emphasis in project design on these two interventions is well
placed. The key role played by agriculture in the economic development
of the province is also undisputable and thus the inclusion of support
for agricultural, livestock and forestry development can also be justified.
A case could also be made for including rural finance but the opportunities
for sustainable provision of such services then and even now are limited
and the sector is notoriously difficult to handle, especially for a multi-faceted
project with no real expertise in rural finance. Nevertheless, expansion
of agricultural and livestock activities could be constrained without
access to finance and an improved ability to build-up savings.
Scope in terms of geographical coverage and financial
provision is also relevant in assessing the suitability of project design.
The expansion of the original project area from four to the current eight
districts (i.e. the whole province) was a sound decision and has made
it much easier to harmonize development initiatives. Funding is a more
difficult issue. The level of funding available from the two projects
has been brought into particular focus by the limited disbursements that
have taken place during the first phase. Out of the USD 57 million
available from the two projects over the proposed ten-year investment
period, only USD 4.5 million (out of USD 26 million budget for
the first phase) has been spent to-date. As the funding provided for the
third phase is only USD 2.5 million, it means that about USD 50 million
is now available for disbursement in the second phase, a figure almost
certainly in excess to the amount that could be absorbed over the four-year
period. To exacerbate the problem, the mission found that the water sub-component
is over budgeted. Assuming all investments stipulated in the Water
Master Plan for Umutara are financed by the project, only two thirds of
the USD 20 million provided would be spent. It must thus be concluded
that the project will have difficulty in fully disbursing project investment.
Even assuming reallocation of investment ear-marked for water to other
activities, such as livestock development, it might eventually be necessary
to extend the investment period beyond the projected ten years. While
the first project is for ten years, the twin project is for seven years
and would be closed at the end of the second phase. Extension of the twin
project for an additional three years might have to be considered, both
in terms of compatibility and fund disbursement.
Regarding implementation arrangements for the
project and the approach to project management, the broad concepts are
fine but their translation into operating procedures and structures is
flawed. The idea that experienced I-NGOs would work with, help train and
monitor N-NGOs is fine in theory but has worked poorly in practice as
the contractual arrangements employed create little obvious linkage or
line of communication between them; the N-NGO contracts are directly with
the Project Coordination Unit (PCU), as are those of the I-NGOs, thus
making it difficult to establish constructive working relations between
the two sets of service providers. Furthermore, certain of the I-NGOs
were weak and were not able to consistently make available sufficiently
qualified staff to effectively fulfil their terms of reference and to
support the N-NGOs. Similarly, regarding the PCU, the stated functions
are sound: ‘(i) manage project funds, (ii) contract implementation
of project components to service providers, and (iii) monitor the
performance of the service providers and evaluate the outcome of their
activities’. But, the resources allocated to the PCU – three staff – were
insufficient to manage such a project. |
