Evaluation of Project Performance and Impact
Strategy and Approach
Component-by-Component Evaluation
Project Coordination and Institutional
Aspects
Monitoring and Evaluation
Performance Monitoring Indicators
Targeting and Cross-Cutting
Issues
Project Supervision and Loan
Administration
It should be noted at the beginning of this
section that no attempt has been made to evaluate the two
projects – the first project and the twin project – separately.
While there are independent appraisal reports and loan agreements,
they have the same approach, the same components, the same
management arrangements, the same structure of project costs
and, since the launching of the twin project a year ago, they
have been managed essentially as one project by the same management
team.
Strategy and Approach
The strategic foundation for the project
is sound. It is well thought out and well presented in the
appraisal reports. It is worth quoting some of the key statements,
which are as relevant for the second phase as they were for
the design of the overall project:
· ‘The challenge in Umutara is to activate
a process of economic and human development consistent with
efficient, sustainable, and equitable mobilisation of human
and natural resources and with social reconciliation.’ This
is the opening statement of the project strategy and is eminently
sound.
· ‘The project strategy pivots around three basic considerations:
the concept of administration and community partnership …
and the principle of joint financing of most activities by
communities, government and civil society …’; … mobilizing
agents widely … to provide services … (to) be made accountable
to the end users for the services rendered; and … government
has a major role in regulation, a role to share in planning,
… is the supplier of last resort in delivery, and has a major
role in financing, …’
· ‘… rural communities participate in the
decision making and activity implementation and evaluation
processes as major stakeholders, and … they contribute a share
of activity cost sufficient to prove their commitment to acquire
ownership.’
· Resources are made available to all service providers,
… subject to performance,’
· ‘It is important to avoid over-taxing
the limited implementation capacity of the emerging institutions
and to test the practical implication of the arrangements
proposed before adopting them on a large scale.’
Objectives. Project objectives are
divided into two – (i) institutional development and
good governance objectives and (ii) specific objectives.
The first set of objectives is fine provided that it is adapted
to reflect decentralization and the district role in promoting
local development: establishment of an effective bottom-up
planning process, an efficient service provision process,
a continuous participatory service providers’ evaluation process,
and emphasis on sustainability. The subsequent listing of
ten specific objectives indicates a lack of focus. Nowhere
in the section on ‘Project Objectives’ is there a statement
of the overall objective or development goal of the project,
though the ‘dynamic logical framework’ uses the statement
contained in the first bullet point (above) as the development
goal.
Project Approach. The general
approach and strategy for project implementation is centred
on two main elements: demand-driven participatory processes
and project response through performance-based contracts.
The principle of using a participatory planning process to
identify needs and priorities and to match a series of deliverables
to meet those priorities has been employed successfully in
many projects and it is particularly relevant in Rwanda where
the previous regime had employed a centralized top-down system
of administration that allowed little interaction with the
communities. The second element of the design – performance-based
contracting or ‘outsourcing’ – can be an effective way to
promote efficient and flexible use of limited resources but
needs to be adapted to reflect decentralization and district
responsibilities. The aim was to use a range of service providers
to implement all project interventions, with the project coordination
unit intended to provide the facilitation, monitoring, oversight
and financial controlling function. The designated service
providers included both private sector entities – primarily
national and international non-governmental NGOs (referred
to as N-NGOs and I-NGOs) – and public sector entities such
as the National Agricultural Research Institute (ISAR), the
national agricultural research institution, and the extension
department of the Ministry of Agriculture (MINAGRI). While
the principle was sound, its translation into practice has
been less successful. As discussed later, the project has
been successful in carrying out participatory planning in
the communities and in assisting the sectors and districts
to develop identified needs and priorities into district development
plans. It has been less successful in matching the investments
available from the projects – ‘the deliverables’ – with the
plans. Similarly, the detailed implementation arrangements
for management of the service providers and in particular
the working relationship between the N-NGOs and I-NGOs has
been problematic. This is discussed later under project management
and implementation arrangements. It should also be noted that
the very detailed specification of project investments – as
presented in the cost tables – goes counter to the participatory
approach and removes a considerable amount of the flexibility
needed to respond to demand and priorities expressed by the
communities.
The scope of the project is one of
the aspects of project design that has elicited considerable
debate. It is ambitious in its breadth and scale of activities,
covering: agricultural development, livestock production
and veterinary activities, forestry and natural resource management,
development of domestic and cattle water requirements, feeder
road construction, rural financial services, community capacity
building and women in development initiatives. Admittedly,
these activities respond to clear needs within the province
and as the project was the only major development initiative
within the province (and still is), there was pressure to
include as wide a range of agricultural and infrastructure
elements as possible. Nevertheless, even with its principle
of ‘outsourcing’, it imposes a heavy burden on project management.
What is clear from the evaluation is that the strong emphasis
on investment in domestic and cattle water points and the
sizeable allocation of project funds to water development
was correct and reflects the real and expressed need of the
people for improved access to water. Feeder road development
is also clearly a pressing need in most areas of the province,
certainly in those areas that have been taken over from the
National Park where no roads existed in the past. Thus, the
emphasis in project design on these two interventions is well
placed. The key role played by agriculture in the economic
development of the province is also undisputable and thus
the inclusion of support for agricultural, livestock and forestry
development can also be justified. A case could also be made
for including rural finance but the opportunities for sustainable
provision of such services then and even now are limited and
the sector is notoriously difficult to handle, especially
for a multi-faceted project with no real expertise in rural
finance. Nevertheless, expansion of agricultural and livestock
activities could be constrained without access to finance
and an improved ability to build-up savings.
Scope in terms of geographical coverage
and financial provision is also relevant in assessing the
suitability of project design. The expansion of the original
project area from four to the current eight districts (i.e.
the whole province) was a sound decision and has made it much
easier to harmonize development initiatives. Funding is a
more difficult issue. The level of funding available from
the two projects has been brought into particular focus by
the limited disbursements that have taken place during the
first phase. Out of the USD 57 million available from
the two projects over the proposed ten-year investment period,
only USD 4.5 million (out of USD 26 million budget
for the first phase) has been spent to-date. As the funding
provided for the third phase is only USD 2.5 million,
it means that about USD 50 million is now available for disbursement
in the second phase, a figure almost certainly in excess to
the amount that could be absorbed over the four-year period.
To exacerbate the problem, the mission found that the water
sub-component is over budgeted. Assuming all investments
stipulated in the Water Master Plan for Umutara are financed
by the project, only two thirds of the USD 20 million
provided would be spent. It must thus be concluded that the
project will have difficulty in fully disbursing project investment.
Even assuming reallocation of investment ear-marked for water
to other activities, such as livestock development, it might
eventually be necessary to extend the investment period beyond
the projected ten years. While the first project is for ten
years, the twin project is for seven years and would be closed
at the end of the second phase. Extension of the twin project
for an additional three years might have to be considered,
both in terms of compatibility and fund disbursement.
Regarding implementation arrangements
for the project and the approach to project management, the
broad concepts are fine but their translation into operating
procedures and structures is flawed. The idea that experienced
I-NGOs would work with, help train and monitor N-NGOs is fine
in theory but has worked poorly in practice as the contractual
arrangements employed create little obvious linkage or line
of communication between them; the N-NGO contracts are directly
with the Project Coordination Unit (PCU), as are those of
the I-NGOs, thus making it difficult to establish constructive
working relations between the two sets of service providers.
Furthermore, certain of the I-NGOs were weak and were not
able to consistently make available sufficiently qualified
staff to effectively fulfil their terms of reference and to
support the N-NGOs. Similarly, regarding the PCU, the stated
functions are sound: ‘(i) manage project funds, (ii) contract
implementation of project components to service providers,
and (iii) monitor the performance of the service providers
and evaluate the outcome of their activities’. But, the resources
allocated to the PCU – three staff – were insufficient to
manage such a project.
|
