The challenges of conducting impact assessments of community driven development projects
The challenges of conducting impact assessments of community driven development projects
Impact assessments look at whether a project intervention had an impact on the people and communities participating in the project by comparing them with people who share similar social and economic characteristics but who were not part of the project. But how do you identify the people and communities to compare with when a project covers a whole range of activities as is often the case in community driven development projects?
IFAD has a longstanding tradition of implementing community driven development (CDD) projects. In the Latin America and Caribbean region, the CDD approach is one of the main strategies promoted by IFAD. The demand-driven nature of this approach has proven to be an effective tool to empower rural people and ensure that remote and marginalized populations are included in the development of their own communities. However, while the CDD approach presents some well-established benefits for project participants and the development objectives, it poses some particular challenges for the design of an impact assessment.
Two CDD projects from the Latin America and Caribbean region are included in the upcoming impact assessment cycle: the Economic Inclusion Programme for Families and Rural Communities in the Territory of Plurinational State of Bolivia – ASAP (ACCESOS-ASAP) in Bolivia and the Strengthening Local Development in the Highlands and High Rainforest Areas (PSSA) project in Peru. In preparation of the impact assessments, researchers from IFAD’s Research and Impact Assessment Division (RIA) went on a scoping mission to Bolivia and Peru to learn more about the projects through meetings with the country team, and project staff and participants. Based on the scoping missions, impact assessment plans will be prepared, detailing the strategy and steps to conduct the impact assessments.
The target populations of both projects live in remote areas characterized by high levels of poverty, limited access to markets and services, and degrading or threatened natural resources. The ACCESOS-ASAP project provides financial resources to community-based groups, second-tier organizations, and other priority groups, including young women and men, to invest in economically viable natural resource management systems and small agricultural and non-agricultural businesses. Community groups are assisted in the preparation of project and business proposals that are then evaluated to receive a grant. Community members contribute in cash or in kind between 20 and 30 per cent of the total amount. The PSSA in Peru focuses on small-scale producers and encourages them to form an association in order to qualify for a grant. The producers contribute 20 per cent of the total amount and the money is used to improve the assets required for production, technical assistance and inputs. In addition, women receive financial education and access to savings accounts and life insurance.
As the projects are demand-driven, community members themselves select the activities they want to pursue. The projects therefore cover a variety of activities, such as soil conservation, vegetative cover, irrigation, improvement of cooking stoves, as well as the production and transformation of a big range of products, such as guinea pigs, textiles, trout, pigs and coffee in Peru, and high value fruit, flowers, dairy products, handicrafts and textiles in Bolivia.
While communities generally benefit from such a wide range of activities, assessing the impact of the projects becomes more challenging, especially when it comes to identifying the correct comparison group. For an impact assessment to establish whether certain outcomes can be attributed to a project, it is necessary to compare people who participated in the project to people who are in a similar situation but did not benefit from project activities. In a setting where the project entails a variety of activities, identifying the comparison group is not straightforward.
To address this challenge, the researchers need a clear understanding of how the project was implemented and what the underlying commonalities and assumptions are. The scoping missions therefore include meetings with the project management unit and the project team to discuss project design and implementation, including changes that were done during the implementation phase, and to identify core research questions and re-assess the theory of change. The RIA team also visits project sites to gain a better understanding of the realities in the field.
As a result of the scoping mission, the RIA team working on the PSSA project in Peru decided not to evaluate every aspect of the intervention. Instead, the impact assessment will focus on the business plan component which is expected to have the greatest impact. The study will be geographically restricted to areas where the top five products chosen by beneficiaries are most common. This can be done by using secondary data sources, such as the agricultural census.
In Bolivia, the scoping mission was an opportunity for RIA to closely work with the project team using project M&E data to provide a clear picture of both the geographic and thematic distribution of project interventions. It was then decided to focus the impact assessment on specific combinations of project activities with higher outreach for beneficiaries. Furthermore, geospatial data will be used to identify a suitable comparison group looking at non-beneficiary communities in the geographic proximity and with similar bio-physical characteristics of beneficiary communities. This is to ensure that the two groups have the same production potential and similar natural resources endowments, which is critical in a CDD project setting with a large diversity of activities and products.
- For more information about impact assessments
- For information about the Bolivia and Peru missions and impact assessments, please contact Aslihan Arslan and Adriana Paolantonio