Issue 34: October 2010 - Statistics

In this issue

   
 

On 20 October 2010 the world celebrates the first World Statistics Day, to raise awareness of many achievements of official statistics premised on the core value of service, professionalism and integrity

 

Statistics are crucial for project design and implementation, monitoring and measurement of project performance, and evaluation. They are also vital for a deeper understanding of different dimensions of poverty and vulnerability of poor rural people to various risks, so that appropriate policies and programmes can be formulated and implemented to lift them out of poverty.

Following the adoption of the Results and Impact Monitoring System (RIMS) in 2005, IFAD-supported projects have started to systematically collect project performance data through baseline, mid-term and completion surveys. This has allowed important project outputs to be measured,  such as household assets and child nutrition, a proxy indicator for improved well-being of the target population in these projects. Valuable lessons have been learned over the past years to improve the RIMS methodology, so that it can also be used as a project management tool. IFAD’s Asia and the Pacific Division has also introduced an alternative method – the annual outcome surveys – to cost- effectively measure project outcomes and impacts.

A recent policy study that IFAD conducted in collaboration with the Government of Lao People’s Democratic Republic highlighted the importance of reliable, time-series data for such analyses and to generate useful recommendations. Significant opportunities exist for IFAD to work with other development partners such as the Food and Agriculture Organization of the United Nations, the Asian Development Bank and regional institutions to help build the capacity of countries with weak data collection and analysis capacity.

This special issue has been brought out to mark World Statistics Day (October 20), to raise awareness about the importance of reliable, timely data collection and analysis for useful project and policy work and to highlight some of the recent work done in the Asia and the Pacific region to improve the collection and use of statistics. We hope that readers will find the articles informative and useful.    

Ganesh Thapa, Regional Economist, Asia and the Pacific Division


Poverty, vulnerability and statistics – in the context of Lao People’s Democratic Republic

   
 

A farmer collecting vegetables from her garden, Lao PDR

 

"There are three kinds of lies: lies, damned lies, and statistics," lamented Mark Twain. Whatever the basis of his lament, statistics are vital to a deeper understanding of the many different dimensions of poverty, and of the vulnerability of poor people to a range of risks that propel many of them into long spells of poverty. Statistics are also crucial for designing and implementing policies that will help not just mitigate but prevent deprivation that is pervasive and often brutish.

Awareness of the usefulness of comprehensive and reliable statistics has grown. However, many of the poorest countries are overwhelmed by the challenge of collecting data and analysing it in a timely way. Moreover, they lack capacity to address this challenge. IFAD’s Asia and the Pacific Division recently conducted two studies of Lao People’s Democratic Republic (PDR) – Agriculture, GDP and Prospects of MDG 1 in Lao PDR (2010) and A Note on Risks and Vulnerability in Lao PDR (2010). The studies unraveled this challenge and highlighted the policy priorities.

The conclusions drawn from these two studies are reflected below. They follow from the analysis of data sets that the Lao government had collected but was not able to use to identify the Seventh plan priorities for the agricultural sector. Moreover, most of the household data collected were rich in content but required much time to clean and ensure consistency of key variables.

Poverty is a static concept (such as the number or proportion of poor people at a given time). Insecurity and vulnerability, by contrast, are dynamic, as they characterize responses to changes over time. Insecurity is exposure to risk; and vulnerability, the consequent decline in well-being.

As low-income households cannot deal effectively with risks, a poverty reduction strategy must pay careful attention to risk reduction, mitigation and coping mechanisms. As a general observation, there is a case for greater emphasis on risk reduction and mitigation strategies, as opposed to risk-coping strategies. Also, both formal and informal mechanisms through private and public agencies need to be integrated, taking into account their complementarityas well as potential to substitute assistance or support provided by friends and relatives.

Overall, the evidence for Lao PDR suggests that natural shocks and disasters, including pest infestations, livestock disease and flooding, are the main sources of vulnerability. However, health shocks are also important both in terms of incidence and welfare loss. As the Lao economy grows and integrates into global markets, market-related shocks will become a more serious concern.

Recent evidence from A Note on Risks and Vulnerability in Lao PDR points to patchy and highly fragmented risk-sharing and coping mechanisms, as delineated below.

   
 

A farmer demonstrating his garden to project officers, Lao PDR

 

Although Lao households use a variety of coping mechanisms to deal with shocks, their capacity is highly constrained. Use of savings is a frequently adopted coping strategy. Many also rely on social networks of friends and family, borrowing rice from them during lean season months, or seeking their help to restore homes and property during floods. Households also use alternative sources of food, including wild resources. They may also trade livestock and other assets to deal with natural disasters and idiosyncratic risks. However, these mechanisms are far from adequate –  more than 50 per cent of the households reported being forced to reduce their consumption in response to shocks.

A broader range of strategies is imperative for mitigating and coping with shocks, including raising livestock and poultry, gardening or alternative crops, producing handicrafts, fishing and creating social networks. Ownership of assets and diversified livelihoods significantly reduce vulnerability.

In addition to gathering data on sources of risks and vulnerability along the lines indicated in the two studies (for example, markets risks, natural disasters), important priorities for the Agricultural Census (2010/11) are to collect data to assess the efficacy of precautionary savings and social networks in risk mitigation and coping. Will diversified agriculture and non-farm activities help mitigate crop shocks and/or help people cope with these shocks? Does limited access to institutional credit constrain diversification within and outside agriculture? Is there potential for microfinance to be used for agriculture and for promoting micro-insurance? Appropriate indicators could be designed to gather evidence on these aspects to examine policy options (for example, shares of income from different crops and non-farm sources of income). Disaggregation of these indicators by size of holding, ethnic group and location would be useful.

Given the vulnerability of large segments of Lao PDR’s rural population to natural disasters and the inadequacy of precautionary savings and social networks to cushion them against such shocks, it is worthwhile to examine whether there is a demand for disaster insurance –  more specifically, whether individuals and communities are willing to pay in order to mitigate and cope with catastrophic risks, including weather-related hazards. Careful attention must be given to how people process information on low-probability high-consequence negative events. A case in point is the tsunami that struck parts of Asia in December 2004. The probability of another tsunami is low – once in two hundred years – and the devastation is enormous. If people are asked to mobilize their resources for such events, they may be indifferent. But if they are told that the risk of dying is likely to be halved if certain precautionary measures are undertaken, the response of the community is likely to be stronger.

In conclusion, policies to address vulnerability to various risks are best designed and implemented from a rich and comprehensive data base that could be analysed to measure these risks and their impacts, and how individuals, communities and public agencies could mitigate the welfare losses.

R. Gaiha, Department of Urban Studies and Planning, Massachusetts Institute of Technology, USA, and Faculty of Management Studies, University of Delhi, India; S. Annim , University of Manchester, England; and G. Thapa, IFAD, Italy


Project monitoring and evaluation: An important data source for independent evaluation

   
 

Project staff collecting data from project beneficiaries, Indonesia

 

Data and statistics are very important types of information to be captured by project monitoring and evaluation (M&E). If they are collected and used properly, M&E can be a strong tool for project management to improve the performance of a project and its impact on beneficiaries. The evidence from strong M&E systems also provides a more solid foundation for independent evaluations of a project. The credibility and quality of each independent evaluation is largely based on the robustness of its data and analysis. 

Monitoring and evaluation are distinct yet related activities. The main differences between them are that monitoring is ongoing, focused on tracking evidence of movement towards the achievement of specific, predetermined targets by the use of indicators. Evaluation is periodically taking a broader view of an intervention, asking whether the progress towards the target or explicit result is caused by the intervention or whether there is some other explanation for the changes appearing in the monitoring system. Together, both ‘M’ and ‘E’ can be useful tools in assessing the performance of a project and contributing to learning.

Generally, IFAD’s independent evaluations found that the Fund recognizes the importance of project M&E systems. However, their performance has been varied, and concern about weak M&E systems and data collection has been a recurrent evaluation finding.

Some of the issues associated with weak M&E that limit the availability and quality of data and affect independent evaluation are described below.

   
 

M&E staff meet project beneficiaries to collect data on project performance, Pakistan

 

Clarity and complexity of objectives: Objective-based evaluation assesses the attainment of outcomes against objectives specified at project design, making it important to have clear objectives that build on a logical theory of change. Unfortunately, some projects have multiple logical frameworks with competing objectives. A recent Country Programme Evaluation in Pakistan reported contradictory logical frameworks combined with arbitrary and irrelevant indicators. This often causes confusion for project monitoring as there is not a single set of agreed indicators linked to the project objective. As a result, these projects often have incomplete or inappropriate data for measuring progress against the objectives and to use for the evaluation.

Appropriate indicators and reporting methods: Good M&E systems need to select the right indicators and define them clearly. These basic principles are at times forgotten at the project design phase, resulting in indicators that are neither relevant nor measurable. Sometimes, these indicators are also not related to the data being collected during the baseline surveys. In these cases, the data generated from project M&E are less useful for evaluation because they are not consistent or comparable. Another issue that emerged during a project evaluation in China is the need to collect disaggregated data that can be checked for accuracy and used for analysis, as opposed to simply aggregating it for a total amount.

Lack of baseline surveys: The lack of baseline data makes it difficult to assess effectiveness and impact, and can lead to unrealistic targets and expectations. It is important for evaluations to be able to assess the results achieved on the ground and attribute the results to the development project. Evaluation uses the ‘before and after’ technique, which benefits greatly from the presence of baseline data, to assess and attribute effects to a particular project. In some cases if baseline data are missing, not comparable or incomplete, evaluations can use national statistics data as baselines and follow-up to set up comparisons between project and control observations.

Limited analysis and utilization of data: Data, even when available, are often under-analysed and under-utilized. By establishing a partnership with national specialists (national institutes of statistics, universities, research centres), perhaps on a country-programme basis, IFAD could reduce the costs of collecting data and of analysing it and could also provide training to its project M&E staff.

There are also examples of evaluations that have benefited from the availability of comprehensive M&E data. For example, another China project evaluation made use of the multiple records kept by the provincial and county project management offices, based upon the project’s M&E system, which provided valuable insights into the workings of the project in the ten target counties. The evaluation went on to state that, ”The fact that most of the achievements were not only recorded for the project in its strict sense, but also with regard to the relevant universe of the ten project counties, made the evaluation particularly rewarding,” Thus the project was able to compare and analyse data from the project with data of the county and province.

Andrew Brubaker, Evaluation Officer, IFAD

Useful links:


Using statistics and other data in IFAD Country Strategic Opportunities Programmes

In September 2006, IFAD adopted a new process for drawing up results-based Country Strategic Opportunities Programmes (COSOPs). Statistics and data play an important role in drawing up the COSOP document and in reporting the results obtained.

Statistics in COSOP papers

   
 

Staff from the Sunamganj Community-based Resource Management Project meet with a Beel User Group to discuss project performance, Bangladesh

 

The COSOP document includes brief descriptions of the country’s economic background, rural poverty and its agricultural sector. Information on the national economy should include data on income per head, gross domestic product growth, population growth, and inflation. Most of these data can be obtained from national statistical agencies. In general this information is readily available and can often be obtained through the internet. 

However, in some countries, especially fragile and conflict-affected states, official data may not be available. An example of this is the 2008 COSOP for Afghanistan, which points out that the last population census was carried out in 1979. In this situation the COSOP made use of external sources such as Economist Intelligence Unit reports, and World Development Indicators (of the World Bank), or special studies carried out by donor agencies. Agencies such as the World Bank, Asian Development Bank (ADB) and International Monetary Fund can be useful sources of forecasts of future economic growth and key constraints faced in economic development. 

Data on rural poverty and the agricultural sector should include estimates of proportions of the population below poverty lines, and progress on Millennium Development Goal indicators. These data can also be drawn from national statistical agencies, and may include data from national poverty reduction strategies. In addition, the United Nations and other agencies generate much useful data and analysis, often based on national statistics. The Human Development Report of the United Nations Development Programme is an example, and its overall ranking of countries (the Human Development Index) is often quoted in COSOP papers. There should be a summary of gender-related constraints and opportunities which may use the Gender Empowerment Measure (GEM) and Gender and Development Index (GDI), which are available from UNDP.  

For agriculture, key data may include land tenure (access to land), food security (including food price inflation), and growth of the sector and sub-sectors (crops, livestock, fisheries). Key drivers of sub-sector growth (or stagnation) may be identified with statistics on changes in yield of staple crops and changes in crop areas. There may also be data available on natural disasters that have had an impact at the sector level.  

It might also be useful to include data on adoption of technologies (fertilizer use, high-yielding varieties), which may be available from specific programme or sector studies (such as those commissioned by the World Bank or ADB). However, with a limited allocation of space in the main COSOP paper, such detailed information may be more appropriately included in a background working paper – and maybe discussed at a COSOP preparation workshop. 

Statistics for COSOP results

   
 

Female Fish Pond Group during their meeting with M&E staff, Bangladesh

 

The COSOP paper also has a section entitled ‘Lessons from IFAD’s Experience in the Country’. This includes the past results, impact and performance of IFAD-supported projects, and the resulting lessons learned. Recent COSOPs for Asia and the Pacific include data on the outreach of the total IFAD programme as well as the overall flow of IFAD funds. 

However, it has not been easy to produce data on the results of the IFAD project portfolio, and COSOP papers tend to have more subjective information rather than hard data in this section. Results-based COSOPs should generate data on programme performance, including a number of quantified outcome and milestone indicators. Such data need to be collected at the project level and should be generated by project monitoring and evaluation (M&E) systems. The data would be used for regular COSOP progress reports and COSOP mid-term and completion reviews, as well as reporting on results and lessons as part of planning of a new COSOP. 

Although routine project reporting of activities and outputs provides some information on milestone indicators, further information on project outcomes will be needed for most COSOP results frameworks. Up until now, most project M&E systems have only generated outcome and impact data in surveys carried out at mid-term and completion. This meant that, at best, data would only be available from some projects (those that had reached the mid-term stage) and from projects that had carried out such surveys within the COSOP reporting period. COSOP progress reports and reviews produced for Bangladesh and Nepal reflect partial availability of data – reporting outcomes and impacts on the basis of individual projects rather than for the programme as a whole. Some projects are now moving towards annual outcome surveys, which could generate data on at least milestone indicators (such as numbers of farmers reporting yield increases). 

One challenge in reporting country-level results as a COSOP outcome is that individual projects generate data in different ways. Some of the data can be aggregated, such as the total number of microfinance loans or the total length of road constructed. But other information is collected in different ways by different projects. For example, one project that promotes new agricultural technologies may report on the increase in yield on demonstration plots, another may report on the number of farmers who report yield increase, while a third project could record the dissemination of the technology over a certain area. Unless these projects use common indicators (such as IFAD’s Results and Impact Monitoring System’s (RIMS) indicator of number of farmers reporting yield increase), it will not be possible to report overall country-level results. To do this, each project will need the staff and resources to collect data on the agreed common indicators. 

Edward Mallorie, Development Consultant, IFAD, [email protected]  

Useful links:


Measuring project outcomes and impact in Asia and the Pacific

   
 

Maria Donnat during the writeshop on M&E in October 2010 in Thailand

 

As a signatory of the Paris Declaration on Aid Effectiveness, IFAD places great importance on measuring the impact of the projects and programmes it helps finance. Without reliable, quantified outcome and impact data, how can one measure the real performance of projects in achieving their objectives? Yet, even statistics and data may not always be the panacea.

Since the adoption in 2005 of a corporate Results and Impact Monitoring System (RIMS), IFAD-funded projects are organizing standard baseline, mid-term and completion surveys in order to gather data on:
(i) household assets, so as to measure project outcomes on asset ownership; and (ii) child nutrition, as a proxy indicator for overall improvement in the well-being of targeted populations. Here, anthropometric measures provide information on three parameters:

In 2010, for the first time a number of projects had conducted at least two rounds of surveys (baseline and mid-term, or mid-term and completion), allowing the Asia and the Pacific Division to analyse impact as measured under the corporate RIMS. Results for selected projects were as follows:


An examination of these results shows that the variations in the three malnutrition indicators are somehow difficult to interpret. Hence the same project may record a decrease in chronic malnutrition but an increase in the percentage of children underweight. Furthermore, the “Acute malnutrition” indicator appears to be highly sensitive to annual variations, as was the case in Lao DPR, where the RIMS completion survey was conducted at the end of an agricultural season marked by unusual floods and rodent infestation and where an increase in acute malnutrition was noted at project completion (from 4.12 to 5.28 per cent), although there was a decrease in chronic malnutrition (from 53.22 to 47.25 per cent).

Furthermore, and given that the RIMS methodology does not promote the use of a control group, no comparison can be made between the status of households that have taken part in project implementation (the treatment group) with the status of those who have not (control group). Here, the danger is really to under-estimate project impact. Also, the methodology is somehow “IFAD-centric”: it was developed primarily to feed corporate needs for impact data – mainly for reporting purposes – rather than as a project management tool.

   
 

Participants discussing the project outcome survey during the writeshop on M&E in October 2010 in Thailand

 

In seeking cost-effective ways to measure project outcomes and impact more frequently – so that project managers have access to timely performance information that would inform their decision-making – the Asia and the Pacific Division is now promoting undertaking Annual Outcome Surveys, in conjunction with case studies and focus group discussions (the topics of which should be very strategically selected). The methodology is currently being fine-tuned and has been informed by a number of recent project experiments. It is based on the following guiding principles:

Given that most IFAD-funded projects may fall under the label “area-based, integrated rural development” projects, with the implication that project activities are usually in several sub-sectors (e.g. agricultural development, livestock development, irrigation, microfinance, forestry, small-medium enterprise development, off-farm production, land ownership rights), our projects need to be able to measure the differentiated impact they are having in each of these areas.

What Asia and the Pacific Division is suggesting is really to ensure that a sufficient body of evidence consisting of quantitative and qualitative data on project outcome and impact is gathered through more frequent interactions with project beneficiaries, so as to deepen our understanding of the dynamics that the project has introduced in the community and its households. In other words, it is less about precision and statistical significance and more about deepening understanding and knowledge.

Maria Donnat, Country Programme Manager and Results Specialist


Data challenges and opportunities for the Performance-Based Allocation System

IFAD’s Performance-Based Allocation System (PBAS) aims to allocate lending resources based on a country’s need and performance. While a country’s need is calculated using statistics on rural population and national per capita income, the country’s performance is based on IFAD’s assessment of project performance and the Rural Sector Performance Assessment (RSPA) of the country’s rural development policies. Due to a lack of data on rural areas, the RSPA, with its qualitative indicators, can be open to interpretation and the subjectivity of Country Programme Managers (CPMs). The objectivity of the RSPA can be enhanced by ‘triangulating’ available quantitative data with qualitative evidence and comparing RSP country scores within the region and sub-regions. This improved objectivity will enhance the value of the RSP scores and strengthen its use as a tool for policy dialogue on rural development with governments.

A key determinant of a country’s three-year PBAS allocation, the Rural Sector Performance Assessment is updated annually to reflect any changes in a country’s policy performance. The RSPA is calculated on the basis of scores in five policy themes:

The five themes are divided into twelve sub-themes with accompanying indicators. These predominantly qualitative indicators are presented in a detailed questionnaire which quantifies the country policy performance into a rating from one to six. Using relative terms such as “adequate”, “some”, “little”, “partly” and “seldom”, the questions and the descriptions corresponding to the ratings are open to interpretation, especially given the varying contexts of countries across Asia and the Pacific.

Although a uniform questionnaire is used, the methodology for responding to the questionnaire may vary among countries. In the Asia and the Pacific Division (APR), CPMs generally provide initial scores based on detailed written narratives that present qualitative evidence and some quantitative statistics.  An internal economist cross-checks and validates all scores against quantitative indicators in an effort to ensure consistency across countries.  These RSP scores are given a final review by the front office of the Programme Management Department before being finalized. APR’s use of third-party statistics complemented by qualitative evidence has resulted in an average 2009 RSP score of 3.69 which is approximately 0.35 higher than an equivalent 2007 HDI Index score and 2009 IRAI score.    

The limited availability of comparable country data from third-party sources makes the quantitative scoring of the   predominantly qualitative RSP indicators challenging. While comparable national statistical data are available for most of the themes – such as access to natural resources and technology; increasing access to financial services and markets; gender equality in education and political representation; and governance in relation to transparency and corruption – data specifically on rural areas are not.

There is a particular dearth of information on rural organizations, which makes this first theme particularly difficult to score. However, this lack of information also presents an opportunity for IFAD to collect and consolidate comparable data on rural organizations. The validated RSP scores from across the regions could then serve as the definitive international index on rural organizations. By providing comparable country-specific information on rural organizations, IFAD would strengthen its role as a mediator between rural smallholders and the private sector or other potential investors in rural areas or agriculture. 

In addition to the lack of rural data, there is a lack of annually updated comparable statistics. Most third-party statistical data are updated only every three to five years, and some every ten. This makes it difficult to compare countries and annually update RSP scores. Therefore, these statistics, where available, are most useful for setting a baseline every three years, perhaps at the start of a new PBAS cycle or when new data are available. The publication of the data from the 2010 Agricultural Census will provide such an opportunity. 

Finally, for many of IFAD’s member countries, such as the Cook Islands, the Marshall Islands, Kiribati and Niue, very little qualitative or quantitative data are available. IFAD has an opportunity to make a significant contribution to the knowledge on these countries, which could be done in collaboration with other agencies such as the Food and Agriculture Organization of the United Nations, the Asian Development Bank and regional research institutes and networks.

Ironically, the one RSP theme with quantitative indicators for which third-party data are easily available, although not for rural areas, does not have the most relevant rural development indicators. Gender equality is defined as access to education and measured in part by female-male ratio in primary and secondary school enrolment. As a result, most countries receive high scores, making it difficult to use the scores to discern for which countries gender equality is an issue. More relevant indicators of gender equality in rural areas would be women’s access to land and inheritance rights in the legal framework and in terms of number of women with land titles. Maternal mortality and children’s malnutrition are also more relevant statistics as well as being comparable data that are easily accessible. Data such as the number of women agricultural labourers may also be used to inform qualitative narratives on gender equality, although a set level of participation cannot be universally used to indicate greater gender equality. By making the gender equality indicators more relevant to rural development, the RSP score would be a more useful tool for policy dialogue and could serve as the rural equivalent to the Gender Empowerment Measure (GEM) or Social Institutions and Gender Index (SIGI), which are national-level gender empowerment indices.

Given the limited availability of comparable country data, in the short term greater emphasis should be given to the RSPA as a diagnostic tool to identify issues for policy dialogue.  Periodically when new data become available, the RSP scores should be revised and serve as a baseline. These scores can continue to be updated annually to reflect any changes in the country performance regarding these policy issues.  

In the long term, IFAD should use these annual updates and periodic revisions to collect and aggregate comparable statistical data in order to improve the objectivity and comparability of the RSP scores. Doing so would not only fill some data gaps on rural areas, but would also enhance IFAD’s role as a leading player in rural development.

Chitra Deshpande ([email protected]), Operational Policy Economist, Asia and the Pacific Division, IFAD


Upcoming missions and events

Bhutan

Cambodia

China

India

Indonesia

Mongolia

Pakistan

Philippines

Sri Lanka

Tajikistan

Vietnam