Making learning work

A. Introduction

1. In 1999, IFAD’s Office of Evaluation (OE) developed a new approach to evaluation (NAE) in order to increase the effectiveness of the Fund’s evaluation function. The NAE provides an opportunity for thorough interaction and dialogue among key partners in the evaluation process. It balances the commitment to independent and objective evaluation analysis with the objectives of providing useful recommendations and lessons for performance improvement. The NAE was finalized after discussions with various OE partners, including with IFAD Member States during an informal session on evaluation during the Twenty-Third Session of the Governing Council in February 2000.

2. Having recognized that good communication lies at the heart of effective learning, last year OE engaged in a systematic review with a view to improving evaluation communication practices, in particular as far as communication and dissemination of results is concerned. Based on this work, a new communication approach is being developed and this brief document attempts to capture the principal objectives and instruments envisaged under the proposed approach. OE values the opportunity to discuss the proposal at this stage with IFAD Member States in order to benefit from their views.

B. Why Focus on Communications?

3. For an evaluation to be useful in terms of learning, it is not enough to do a thorough and rigorous evaluation analysis and produce a good report. Until quite recently, however, evaluation reports were seen by most development agencies as being the end of the line. Copies of evaluation reports were circulated, but most ended up sitting on shelves, gathering dust and rarely being consulted. All too often, the lessons they contained were effectively buried, made inaccessible by their terminology and language, and hardly ever reaching the people who could benefit most.

4. Development-agency evaluation units recognize that far more needs to be done to capitalize on the knowledge generated through the evaluation process. In September 1999, the OECD/DAC Working Party on Aid Evaluation held a major workshop on this subject in Tokyo. It focused on evaluation feedback for learning and accountability purposes, but concluded that learning is the "new frontier" where the greatest challenges – and opportunities – exist.¹

C. Communication Challenges

5. To increase the impact of evaluation lessons, we need to face up to the very real communication challenges involved. Prominent among these are:

• the very wide range of potential target audiences for evaluation lessons – within the agency, in partner countries and among the broader development and policy communities;
• the diversity of these audiences – some of whom already suffer from acute information overload, others of whom are starved of relevant information or are limited by language and communication barriers;
• finding the right balance in presenting evaluation lessons so as to avoid being either too specific or too general;
• integrating evaluation lessons with other forms of learning, recognizing that evaluation departments do not have a monopoly on knowledge;
• the barriers to learning that can exist – institutional, cultural and otherwise – which can mean that even when lessons are clearly articulated and disseminated, they are not necessarily picked up or taken seriously;
• taking advantage of the new generation of Internet-based communication tools, without forgetting those who do not yet have access to the Internet.

D. Mapping out a New Approach

6. The rationale for mapping a new communication approach in evaluation was based on the premise that a proactive approach would contribute to advancing OE’s strategic objectives. That is, more systematic and efficient communication would increase learning and result in greater adoption of lessons and recommendations, thus leading to performance improvement in the Fund’s operations and policies. The overall goal of the new approach was summarized as follows: "To achieve good communication during and after evaluations to heighten learning and thus improve the performance of IFAD and its partners". It is necessary to note, however, that the approach focuses primarily on the dissemination of evaluation results. It touches upon, but does not fully address, communication during the implementation of evaluations. To start with, two basic sets of questions were asked, as sketched in the following diagram:

• To whom
• why
• what
• how
• what
• by whom?
• when

7. The process involved a series of steps, including:

• a stakeholder analysis – which identified a total of 23 separate stakeholder groups² who have a role as participants in evaluations, communication intermediaries, learning partners or target audiences of various kinds;
• consultation – with partners in the field and with colleagues in PMD and other relevant IFAD departments – on where improvements could be made;
• a review of OE publications – to assess strengths and weaknesses and recommend improvements;
• improving OE use of Internet facilities – to integrate OE material within the new IFAD corporate website and explore options for a more effective use of the World Wide Web and e-mail;
• a review of EKSYST³ structure and content – to assess the possibility of better ways of capturing and presenting lessons learned;
• comparing practices – with the World Bank, UNDP, and six bilateral agencies⁴; and
• obtaining advice from communication specialists – at the World Bank and the Institute of Development Studies (United Kingdom).

E. Key Elements of the Unfolding Approach

8. The work done so far on developing a new approach to evaluation communication has resulted in the formulation of a clear objective and in the identification of a number of new instruments. However, further work and reflection is required on some of the proposed communication instruments before OE can experiment with their implementation in its evaluation work. A brief overview of some of the possible instruments identified so far is provided below.

• The first innovation involves a customized approach to communication that will be defined in each evaluation from the outset of the exercise. Communication and dissemination will no longer be considered an option. Depending on the nature of the evaluation and the partners involved, a communication strategy will be tailored and built into each evaluation activity. Appropriate planning and budget allocation will be made during preparation of the evaluation’s approach paper, and members of the "core learning partnership" (CLP)⁵ will be consulted in this process as appropriate. Should particularly important findings of widespread relevance for a broader audience emerge at a later stage in the evaluation process, efforts will be devoted to ensuring greater dissemination of the main evaluation messages through customized products. In the year 2000, for example, OE undertook translations into Tamil and Portuguese of the "agreement at completion point"⁶ of evaluations undertaken in India and Mozambique⁷ in order to ensure a wider dissemination of evaluation lessons within the NGO community and civil society at large. Similarly, brochures were prepared on the principal evaluation lessons and recommendations of project evaluations in Ghana, Nepal and Mozambique. Such brochures are more reader-friendly than evaluation reports and reach a wider audience. Finally, in cooperation with the Government of Tamil Nadu in India, OE is preparing training modules for different project partners based on the key lessons learned and recommendations from the evaluation of the Tamil Nadu project. The purpose of these training modules is to increase the operational capacity of the more than 250 NGOs currently involved in the project and improve their provision of training and technical assistance to women’s self-help groups. Similarly, a video documentary is being prepared demonstrating the main messages of the evaluation of the Andhra Pradesh Tribal Development Project. This new medium of communication is being specifically designed to reach illiterate rural people and their communities.
• Another new instrument is a very brief, summary document capturing the key conclusions from each evaluation in a reader-friendly format. These will be 500-700 words and will compress major evaluation conclusions and recommendations. Such documents will be written by professional writers and, for the time being, have been named profiles⁸. Profiles will have a wide audience and be particularly useful to those who simply may not have time to read even the executive summary of a full report, for example, senior civil servants in developing countries and senior management at IFAD. The objective of a profile is to provide a ‘taste’ of the evaluation and an incentive to the reader to deepen his/her understanding by referring to further information in the executive summary of the evaluation or the main report. Other audiences for profiles include partners and stakeholders related indirectly to the project or programme evaluated, such as IFAD staff and consultants, cooperating institutions, other IFAD projects, NGOs and so on. Profiles will be produced in paper and electronic format, in both the original language of the evaluation and in English. Clear signposts will be provided at the end, indicating where further evaluation information can be found.
• A standard package of core products will be developed for each evaluation, so that standards of consistency and quality can be raised. The core products will include the main evaluation report, the executive summary and the agreement at completion point. They will not only have a similar look and format but, more importantly, will have harmonized tables of contents. This will facilitate the aggregation and reporting of evaluation results and impact by theme, as well as comparisons across projects, countries and regions. Other standard products include the profile mentioned above and Insights (see below).
• Creative use of the Internet is another key element in the proposed approach. Considerable work has already been done to integrate OE's existing knowledge resources into the IFAD corporate website. In this context, OE is in the process of revamping and upgrading its computerized knowledgebase, the Evaluation Knowledge System (EKSYST). EKSYST and its Internet version (IFADEVAL) have been merged into a single application, and the new knowledgebase is available on IFAD's Corporate Website under Operations and Activities - Evaluation, Learning and Impact. All evaluation-related documents, including lessons learned, agreements at completion point, executive summaries and other evaluation-related documents are available through the website, in a well-structured and easily accessible format. The creation of the new evaluation sub-site is regarded as an important step in the implementation of the Fund’s disclosure policy, which was approved in May 2000 and mandates that all evaluation-related documents be shared with the public at large.
• The new approach also proposes Insights – designed to focus on one key learning effort and conclusion emerging from thematic or country programme evaluations (CPEs) and, when feasible, from project evaluations as well. Insights will form the principal knowledge element in the new EKSYST and will replace the concept of lessons learned⁹ used in the past by OE. Insights will typically contain the major conclusions, learning and recommendations from a thematic evaluation or CPE. Insights will not have the character of prescriptive lessons. Rather they will primarily serve to direct attention to critical learning hypotheses and form the basis for further discussion among professionals and policy makers at IFAD and outside the institution, project staff, implementing institutions, development consultants and other partners. Another major difference between Insights and lessons learned lies in the process through which they are selected, written and validated. In the past, the selection of topics and writing of lessons learned to be inserted into EKSYST were generally done by OE, thereby largely reflecting the position of evaluation teams and OE. However, through the proposed process in the new approach to evaluation communication, any member of the CLP may suggest suitable topics for the development of an Insight. Following an agreement among members of the CLP on the topics that merit treatment, an Insight is actually prepared through discussions and interactions among the CLP members and other partners. This process will also act as a validation process, since the key partners in the evaluation were actively engaged in and agreed to the final contents of an Insight. Another departure from the old EKSYST and lessons learned is a more systematic and timely approach to the generation of Insights, which will also lead to sounder management and status of the evaluation knowledgebase. In this respect, Insights will be a mandatory output from thematic and country programme evaluation and regularly introduced into EKSYST¹⁰. An Insight will typically be one to two pages.
• Another proposed instrument of particular significance is the establishment in OE of an organized evaluation help desk. The help desk will respond systematically to electronic, phone and written queries. It will retrieve OE material for dispatch and will be particularly oriented to sharing information and knowledge with project partners in the field and other development partners lacking access to Internet facilities. That is, the help desk will not only provide responses electronically, but also make possible opportunities for interactive contact with a human face in order to respond effectively and efficiently to the requests received. The help desk will aim to provide real-time answers.
• The need for standard operating procedures for communication throughout the evaluation process has been recognized as an important dimension in the proposed approach. IFAD’s evaluations are a participatory process unfolding around an independent evaluation analysis. Communication is complex and must be well organized. Standard operating procedures will introduce a logical order and discipline in the communication process among partners and, more importantly, define who should communicate with whom, as well as what should be communicated and when. This is expected to enhance the effectiveness and efficiency of communication in the entire evaluation process.
• One idea still being discussed is a dedicated Internet gateway on evaluation of poverty reduction and rural development. It would provide links to relevant on-line material anywhere in the world, guidance notes, and a focal point for learning and innovation on the subject. As well as being a useful tool for OE staff and partners, it would provide a window for IFAD and help reinforce its reputation as a centre of excellence in rural poverty reduction. Rather than setting up the necessary on-line databases and Internet tools, something that would require substantial investment, the suggestion is to develop the gateway in collaboration with an existing on-line service, such as ELDIS¹¹. This would allow IFAD to concentrate on the substantive content issues, rather than on the technology.

F. Making it all Happen

9. Implementing the new communication approach will require a much more active awareness of communication issues among OE staff and its evaluation partners, and a more-consistent follow through on good practice. Together, the various measures outlined in the new approach represent a significant departure for OE. They signify that OE is raising its sights in terms of the impact that IFAD evaluations have – both within the organization and beyond. Better and more-proactive evaluation communication is also expected to contribute to achieving selected objectives contained in IFAD’s Plan of Action, in particular those related to impact and knowledge management.

10. Any new endeavour, of course, has its financial consequences and these need to be assessed carefully in view of OE’s limited resources. The incremental costs associated with the new approach have been estimated, both in terms of setting up the required systems and implementing the proposed approach. These preliminary estimates show that many of the incremental costs could be accommodated within OE’s operational budget through a reallocation of resources and rationalization of evaluation activities and processes. In addition, OE is working to develop a streamlined evaluation methodology and this, too, is expected to generate savings that could be used to introduce some of the instruments for enhanced evaluation communication. OE is also looking forward to developing partnerships in this area with other development agencies, not only with a view to mobilizing additional financial resources, but also to promoting cross-fertilization of ideas, experience and approaches. Such partnerships are expected to foster intellectual contributions among development-agency evaluation units as well as exchange of experiences and knowledge in the area of evaluation feedback.

11. The proposals contained in this paper are both ambitious and forward-looking. They should be seen as work in progress rather than end results, since OE recognizes that it is still learning how best to communicate evaluation knowledge in order to stimulate learning and thereby contribute to poverty reduction. OE would welcome the observations of IFAD Member States on this topic.

---

1/ The workshop was entitled Evaluation Feedback for Effective Learning and Accountability. The draft synthesis report is available at www.ids.ac.uk/efela. The proceedings will be published shortly by OECD.

2/ Including: (a) primary stakeholders, who are ultimately affected by the communication approach (such as OE staff, project staff, operational divisions at IFAD, implementing agencies, governments, IFAD management and others; and (b) secondary stakeholders, who are intermediaries in the process of implementing the communication approach (such as staff in other IFAD divisions and others).

3/ EKSYST is the OE knowledgebase, containing lessons learned and other evaluation-related documents, and which was developed in 1995.

4/ Danish International Development Assistance (DANIDA), Department for International Development (DFID) (United Kingdom), German Agency for Technical Cooperation (GTZ), The Netherlands, Swiss Agency for Development and Cooperation (SDC) and Swedish International Development Cooperation Agency (SIDA).

5/ A CLP is formed for each evaluation, composed of representatives of the main evaluation partners, who commit to steering the evaluation process from inception through to the end, under the overall guidance of OE.

6/ The agreement at completion point includes the key lessons learned, recommendations and follow-up actions that evaluation partners have agreed upon and are committed to implementing.

7/ Tamil Nadu Women's Development Project (India) and Nampula Artisanal Fisheries Project (Mozambique).

8/ These will be prepared along lines similar to the World Bank's Fast Track Brief (FTB). FTBs are primarily intended for the World Bank President and other senior managers in the Bank, and serve to provide early warning signals about major issues emerging from evaluation that may require immediate attention. FTBs are re-edited and disclosed to the general public at a later stage in the evaluation process. Depending on the requirements, profiles may also serve this purpose and be used as an instrument to raise issues before the end of the evaluation process.

9/ A 'lesson learned', in the context of evaluations, can be defined as a generalization based on an experience (e.g., project, policy or programme) that was evaluated.

10/ Insights may also be generated from project evaluations in selective cases.

11/ ELDIS is one of the leading on-line information sources on development and is hosted by the Institute of Development Studies. A similar gateway, building on ELDIS, has been created for the CGAP microfinance consortium (www.ids.ac.uk/cgap).

 

Hot links Evaluation policy Evaluation profiles Evaluation insights