4. Thematic Evaluation/Studies

TE/S Objectives:

The purpose of Thematic Evaluation or Studies (TE/S) is to examine IFAD’s experience with specific aspects, themes and processes of IFAD’s operations and policies. These studies are designed to assess the effectiveness of IFAD operational policies, processes, and practices. TE/S synthesise and analyse the Fund’s accumulated experience across countries and regions and draw crosscutting lessons. The objective of TE/S is to provide the building blocks for revisiting or reviewing existing policies or formulate new ones.

TE/S will be undertaken based on findings from project as well as Country Programme Evaluations, supplemented by further investigation, including staff fieldwork and surveys.

New TE/S Process:

In consultation with core partners, we identify and define the expected outcomes for the TE/S. Our approach is practical, applied, and based on the assumption that the degree to which our evaluation outcomes are relevant, they will be used to strengthen operational policies and procedures. For this reason, we consider it fundamental to collaborate with our partnership in defining what they consider the critical outcomes towards which the evaluation process drives.

The new TE/S process is based on the New Evaluation Process (NEP) defined in the preceding pages. Although the steps are very similar to the standard NEP, the process is adapted to the specific requirements of the type of evaluation.

Step #1:

During the formulation of OE’s Annual Work Programme, we collaborated with our core partners to define and prioritize the TE/S for the coming year based on the following criteria.

• The justification and rationale of the demand from our partners;
• The evaluation knowledge and relevant expertise already accumulated by OE on this theme or issue; and.
• The potential contribution to the development or improvement of a particular IFAD operational policy.

Step 2: The Approach Paper ⁹

The EO will consult with others (institutions and knowledge centers) before and/or after the Approach Paper, on a case-by-case basis. The EO drafts, shares and discusses the Approach Paper with selected members of the core learning partnership (see Annex 2).

This Approach Paper will follow the basic OE guidelines and will include:

• The rationale and partnership of the TE/S (including the origin of the demand, the policy development potential and the users of the TE/S’s results).
• The geographic orientation (area of origin and validity of the knowledge to be generated, regional or corporate nature of the experience or strategy to be assessed).
• The "researchable questions", hypothesis, sources of information and knowledge, methodology and expected outcome of the TE/S.
• A clear definition of the "core learning partnership" involved (typically colleagues from the Regional Divisions, PT and PD front office).
• The process, timeframe and resource of the TE/S.

Step 3: Carrying out the TE/S

There are several activities in conducting the evaluation. The administrative requirements can vary widely, as can the type and degree of participatory methods. The typical steps include:

• Mobilizing the TE/S’s CLP and Resource Persons (RP)
• Recruiting consultants/RP
• Stock taking of relevant knowledge in OE and elsewhere
• Conducting field work when required/relevant
• Comparative analysis and draft report writing, including preliminary identification of the "building blocks" for future policy dialogue

Step 4: Consultation

The EO discusses the preliminary results with the CLP and facilitates an agreement on next steps to be undertaken. This group will decide whether to share the evaluation results with a wider audience at this point.

Step 5: Sharing and validation with a wider audience:

If and when the core learning partnership decides to share and validate the results of its work, it may use some of the following methods:

• Conduct a workshop in house with external partners and RP
• Conduct a workshop in the region
• Organize an Email forum, video conference, etc.
• Integrate Lessons Learned into the EKSYST knowledge base (mandatory)
• Subsequently, the CLP will refine the recommendations and lessons learned, and determine the process leading to the completion point.

Step 6: Completion point:

The objective of this step is to reach an agreement/understanding among the decision makers about the "building blocks" produced by the TS/TE and on the follow up measures in terms of policy development and implementation and monitoring arrangements. Depending on the nature and importance of the TE/S, and on the desired level of consensus on the outcome, the completion point could take place in the form of a final meeting with Senior Management, OSC, PD Department or Regional Divisions. It will be summarized in a written agreement/understanding regarding Lessons Learned, Recommendations and follow-up measures required.

5. COUNTRY PROGRAMME EVALUATION

Objectives of the Country Programme Evaluation (CPE)

Focused, results-oriented CPEs are a strategic priority for OE. They are strategic in that CPEs are expected to provide inputs for establishing effective Country Strategic Opportunities Paper (COSOP) frameworks, the importance of which – as OE anticipates – will increase in the future.

Our approach to the CPE is increasingly participatory. In our view, IFAD will move progressively to a COSOP process that includes a broad partnership of all relevant country stakeholders in its formulation. We have tried to design a CPE process that anticipates this kind of engagement by the borrowers and civil societies.

One essential component of any CPE is to analyse and assess IFAD’s experience in a given country, including the most salient features of IFAD’s approaches and selected characteristics of IFAD’s portfolio. This will be done by drawing lessons from all IFAD-financed projects in that country. CPEs are, however, not intended to evaluate projects, closed or on-going, but rather to provide - based on evaluations, supervision work, the Project Implementation Status Report and Country Portfolio Reviews – focussed field work and other data collection, as required, comparative information on the most salient aspects of project performance and to develop strategic and operational orientation for IFAD’s future project pipeline.

The second essential component of any CPE consists of a review of the most recent information, including trends and emerging opportunities and risks affecting the rural poor in the country as well as the rural poverty profile and characteristics.

In conclusion, the main expected outcome of any CPE is a set of agreed-upon inputs in IFAD’s COSOP, including inputs for IFAD’s Policy Dialogue on rural poverty alleviation in a given country. Other expected outcomes are recommendations for the improved implementation of some of the on-going projects, as well as lessons learned at country level.

We have decided to replace the name Country Portfolio Evaluation with Country Programme Evaluation. The word programme incorporates lending and non-lending activities and better reflect the time dimension involved in the evaluation of IFAD’s approach in a particular country.

The Process

The CPE process is an adaptation of the New Evaluation Process and includes the main innovative features we believe strengthen its impact: the Approach Paper, interaction with a core learning partnership, and an agreement among the Partnership at the evaluation completion point.

A shortcoming was that in the past CPEs could take from between one to three years before completion, and involve a sizable amount of resources. Therefore we have designed an instrument that takes less time to complete (eight months) and uses less resources. In addition to that, we intended the CPE process to focus on the production of outcomes that can be used as inputs to the COSOP and be driven by a strong justification to be provided by our partners. Thus, the new CPE should be timely, leaner, focused on selected issues, as well as demand-driven and results-oriented. The new CPE Process engages potential users and implementers of CPE recommendations up-front, to ensure later agreements on the recommendations.

Step #1: Rationale and Justification

The CPE is determined initially during the discussion of OE’s Annual Work Programme with PMD. A planned COSOP by the Regional Division is a necessary but not sufficient criterion. Another criterion is the importance of the country in the Region’s overall portfolio. The availability of recent evaluation works in the country concerned will help determine the scope and intensity of the CPE. The third criterion will be the quality of the statement of justification and expected outcome provided by the regional division. A fourth criterion would be the additional value the CPM is expected to provide over the Country Portfolio Review. A fifth criterion could be the Government’s direct request to undertake a CPE once the existence of this instrument is brought to their attention. Based on these considerations and others we will determine the justification for a CPE jointly with the Project Management Department (PMD).

Step #2: Preparatory Stage:

Identification of CPE issues, focus and scope. This step entails:

• Desk review of documents which are available at IFAD
• Review of recent trends in rural poverty and policy and institutional development in the country
• Review of country assistance strategy, including poverty assessments, economic and sector work, etc. of other donors, and the government’s own poverty alleviation strategy in the rural sector
• Country reconnaissance mission by the Evaluation Officer (EO) to consult with partners and investigate possible partnerships with donors and civil society groups and individual resource persons during the CPE
• Identification of and discussion with core partnerships in PMD and determination of the focus of the CPE

Step #3: Formulating and Sharing the Approach Paper.

The Approach Paper should contain the following elements:

• The rationale and expected outcomes of the CPE.
• The focus and coverage of the CPE relating to the specific sub-sectors, themes, issues and/or well defined operational aspects to be emphasized by the CPE.
• A definition of the Core Learning Partnership involved and its role throughout the CPE.
• Methodology and description of processes involved; approach to fieldwork should be clearly stated, including the role of Community-based Organizations (CBOs), and projects visited should clearly reflect the CPE focus.
• Resource requirements and external inputs in form of consultants and resource persons.
• Time frame for the important stages of the process and the expected delivery date for deliverables (report, summary, understanding at completion point).

The Approach Paper will be finalized after consultation with the Core Learning Partnership.

Step #4: Undertaking the CPE:

In conducting the CPE, the EO is responsible for a variety of activities. Some of these are administrative, while other require the use of a broad range of participatory methods, evaluation tools and facilitation skills.

• Recruitment of consultants
• Mobilization of CLP and local resource persons
• Field work
• Consultation with the partners in the field
• Time for the Mission to reflect and draft initial findings
• Workshop at country level at the end of the fieldwork with the participation of the Country Programme Manager and Cooperating Institution and other members of the CLP. An important aspect of this workshop will be the strong participation by CBOs and civil societies along with government institutions to open up dialogues and identify channels of communications between the concerned parties
• Report writing
• Consultation with CLP in PMD and identification of possible next steps

Step #5: Finalizing Conclusions and Recommendations

• The CLP discusses the preliminary report and defines the process leading to the completion point
• The core learning partnership works out the conclusions, recommendations and lessons learned, possibly through the organization of a round table workshop/conference in the country.

Step #6: Completion Point

There are several levels of agreements/understandings that can be worked out with our partners: the Government, possibly the civil society groups, PMD and OE:

• the inputs to be retained for COSOP and the required follow up;
• elements/inputs to IFAD’s future policy dialogue;
• recommendations for future project design and implementation;
• country-based lessons learned to be added to EKSYST.

As with the other evaluations, the EO may use a variety of forms for documenting these agreements or understandings. It is important that the Partnership complete the process with a clear written understanding of what the evaluation outcomes are, and how they will be applied or adopted by the various members of the partnership.

⁹ The Evaluation Panel was not convened in the majority of cases: roughly only 71 evaluations, or about 24%, of all evaluations conducted by OE were discussed in the In-house Evaluation Panel since its establishment in 1990. The Evaluation Panel was established on an experimental basis for the following purposes : a) review evaluation reports to improve quality, b) share evaluations lessons in-house in order to for them to be included in the design of new projects; and c)  decision-making on their dissemination. The Evaluation Panel also served to: (d) signal the end of each evaluation. OE believes that the learning function (objectives a and b above) of the evaluation panel in the past was limited to a relatively brief interaction among divisions in a rather formal meeting. Clearly the New Evaluation Process in general and the CPL in particular will go much farther than the evaluation panel in promoting learning through evaluations. As to the formal completion objective (d), this is included in the New Evaluation Approach by the new concept of the agreement among the parties at completion point to be prepared by the CPL. Finally, as to the dissemination of the lessons learned, OE agrees that a specific arrangement that goes beyond EKSYST/IFADEVAL is necessary. This is why OE has decided to develop a dissemination and communication strategy for the results of its evaluation work in 2000.