Enabling poor rural people
to overcome poverty
Part Two: Implementation Procedures and Arrangements
These procedures and arrangements span the entire evaluation cycle from the formulation of the OE work programme and budget to the finalization and disclosure of evaluation reports. They are the means by which the policy framework described in Part One of this document is implemented in practice.
I. Annual Work Programming and Budgeting
A. Work Programme and Budget Formulation Process
Each year, OE, while retaining its final authority to decide on the content of its annual work programme, will register the interest of its partners and prepare a two-year rolling work programme for independent evaluation. This work programme will be based on the selection of a critical mass of evaluations that, according to OE, is required for promoting accountability and learning in IFAD as well as for the preparation of the annual report on the results and impact of IFAD operations. Every work programme will include a mix of different types of evaluation.
The OE budget builds on the annual work programme and will be divided into two basic categories: (i) staff; and (ii) evaluation work. The staff budget covers regular and fixed-term staff costs. The evaluation work budget category contains items reflecting the major priority areas of OE’s evaluation work, namely project evaluations, country programme evaluations, thematic and corporate-level evaluations, etc.1
B. Work Programme and Budget Approval Process
The OE Director will formulate the annual OE work programme and budget independently of the management and transmit it to the President, who will submit it unchanged to the Executive Board and Governing Council for approval.
The OE annual work programme and budget will be presented together, but as a separate submission, with IFAD’s annual work programme and budget to the Executive Board for approval, and to the following Governing Council meeting.
The President will convey to the Director of OE without change any changes requested by the Board to the OE work programme and budget. The Director of OE will then resubmit the work programme and budget to the Board via the President, as described above.
The Governing Council will be requested to delegate to the Board the authority to amend or supplement OE’s work programme and budget by separate decision during the year. The levels of the OE component and the remainder of IFAD’s budget will be determined independently of each other.
II. Devising the Evaluation Approach
For every evaluation, OE currently prepares an approach paper, which is the first step in the evaluation process. This document has a standard format covering the following aspects of the proposed evaluation:
(i) background and rationale;
(ii) objectives;
(iii) expected focus and outcome, key questions and methodology;
(iv) CLP and the other partners involved2;
(v) process and workplan;
(vi) human resource requirements for the evaluation; and
(vii) communication and dissemination of results.
As in the past, the approach paper will be shared for comments with all the parties involved in the evaluation. This makes the evaluation process transparent to stakeholders and helps coordinate their inputs and participation according to a realistic timetable.
While preparing the approach paper, OE will identify, as it does now, members of the CLP, which consists of the main users of evaluation. At the beginning of the process, the CLP helps flag issues and information sources for the evaluation. After the completion of the independent evaluation report, the CLP discusses the evaluation findings, deepens the understanding of the findings and recommendations, and eventually works out the operational implications of evaluation recommendations and the division of labour and responsibilities for their implementation among the various stakeholders involved. The CLP is assigned this role because evaluation reports by their very nature often cannot make clear-cut recommendations that can immediately be adopted and implemented. The CLP’s output is recorded in an understanding or agreement at completion point (ACP) among the stakeholders involved.3
III. The Evaluation Analysis and Report
A. Conducting the Evaluation Analysis
The overall responsibility for the conduct of the evaluation analysis rests exclusively with OE. As in the past, however, OE will engage relevant IFAD officials and stakeholders at appropriate stages of the evaluation process, taking into account the role of the partners concerned.
IFAD management will ensure that IFAD officials and IFAD-assisted projects promptly provide all documents and other information required by OE and participate and cooperate actively in the evaluation process.
Again as in the past, before initiating an independent analysis, OE will invite the implementers3 and the beneficiaries concerned to provide a self-assessment. This is followed by OE’s independent analysis based on internationally accepted evaluation criteria, and a methodology aimed at promoting accountability through impact and performance assessment. This analysis will continue to be grounded in extensive fieldwork and a review of all information made available by relevant stakeholders.
Normally consultants recruited by OE to undertake evaluation work will carry out the field evaluation mission. This fieldwork guarantees the quality and impartiality of the information on which the evaluation analysis and report are based. It is OE’s main instrument for fact-finding, data and perceptions gathering, triangulation and validation. As a rule, OE evaluation missions are carried out with the participation of in-country stakeholders, in particular the rural people involved in IFAD-assisted projects, the project management units and NGOs involved in project or programme implementation. The evaluation mission contributes to strengthening the position of the rural poor in their interaction with implementing agencies, governments and IFAD itself, through intensive works at community level and evaluation workshops with all stakeholders. It is the main instrument to enable the rural poor and their partners to participate in the evaluation learning process with IFAD, and to enable IFAD to learn from them.
In continuation of the prevailing practice, the evaluation mission will present and discuss its preliminary findings and conclusions at meetings with all evaluation partners. This interaction allows the mission to provide feedback to all partners, while in turn giving them an opportunity to provide additional information and insights that can be used in the draft evaluation report, for which OE remains solely responsible.
The evaluation team comprising consultants recruited by OE to conduct the evaluation will prepare the report, which will consist of the executive summary, main text and working documents as annexes, if necessary. The evaluation team will work under the supervision of a lead evaluator, assigned by the OE Director to manage the evaluation process. The lead evaluator will be responsible, as now, for managing the evaluation process and ensuring the quality and content of the evaluation report, which should be short and user-friendly.
OE will use peer review from within the Division to ensure quality standards. When undertaking a complex evaluation, it may also engage an ad hoc advisory committee to provide expert advice and feedback.
Before the report is issued, OE will share it with IFAD management and, whenever applicable, with the concerned borrowing country’s authorities, the implementing agencies and the cooperating institution in order to check facts and accuracy and obtain comments.
OE will decide which comments should be incorporated in the revised (final) report. As a general rule:
(i) The draft report is revised to incorporate comments that correct factual errors or inaccuracies.
(ii) It may also incorporate, by means of a note in the report, judgements that differ from those of the evaluation team.
(iii) Comments not incorporated in the final evaluation report can be provided separately and included as an appendix to the report.
The OE Director will have the authority to issue final evaluation reports, including the ACP directly and simultaneously to the Executive Board and the President without prior clearance from anyone outside OE.
IFAD management may receive, comment on and respond to the draft and final evaluation reports, but the President and other members of IFAD management will not have the right to approve, hold back, request changes to or otherwise modify such draft or final evaluation reports.
IV. Learning with Partners to Operationalize the Recommendations
As per current practice, upon completion of each independent evaluation report by OE, OE and relevant IFAD officials and other stakeholders will develop a separate action-oriented document, called the understanding or agreement at completion point. The ACP is the end point of a process that aims to determine how well evaluation users understand the recommendations proposed in the independent evaluation, and how they propose to make them operational. Interaction among the stakeholders working through the CLP helps deepen the understanding of evaluation findings and recommendations contained in the independent evaluation report, and elicits ownership for implementing the recommendations. The ACP illustrates the stakeholders’ understanding of the evaluation, findings and recommendations, their proposal to implement them and their commitment to act upon them. OE will participate in this process to ensure a full understanding of its findings and recommendations.
The ACP will continue to be the outcome of the work of the CLP. The two objectives of the ACP are to: (i) clarify and deepen the understanding of evaluation recommendations, document those that are found acceptable and feasible and those that are not, make the former more operational, and eventually generate a response by the stakeholders on how they intend to act upon them within the framework of an action plan that assigns responsibilities and deadlines; and (ii) flag evaluation insights and learning hypotheses for further future discussions and debate.
The ACP will make explicit reference to the partners with whom it was concluded. These include all major users of evaluation results such as the relevant IFAD operational unit(s), project and borrower country authorities and other relevant stakeholders. OE’s participation in the ACP process will be as explained above.
V. Reporting, Follow-up, Disclosure and Dissemination
A. Reporting and Follow-Up at Management Level
The OE Director will convey completed evaluation reports including the ACP and other evaluation documents, such as the annual report on the results and impact of IFAD operations and the annual OE work programme, simultaneously to the Executive Board of IFAD, the President and, whenever applicable, the concerned borrowing country’s authorities, the implementing agencies and cooperating institutions.
The President will be responsible for ensuring that evaluation recommendations found feasible by the users are adopted at the operational, strategic and policy levels (as appropriate) and their implementation adequately tracked. The President will provide the Board an annual report on the status of adoption and implementation of evaluation recommendations and OE will provide to the Board its independent comments on this report, including an inventory of recommendations not found feasible by the users, hence not implemented.
B. Reporting to the Executive Board and the Evaluation Committee
All evaluation reports will be submitted to the Executive Board at the same time as they are forwarded to the President of IFAD. The reports will be issued in the original language with English translation of the executive summary and the ACP. A translation of all evaluation reports into all official languages could be considered upon verification of the cost involved relative to the benefits associated with such practice.
Every year OE will also submit to the Executive Board an annual report on the results and impact of IFAD operations in its September session. This report will present a consolidated picture of results and impact achievement, and a summary of cross-cutting issues and learning insights on the basis of the project evaluations undertaken during the reporting year.
As is currently the case, the Evaluation Committee will select from OE’s work programme a number of evaluation reports to review and discuss at its three regular sessions during the year or at additional informal sessions.
The Evaluation Committee will also continue to provide feedback to OE and report to the Executive Board on specific evaluation issues. The outcomes of each Evaluation Committee meeting will be summarized in official minutes. The Committee will report to the Board on its deliberations following each and every Evaluation Committee session.
C. Disclosure and Dissemination to the Public
OE will continue to produce evaluation summaries, called “Profiles”, that provide an overview of the main evaluation conclusions and recommendations, and “Insights” that contain one learning theme from the evaluation and serve to stimulate discussion among practitioners and other development specialists on some important issues.
As in the past, OE will ensure that all evaluation reports including the ACP and Profiles and Insights are disclosed to the public at the completion of the evaluation process and disseminated widely through the print and electronic media in accordance with IFAD’s disclosure policy.
The President will nominate a candidate for the position of Director of OE to the Board for endorsement, as recorded in the Executive Board minutes, whereupon the President will appoint the Director for a fixed term of five years, which may be renewed only once.
Similarly, the President will remove the OE Director upon and only upon the endorsement of the Board, as recorded in the Executive Board minutes.
The OE Director will not be re-employed by IFAD upon completion of his or her term(s).
The OE Director will be directly responsible to the Executive Board4.
B. OE Staff and Evaluation Consultants
The President will delegate authority to make all personnel and operational decisions concerning OE staff and consultants to the OE Director, in accordance with IFAD rules and procedures covering human resources. Within these rules and procedures, the Director will have authority for managing OE personnel, their workplans and the demands on their time.
The OE Director will ensure that OE is staffed by independent-minded, experienced and sufficiently senior evaluators.
As per current practice, OE will make certain that the engagement of any individual in an evaluation exercise will not generate a conflict of interest. In particular, an evaluation will not be entrusted to an OE staff member who has been responsible in the past for the design, implementation and supervision of the project, programme or policy to be evaluated.
A consultant who has worked previously on the design or implementation of a project, programme or policy may be engaged as a resource person for providing information to the evaluation team but not as a consultant entrusted with the conduct of the evaluation analysis and the preparation of the evaluation report.
OE staff other than the OE Director, will be entitled to seek employment in other units of IFAD. IFAD management will treat OE staff who apply for positions outside OE as other IFAD staff, and in accordance with IFAD personnel policies and procedures.
1/ These different types of evaluation are described in Annex III.
2/ While the composition of the CLP depends on the nature of the evaluation and the stakeholders involved, the CLP typically consists of representatives of the Programme Management Department, the borrower, the implementing agency, the cooperating institution, NGOs involved in the project’s implementation and, where feasible, organizations representing the rural poor, in adition to OE as a facilitator.
3/ Project authorities and other agencies involved in the implementation of the project.
4/ The President’s non-voting participation in the Board as its chairperson will not imply in any way the exercise of supervisory or other authority or responsibility by the President, in his or her capacity as chairperson or any other, over the OE Director.