Section 8 - Reflecting Critically to Improve Action

8.1 An Overview of Reflecting Critically to Improve Actions

8.1.1 What Is Critical Reflection?

Critical reflection in a project means interpreting experiences and data to create new insights and agreement on actions. Without critical reflection, your M&E data will not help you to manage for impact (see Section 2). Active discussion during team meetings and in meetings with primary stakeholders are vital if M&E information is to be shared, analysed and acted upon. Making analysis "critical" means moving beyond collecting, processing and reviewing data. After asking "what is happening", also discuss:

"Why is it happening?"
"So what are the implications for the project?"
"Now what do we do next?"

If you manage to discuss these questions regularly with project stakeholders, then you are well on the road to reflecting critically. Reflecting critically means questioning what is normally taken for granted, particularly project assumptions (column 4 of the logframe matrix, Section 3). This requires reflecting on what did not work or is not working. The project objective hierarchy is based on assumptions about the context and the likely effect of project activities. Only time will tell if these assumptions are valid or not as it will become clear what has and has not worked. Learning from what did not have the desired effect enables you to adjust your mental model of how the project works and work with more valid assumptions.

8.1.2 The Importance of Critical Reflection for M&E

Normally, relatively little attention is invested by projects in critical reflection. Most M&E efforts and resources are spent on collecting data, statistical analysis and basic reporting of activities. This has several causes (also see Section 1). Data collection is more mechanical and requires less capacity. M&E is often seen as a bureaucratic obligation requested in the contract and by government regulations and not as valuable to the project. No one seems to think they gain from frank analysis of under-performing projects. Reversing such views requires building capacity, sensitisation and putting in place effective incentives (see Section 7).

The following dialogue shows how embedded a non-reflective approach can be in M&E. The dialogue took place in Bangladesh between an M&E consultant (C) and a sector coordinator (X):¹

C: "What purpose do the weekly reports serve that you request of the district coordinators?"

X: "It is policy that they must send reports weekly."

C: "Yes but what purpose is the policy serving?"

X: "To encourage them to work harder and to inform higher management."

C: "Higher management does not have the time to read weekly reports and the district coordinators are being paid to work hard. Why do you ask them to give you weekly reports?"

X: "When they know that they are being watched, they will pressure their thana managers to work harder."

C: "They are already pressuring their staff hard. You can find the results in the monthly reports. Why do you need weekly reports?"

X: "To inform higher management."

Focusing on producing data because someone wants it rather than analysing it critically in order to learn from it defeats the purpose of M&E. Project stakeholders can only improve their actions by reflecting regularly on data, planning moments for such reflection and taking time to learn lessons. Decisions need to be documented and shared with those concerned. Managers need to check that decisions have been implemented as agreed. Then you can say that learning has taken place.

8.1.3 Reflective Events in a Learning Sequence

Project M&E involves an extensive series of potentially reflective events � from weekly team meetings and informal sessions to the more formal supervision missions and mid-term reviews/evaluations. These events, whether self-organised or externally initiated, occur alongside data gathering. During these events, project stakeholders can use the data gathered to indicate areas of improvement.

Learning does not happen in one sitting. It evolves, starting with individuals raising important issues and questioning assumptions through group-based analyses (see Box 8-1) that bring out different perspectives and information inputs. So you will need to plan "learning" as a series of events. Knowing how to structure the sequence is important (see Box 8-2). The optimal sequence of learning events follows reporting lines and hierarchies of decision making.

Box 8-1. The advantage of critical reflection in groups

To uncover new information � by sharing ideas with others, individuals� memories can be triggered and new information and more refined insights can emerge.
To limit biases � a thorough and critical discussion about information (impressions and data) means it is crosschecked and people can point out when they feel an issue has been represented incorrectly.
To build a clear picture of a situation/event/process and reach consensus � by discussing data, contradictions and gaps can be identified and can be understood or filled.
To ensure well-reasoned, meaningful actions � joint analysis of, for example, the number of people experiencing food shortages, can reveal the structural causes of problems and solutions and so lead to more focused project reactions.
To facilitate action that has broad ownership � the more people who understand the causes and extent of problems and how they relate to aspirations, the more this can motivate people to invest in making changes happen.

But do not rely only on groups and be sure to structure them well. Groups can also inhibit sharing. Compose groups in such a way that sharing is possible for those who are shy and relatively marginalised.

Box 8-2. Linking learning in Colombia

In the micro-enterprise focused PADEMER project (Colombia), M&E staff started monitoring the work of implementing partners by visiting primary stakeholders, who are all local businessmen and women. This gave project staff much better insights. Initially, these visits were made without a checklist of questions or clear methodology. M&E staff arrived and did not know what to ask or how. Soon a plan for these field visits was established. Currently, three visits are undertaken per year. In the first, primary stakeholders are acquainted with PADEMER, information is given on available resources for local micro-enterprises, and a local committee is formed to work together on project monitoring.

The approach for the participatory evaluation workshop has been adapted to fit in with these monitoring visits. First, the workshop is held with primary stakeholders and the Corporation for Small Business Development (a kind of coordinating organisation) to analyse achievements described by the implementing partner in progress reports. The implementing partner is not present. This workshop verifies progress in every sense (social, productive process, marketing, etc.) and how satisfied primary stakeholders are with the implementing partner.

Then a second workshop is held with the implementing partner (and with the corporation but without primary stakeholders) to analyse the work based on first workshop outputs. Progress is assessed, strategies analysed and lessons identified. In the third step, project management analyses project performance based on information coming from the previous two workshops. An assessment is made of the feasibility and suitability of continuing with the implementing partner in a second phase.

This build-up of insights by consulting different project stakeholders in a structured sequence has allowed information crosschecking and also decision making on what each party needs to adjust.

As project manager or M&E officer, you will no doubt find that project stakeholders have different levels of ability and willingness to reflect critically on their practice. You cannot force people to engage in a change-oriented learning process but you can put in place some simple opportunities that encourage it. Sections 8.2 and 8.3 discusses concrete ideas for stimulating individual reflection and reflective events, while 7.3 offers ideas on putting in place incentives.

An important moment in the learning sequences is when lessons are identified. Project and partner staff are continually learning, sometimes unconsciously, what to improve and changing their everyday actions accordingly. Sometimes it is useful to systematise this learning in the form of "lessons learned". A lesson learned can have an internal audience � the project and partners themselves � as well as an external audience consisting of other projects, other funding agencies, IFAD and so forth. Their value for projects that are managing for impact is outlined in Box 8-3.

Lessons for internal learning are particularly important when a project is innovative or for external learning after several years of implementation. PADEMER, in Colombia, is an example of an experimental project. Its appraisal report explicitly states the role of lessons learned: "Given the project�s pilot nature in terms of experience, the evaluation function will place special importance on reviewing lessons of implementation, when informing the expansion of activities into new areas." For TNWDP (India), lessons were identified in its completion report and they focused on nine themes: an enabling environment, group formation and cohesion, NGOs, targeting, training, income-generating activities, marketing, financial operations and executing agency.

Section 8.2.2 describes how to undertake a "lessons learned" exercise.

Box 8-3. The value of sharing lessons learned ²

Present successful alternative development models, for planning and replication purposes, that have been analysed well and documented and are based on practical field experiences.
Facilitate others in learning from our mistakes, thereby helping them to avoid making similar errors.
Permit others to learn from the problems that were encountered in the project � and how they were solved.
Increase the project�s impact by positively influencing organisational policies and the design and implementation of other projects. For example, IFAD project proposals include a section "Lessons Learned from Previous IFAD Experiences".
Promote networking by exchanging knowledge and information, thereby increasing cooperation among different organisations.

8.2 How to Encourage Critical Reflection

8.2.1 Starting with Individual Reflection

Learning starts with the individual. One critically reflective and innovative staff member can make a considerable difference in a project (see Box 8-4). Your management style will strongly determine whether or not project staff and staff from implementing partners and primary stakeholders will feel free to initiate and participate in similar types of learning exercises.

If individuals do not reflect during their work on their own, then they will probably find it difficult during group events, such as annual project reviews or monthly meetings with implementing partners. While not everyone is equally capable or interested in developing a reflective working style, everyone can start somewhere.

Box 8-4. Facilitated self-evaluation of cooperatives in Morocco

Lubna Khalil works with the SPA (Service de la Production Agriculture) in Morocco. On her own, without support from project management, she initiated a process of organisational self-evaluation with nine of the cooperatives in one area. The process started by bringing together the administrative staff of the cooperatives to discuss some issues they are facing in their work. The first meeting focused on problems of financial accountability and the tenure of the administrative staff.

The meetings offered an opportunity for administrative staff to analyse problems and offer solutions based on their understanding of the local situations. Lubna provided facilitation support as well as technical assistance in areas where the staff did not feel comfortable, such as designing monitoring forms and formats for cooperative activities like vehicle maintenance and supervision. The meetings between administrative staff of the different cooperatives have become regular. The meetings are also held when a particular problem arises that requires more time and effort.

Although the process has just started, there are already some visible positive impacts on organisational effectiveness and efficiency:

Purchase of computers to assist in financial and other types of accountability
Training of cooperative administrative staff in the Moroccan laws that govern cooperatives
Training of cooperative administrative staff in financial bookkeeping, resulting in improved financial accountability
Improvements in the staff contracts to improve protection of both the staff and the cooperatives
Restoring or creating a monitoring system for management, e.g., a monthly check on vehicle status and finances

Simple practical steps can make a significant difference in encouraging individual reflection in project staff and in staff of partner organisations.

Including the expectation of review and reflection in job descriptions and TOR and memoranda of understanding. You can include a clause such as the following in a job description: "The postholder is expected to review regularly (at least once a year) his/her performance with the direct manager, with the intention of identifying areas for improvement." In a memorandum of understanding with an implementing partner or primary stakeholder group, stipulate that not only will raw data be reported, but also the implications of the information for action are to be given. Make explicit the expectation that partner staff participate in regular workshops to assess implications for action.
Encouraging reporting that asks staff for their opinions. In the TROPISEC project, Nicaragua, the implementing partner has introduced three monitoring tools: the community promoter�s notebook, the notebook of the assistant soil conservationist and the extension agent�s agenda. Staff elaborate their monthly reports directly in the notebook or agenda. They note down field activities developed, primary stakeholders assisted, achievements, problems and proposed solutions.
Regularly asking project stakeholders their views. This means not just at formal meetings but also in chance encounters to stimulate people to form and share opinions. Ask what surprises they have encountered, what innovations they have tried out and what changes they would like to see in the project. In TROPISEC, Nicaragua, it is a requirement that "each technician or professional of the [management unit] must have as an annual purpose the capitalisation of at least one experience to exchange with other colleagues".
Providing constructive feedback. If you are managing others, give positive reinforcement about what you like and constructive criticism about what could be improved (see Box 8-5).

Box 8-5. Formulating constructive feedback

Specific rather than general: "You missed this morning�s meeting," rather than, "You never bother to come to our meetings."
Tentative rather than absolute: "You seem unconcerned about this problem," rather than, "You don't care what happens." It also helps to present negative feedback as something that is your own problem, rather than their fault ("I felt upset when you�").
Informing rather than commanding: "I haven't finished yet," rather than, "Stop interrupting me."
Suggesting rather than directing: "Have you ever considered talking to him directly about this?" rather than, "Go talk to him."
Tied to behaviour rather than abstract: "You complain frequently," rather than, "You are immature."
Timely, since looking back to something that happened several days or weeks ago is difficult for people.

8.2.2 Capturing Lessons Learned with Project Stakeholders

Learning needs to be systematised. Accidental learning happens all the time but is not the most efficient way to learn nor does it necessarily lead to improved actions. Increasing successes and avoiding pitfalls is best when conscious efforts are made to learn lessons.

A lesson learned is defined as "the knowledge derived from experience that is sufficiently well founded and can be generalised so that it has the potential to improve action". Many projects and organisations talk about the importance of identifying such lessons. But in practice, few are able to identify lessons that actually help improve actions. Two examples of lessons from IFAD-supported projects illustrate how to include an action orientation (see Box 8-6).

Box 8-6. Lessons learned in Colombia and India

In Colombia�s PADEMER project, one of the lessons learned was that it is good practice when contracting out services to spell out the exact commitments and responsibilities of each actor, stipulating the information that should be reported in terms of effects and impacts, and establishing clear, concrete mechanisms to insist on this. This was not done with the implementing partner that had been contracted to undertake monitoring. It only reported on achievement of activities, leaving the project empty-handed in terms of information on impact.
In the TNWDP project in India, an important lesson learned about NGOs was: "The capacity and performance of NGOs is critical to the continuing success of self-help groups, especially in the early stages of group formation." A related lesson that could ensure the performance of NGOs was: "the importance of rigorous screening procedures to select committed and capable NGOs that are driven primarily by the desire to improve the well-being of people at the grassroots level".

How to distil lessons learned

"Lessons learned" events are not daily occurrences. The frequency with which lessons are identified will be less than weekly or monthly team meetings. In some projects, the team and partners identify lessons at annual project reviews, at mid-term reviews and again at completion. If you want to have an active learning organisation, then including lessons as part of the annual review is a good idea. Waiting until the end of the project will mean wasting many potential learning opportunities.

A lessons-learned event can take several days of intense discussion and must be prepared well. In the planning, answer these six questions with other key stakeholders before you start:

What do we mean by a "lesson learned"?
Why do we want to identify lessons?

For reporting (to primary stakeholders, partners, to funding agencies)?
To learn for ourselves for a next phase of the work?
To deal with a crisis?
As a strategy to support fundraising (by being able to report on lessons)?

For whom are these lessons and, accordingly, how are they best shared (written, verbal, video, drama, etc.)?
Whose lessons are they � primary stakeholders, field workers, management? And, therefore, who should we involved in identifying the lessons?
What are the lessons � per stakeholder group (source and audience)? And how do we prioritise them if there are too many to share (see Box 8-7)? To narrow the discussion around key lessons, first decide on the themes or issues where most learning occurred or is needed.
How do we document the lessons and how do we link the lessons into the next phase of planning?

Box 8-7. Prior selection of your themes for lessons-learned events

In an IUCN (World Conservation Union) integrated natural resource management project in Tanzania, a lessons-learned event after phase one and prior to phase two focused on eight key themes:

Participatory resource assessments and socio-economic profiles
Supporting community resource management plans
Making participation of key stakeholders possible
Building awareness, capacity and commitment
Community development fund
Integration of gender considerations
Decentralisation and coordination
Law enforcement

These were the main areas of methodological experimentation and innovation for the project and thus where it most needed to stop, reflect and learn in order to implement an improved phase two.

During the workshop, participants listed hundreds of potential lessons learned. To keep it manageable, they clustered lessons into three categories: critical for success, important for success or desirable (under the headings "Must Do", "Should Do", and "Could Do"). Only the "MUST DOs" were documented. Part of this entailed a discussion on which lessons were specific to the project area or only to Tanzania and which would be widely applicable.

Formulating a lesson learned

To formulate a useful lesson, consider the following.

Include a generalised principle that can be applied in other situations. Do not write the lesson only as an observation, description or a recommendation that lacks justification.
Explain the lesson in the context of the project. For it to be useful to others, they need to understand the situation in which it occurred. Otherwise, they will not know if the lesson might be appropriate for them or useful. Relating the lesson to one (or more) assumption(s) on which the project is based can help others understand what exactly has been learned.
Justify the lesson with proof of why it is valid. But if it is a hypothetical lesson, test it. Do not rely on it without ensuring it is valid.
Check that the lesson is not too general or too specific to be useful.

Document the "lesson learned" with at least these five elements.

Theme of "lessons learned"

The core question that the project asked itself due to a methodological innovation or problem encountered, or because it is a key theme of the project.

What was our original understanding or assumption?

A short description of the original understanding of the theme/question. This is what people assumed before the experiences on which they are now reflecting. For example, "We assumed, incorrectly, that there would enough women able to participate in the micro-enterprise training workshops."

What is our revised understanding or assumption?

The new thinking about the original question/problem or the reworded assumption. Taking the example above, "We now know that we not only need to create interest in micro-enterprise training but also need to find ways to make it possible for women to attend the course."

One or two examples that substantiate the new understanding

To be sure you have a high-quality lesson, you need to provide evidence that supports the proposed lesson learned. Only with multiple sources of proof will it be solid enough to apply in future. The greater the number of sources for a "lesson learned", the more rigorous the supporting evidence.³ The more the evidence supports the lesson learned, the more confidence you can have in the lesson�s significance and meaningfulness. If you have only one type of evidence then you have a "lessons-learned hypothesis". This will need to be tested and verified.

How the project came to this insight
Description of what triggered the project team to be challenged in its views and revise its understanding (Was it a crisis, monitoring data, a field observation that contradicted earlier ones, etc?).

8.2.3 Planning for an Integrated Sequence of Reflective Events

All projects have several routes for learning and decision making. These need to be interlinked if they are to be complementary and not duplicate efforts. In the example from Tanzania, in Box 8-8, each step in the sequence of meetings has its own mandate in terms of types of information shared and discussed and the types of action-decisions that it can make.

Box 8-8. Meetings at the appropriate level for useful action

In Tanzania, each of the levels of water users is involved in regular meetings to discuss and solve problems as they arise. The discussion of problems starts at the (irrigation) section level, with problems beyond the capacity of the section being sent to the block meeting and those beyond the capacity of the block to solve are sent to the association. The section meetings are informal, although meeting minutes are taken. The advantages of regular (normally monthly) section meetings noted by the Bahi Irrigation Scheme are:

They are able to solve problems before they become too difficult to solve. For example, a damaged canal area can be repaired or replaced before the irrigation water causes more damage, requiring more money and technical expertise to repair.
The smaller numbers are easier to manage and administer. With 12-15 people from the same area, it is easy to call a meeting and notify members. If the meetings were too big, then they would be less effective. Sometimes the meetings even take place in the field so local members do not have to travel far to meet.
Because the section is the nucleus, then even the demonstrations and trials are easy to establish so that others can learn. People can move easily to the demonstrations.

Knowing how to construct a useful learning sequence is a skill that must be learned. This may require some trial, error and innovation. The optimal sequence of learning events follows levels of aggregation (see Box 8-9), reporting lines and hierarchies of decision making. Start a sequence at the lowest possible level where decisions are made (Box 8-10). If a decision must be made elsewhere or discussed with others, then link this into a sequence by feeding any lessons or conclusions from one discussion and decision forum to the other.

Sometimes you will need to include special evaluations in a learning sequence in order to fill a knowledge gap. For example, the WUPAP programme (Nepal) planned two human rights assessments as part of its M&E process. These will look at the extent to which the programme has promoted awareness of rights, advocacy, and how to improve the human rights situation for the next phase.

If you have a sequence of learning events in which conclusions are documented and passed to another decision body, provide feedback on any conclusions. If, for example, project management receives progress reports from implementing partners, provide them feedback on the quality of the reports and on how the information has been used. This is important not only to let partners know how information has helped influence decisions, but also to redirect them in what they include in reports. It can help move them away from giving lists of activities to reporting on results, problems and proposed solutions.

Box 8-9. Linking in Bangladesh, ADIP planning for an integrated sequence of reflection and reporting events

The reporting system of the ADIP project in Bangladesh functions well and reasonably meets the information needs of stakeholders: the implementing partners, NGOs, project management and IFAD. Monitoring and reporting is carried out relatively systematically through:

quarterly data collection, compilation, consolidation and summary reporting;
half-yearly and yearly data collection, compilation, consolidation and detailed reporting;
special report on any issue of concern, at any time.

In addition, brief monthly reporting is being planned. The main flows of reporting were:

data collected at the group or field level were forwarded to the municipal level for compilation and consolidation;
data compiled at the municipal level were forwarded to the district level for consolidation and onward transmission;
data compiled at the district level were forwarded to HQ (one copy to the concerned agency and one to management) for project consolidation and feedback.

Along with this, a feedback loop incorporated the following workshops and review meetings:

review meeting with NGOs, bi-monthly;
project management coordination committee meetings, bi-monthly;
inter-ministerial project steering committee meetings, half-yearly;
special review on extension activities, annually;
supervision missions by IFAD for reviewing, including mid-term review, project performance with recommendations approved at feedback sessions and follow-up at consecutive supervision meetings.

Box 8-10. Tiered learning in Tanzania

In Tanzania�s PIDP project meetings are held at different levels of the irrigation system. Besides group meetings at the section level, there are "block meetings" and "association meetings".

The section leaders bring progress information and problems that they cannot solve to the block meetings. These meetings are more formal and happen after the section meetings have been held (especially during the annual planning process). The block leaders oversee which sections are doing well or not. If the sections are not doing well, then they receive a warning. If the warning is not heeded, then the block leaders can forward the problem to the association level, which can fine or otherwise punish the section/individual according to the association�s by-laws. The block meetings are small so it is easier to make decisions and take action at this level than to wait until the general assembly.

There are three types of association meetings.

The subcommittees of the association�s executive committee. The three subcommittees are the "workhorses" of the association: finance and planning, education, and water management. They meet about once a month to deal with problems that cannot be addressed at either the lower section or block levels.
The quarterly executive committee. The executive committee reviews the progress of the subcommittees. It has a limited budget, which it can allocate without approval from the general assembly.
The general assembly. Meeting three times a year, its responsibilities are to elect new leaders, discuss work plans and authorise the use of funds. Sometimes there are special agenda items concerning training or information dissemination.

8.3 Making M&E Events More Reflective

Critical reflection occurs in everyday planning, implementation and M&E activities. Each person involved in the project filters or changes information, prioritising and rejecting data continuously and often unconsciously. This happens in each informal chat, as well as during formal external missions.

Making a project reflective means planning more consciously when and how to deal critically with information. This is not just about ensuring you have one annual review with primary stakeholders. Box 8-11 illustrates how one IUCN (World Conservation Union) programme in Tanzania uses various elements to encourage ongoing and participatory critical reflection about its innovative collaborative coastal resource management initiative.

Box 8-11. Learning in the Tanga Coastal Zone Conservation and Development Programme, Tanzania (TCZCDP)

General project conditions

Limited geographic area in phase one allowed for deeper analysis and more discussion with and feedback from communities.

Working practices in the field

Frequent visits by a technical team to communities to help understand community reactions to project activities
Quick follow-up to problems and other issues that emerged in the pilot villages to allow for more feedback from communities and led to quicker learning about what did and didn�t work
Quarterly village feedback meetings (with an evaluation aspect during discussions)
Biannual action plan reviews
Annual regional meetings with all stakeholder groups (that provide community reactions) to share progress and problems

Programme team member attitudes

Team open to feedback from each other; effort made to communicate
Attitude of programme team towards listening
Principle of building on local knowledge and local resource management
Focused observations based on a clear plan
Not being afraid to try new ideas, because joint planning also means shared responsibility in case of problems
Problems valued � not seen as mistakes but as learning opportunities

Programme management mechanisms

Universal monitoring � everyone monitors
Joint planning
Enabling environment via open leadership and encouragement from the beginning of the project
Regional steering committee for adaptive management
Fortnightly team meetings
Participatory evaluations and reviews � with villagers, district teams, external evaluations and the project team
Periodic, targeted (i.e., activity-focused) reviews
Training to provide new knowledge and skills for the whole team � everyone having received training

Ensuring documentation

Widespread documentation that is systematic, easy, and accessible (but not perfect)
Quarterly progress reports that include a section on lessons learned

Ad hoc external opportunities

Opinions and ideas from visitors who are asked to share them
Seminars and workshops run by partners and others
A vocal public � partly encouraged by TCZCDP
Consultants� recommendations

8.3.1 Making Project Team Meetings Reflective

The critical contribution of the project team to overall success makes it worthwhile to invest in team meetings as an important opportunity for reflection. Project team members may include project staff, implementing partners and primary stakeholder representatives � this depends on how the project is structured. Team members are actively engaged in implementation, and reflecting on their experiences can contribute to refining implementation.

To make project meetings reflective, consider how to prepare for them, conduct them and follow them up.

Before the meeting

Decide who should be at project meetings. Besides the core members, you might want to invite other stakeholders from time to time. If so, how often should they be invited?
Agree on scheduling. Meetings must be long enough apart so that there is new material for reflection and yet frequent enough to allow for timely decisions to be made. Weekly meetings are common to most projects (see Box 8-12) but if other stakeholders are involved this may need to be less frequent. Monthly meetings with a different focus may be needed (see Box 8-13). For example, in Uganda, one project�s design has compartmentalised activities into components, with each component head responsible for monitoring the activities of that component. This has led to limited cross-linkages and poor learning across the components. Monthly meetings with component heads could focus on how the components interact and contribute together to the broader goals.

Box 8-12. Weekly staff meetings in Tanzania PIDP

The project team meeting is held each week with whoever is in the office, but is skipped if less than half the staff are available. The meetings focus on current issues, reviewing schedules and revising work plans as needed. The staff meetings run for about 30 minutes to two hours, depending on the issues. Staff share M&E reports from the fields and review recommendations. For example, in Lusilile village, the water users� association rejected the contractor because he was incompetent. The project team discussed this event. From this, they decided that all prospective contractors are to meet with the associations before a contract is signed. The association decides on the contractor after screening two or three potential contractors.

Box 8-13. Monthly monitoring meetings with local organisations

At monthly monitoring meetings in PIDP (Tanzania), members of formal and non-formal organisations present and analyse, among other issues, the summarised information concerning the project�s actions at the family and association levels. The data used come from family observation sheets that are filled in by extension staff or others responsible for monitoring. The extension staff used to avoid filling the sheets because they were bulky. So the formats have been simplified to facilitate use. The summarised information, plus agreements emerging from the meeting, are documented and delivered to the representative of the implementing partners.

Agree what M&E findings are to be discussed. By putting outputs of M&E activities on the agenda, they are fed directly into decision-making forums. In PROCHALATE (El Salvador), meetings of the technical evaluation committee and evaluation workshops include a discussion on monitoring surveys. Management progress reports are a key agenda item in the meetings of the technical evaluation committee.

During the meeting

Ensure everyone has the same agenda and that expectations are clear.
To share responsibility, build skills and create a team spirit, you can rotate the chairing of the meeting. Each meeting could be prepared and facilitated by a different project team member.
Ask staff to raise problems or dilemmas they are facing and invite everyone to find solutions.
Most importantly, when someone raises a critical incident or issue of importance, encourage analysis on these four questions:

What did I/we do?
What does this mean?
Why did this happen?
How can I/we do things better in the future?

At regular intervals, include constructive feedback exercises. Box 8-14 describes one exercise that is commonly used in groups.

Box 8-14. Constructive feedback using "Johari's window"⁴

This exercise can help increase self-awareness and trust in project teams, by helping people understand how their behaviour affects others. By getting feedback, people can help each other remove points of friction.

Present Johari's window, as below, clarifying the contents of each of the four squares and key terms as below.

- Feedback. One way by which others make you see your own blind area; for example, how you sound or what impressions you make

- Sharing. One way of opening yourself up more to others

- Revelation. Suddenly seeing part of the unknown area of yourself � an experience that cannot be planned

Ask those present to draw their own Johari�s window, filling in several examples in each box.
Ask participants to discuss some of their examples.
Ask participants to discuss in small groups why and how feedback is important in managing a project.
If you are doing this in a several-day workshop, get the group to practise giving and receiving feedback, with five minutes set aside everyday for giving feedback in small groups.

Known to Self

Unknown to Self

Known to others

OPEN KNOWLEDGE

What is known about you by others and by you � your name, colour of hair, etc.

BLINDNESS Feedback

The part of you that is known to others but not to you � your tone of voice, a conflict in which you are involved or a trait, etc.

Unknown to others

HIDDEN KNOWLEDGE Sharing

What you know of yourself but do not share with others. Some things may best remain hidden, but sharing with others might clear the air and build trust, making teamwork easier.

KNOWLEDGE YET TO BE REVEALED Revelation

What is unknown to others and also to you about you. Here are talents and abilities that you do not know you have and others have never seen but are part of you and may one day appear.

Make sure outputs are action-oriented

Document the meetings well. Focus on documenting "actions needed", "person responsible for implementation", "deadline" and "person responsible for follow-up". Box 8-15 shows how PRODECOP follows up its analysis of actions needed to assess if the actions have been implemented or not.
It is good practice to share the draft minutes with all present to ensure that there is an accurate and complete record of what was agreed. Another common practice is to put the minutes of a previous meeting on the agenda of the next one to sort out any differences of opinion.
Distribute the minutes from the meetings to all relevant people as soon as possible after the meeting to ensure timely action.

Box 8-15. Documenting the actions needed to ensure follow up (PRODECOP, Venezuela)⁵
(E=excellent, VG=very good, G= good, M=minimum, D=deficient, NA=cannot be assessed)

Weakness identified	Improved action suggested	Implementation of suggestion			Effectiveness
Weakness identified	Improved action suggested	Yes	No	Why?	E	VG	G	M	D	NA

8.3.2 Reflecting with Stakeholder Groups

Many of the ideas outlined in 8.3.1 are equally applicable, not only to team meetings but also to other group meetings. Common types of group events in many project M&E systems include water users� associations, micro-credit groups, micro-enterprises and village associations. In each project context, there will also be forums where implementing partners interact with each other, with project staff and with primary stakeholders, annual reviews and external missions (see 8.4).

Each of these regular events offers a chance for reflection on daily experiences. Sometimes special events may be needed. A particularly innovative type of group reflection is the "citizens� jury" (see Box 8-16). This is a way to organise debate at a societal level on issues that are of paramount importance to primary stakeholders.

Box 8-16. A citizens� jury/scenario workshop on food futures for Andhra Pradesh, India

Background. The state of Andhra Pradesh (AP) in south India is currently rethinking its approach to farming, land use and marketing. The AP government�s vision of the future of the state�s food system is presented in its Vision 2020. There has been little or no involvement of small farmers and rural people in shaping this policy scenario. Discussions with local and state level partners have revealed considerable concerns over the possible impacts of Vision 2020 on livelihood security, agricultural biodiversity and local food systems and economies. The UK-based International Institute for Environment and Development (IIED) and the Institute of Development Studies (IDS) facilitated a participatory process to encourage more public debate in policy choices on food futures for the state of Andhra Pradesh. The five-day "citizens� jury" on food and farming futures involved citizens from all three regions of AP and key organisations.

The citizens� jury. The jury was made up of representatives of small and marginal farmers from AP, small traders and food processors and consumers. To reflect the rural reality, jury members were mostly small and marginal farmers and also indigenous people. Over two thirds of the jury members were women. A panel of external observers oversaw the jury process. They checked the videos produced and observed the whole process. It was their role to ensure that each potential future food system scenario was presented in a fair and unprejudiced way and that the process was trustworthy and not captured by any interest group.

Visions of the future. Jury members were presented with three different scenarios. Each was advocated by key opinion-formers who tried to show the logic behind their scenario. It was up to the jury to decide which of the three scenarios is most likely to provide them with the best opportunities to enhance their livelihoods, food security and environment twenty years from now.

Vision 1 � Vision 2020. This scenario was put forward by AP�s chief minister, with backing by the World Bank. It proposes to consolidate small farms and rapidly increase mechanisation and modernisation. Production-enhancing technologies such as genetic modification will be introduced in farming and food processing, reducing the number of people on the land from 70% to 40% by 2020.
Vision 2 � An export-based cash crop model of organic production. This vision of the future is based on proposals within IFOAM and the International Trade Centre (UNCTAD/WTO) for environmentally friendly farming linked to national and international markets. This vision is also increasingly driven by the demand of supermarkets in the north to have a cheap supply of organic produce and comply with new eco-labelling standards.
Vision 3 � Localised food systems. A future scenario based on increased self-reliance for rural communities, low external input agriculture and the re-localisation of food production, markets and local economies, with long-distance trade of local surplus goods and goods not produced locally. Indigenous peoples� organisations and some farmers� unions in India and elsewhere support this vision, and further support can be drawn from the writings of Mahatma Ghandi.

Presenting the visions. Each vision was presented through videos. Video footage was assembled to illustrate the salient features of life under each particular vision. After this, expert witnesses presented the case for their vision. Members of the AP government, the corporate sector and civil society organisations were given equal amounts of time to present their case to the jury. Jury members were allowed to cross question expert witnesses after their presentation.

Jury deliberations. Jury members assessed the pros and cons of each vision. They were not asked simply to choose between Vision 1, 2 or 3 but, if necessary, combine elements of all three futures to create their own unique vision(s).

The verdict. The key conclusions reached by the jury included a desire for: food and farming for self-reliance and community control over resources; to maintain healthy soils, diverse crops, trees and livestock; and to build on indigenous knowledge, practical skills and local institutions. They opposed:

the proposed reduction of those making their livelihood from the land from 70%-40% in Andhra Pradesh;
land consolidation and displacement of rural people;
contract farming;
labour-displacing mechanisation;
GM crops - including Vitamin A rice and Bt cotton;
loss of control over medicinal plants including their export.

Organise heterogeneous groups appropriately. Primary stakeholders are not a homogeneous group. Group events need to consider this. The yearly participatory evaluation workshops of the Cuchumatanes project in Guatemala were originally run on the basis of production systems, with different workshops for farmers from different systems. However, it was found that the farmer groups who attended each workshop had different levels of development and so their degree of participation was unbalanced. As one project implementer described, "The more developed organisations finished the exercises first and then the rest were under pressure and did not go deep enough." Because of this, later workshops were organised on the basis of a typology of organisation, based on financial capacity, productive potential and so on. This produced better results at the workshops.

Focus on understanding and deciding, rather than describing. Much discussion in groups tends to focus on people telling "what I did since we last met" in considerable detail, rather than discussing what worked well and why or why not. Working with questions like "What would you do differently next time and why?" or "What would you do the same and why?" helps the group focus discussions around understanding experiences in order to improve actions. In a Moroccan project one project team member has initiated self-evaluation with cooperatives. Examples of some of the questions asked during the self-review meetings include:

What do you think you have been able to achieve in your work?
What are the most important weaknesses that you find in the monitoring methods in the management of the cooperatives?
What are your suggestions for improving your work?
What are your training needs?

Create thematic learning groups. Another idea with positive outcomes is that of formal "learning groups" focusing around a theme or problem area. For example, national and local learning groups interact in Senegal to understand better how village-based resource management functions.⁶The national learning group draws together key people from different hierarchical levels and sections within relevant bureaucracies and from external organisations. Local learning groups consist of representation from different groups within communities involved in participatory natural resource management in the case study areas. The groups enable shared analysis between people and organisations that might not otherwise exchange experiences, ideas and personal insights. Many projects undertake thematic evaluation or monitoring studies when specific issues arise. Learning groups could be assembled around these themes.

Make good use of the group by agreeing together how you will:

Prepare for group meetings. What must each group member do and share beforehand to make the best use of the meeting time? What questions are guiding these preparations? Do all members have sufficient time, resources and guidance (e.g., clear questions) to make a meaningful contribution?
Share information during group meetings. How do you use the limited time you have to do good analysis? What outputs do you expect from each meeting? How will you ensure that all members are able to share their thoughts?
Write up notes about the meeting. Who will be responsible for documenting the meeting? What will be included? How will decisions be documented? How will you ensure that all the group members� insights and information are represented in the document? How will the document be shared?
Regularly assess the quality of meetings. Keep improving your effectiveness by evaluating the meetings. Box 8-17 suggests a list of possible criteria for evaluating the meeting. Develop your own criteria based on what you value in meetings.

Box 8-17. Checklist of criteria for a reflective group event

Did we � ?

� hear the voice of everyone at the meeting in equal measure?

� not only share what we did but also what surprised us and what we might have learned?

� provoke each other to think by asking questions about the mistakes we made?

� leave the meeting with clear consensus about what each one of us is going to do before we next meet?

� discuss not only what went well but also what did not go well and why?

8.3.3 Using Steering Groups for Reflection

Most projects use steering committees of some sort to provide strategic guidance. In the Tanzania PIDP project, the project steering committee is the final decision-making and policy-making body. It is made up of the regional administrative secretariat, the permanent secretaries and funding agencies. It decides in which districts to work, gives final approval of the budget and approves the progress reports before distribution. Similarly, in PROCHALATE, El Salvador, the global and sectoral planning of the project and the general supervision of implementation is the responsibility of the national consultative committee. Different stakeholders sit on the committee: national organisations related to the project, representatives of implementing NGO partners, three representatives of primary stakeholders and the project director.

In other projects, the powers of such steering committees may be less extensive. For example, in ADIP, Bangladesh, the inter-ministerial project steering committee meeting is held only twice a year so does not have much direct engagement with project implementation.

Of primary importance in the establishment of the steering committee is clear extent � and limits � of responsibility. Once this is clear and agreed, and this can occur as part of project formulation, then the committee�s composition and frequency of meetings can be decided.

To use steering committees as learning opportunities, the ideas discussed in 8.3.1 are relevant. Use the meetings to assess dilemmas and problems and find solutions � rather than to report progress, as progress can be shared in documents beforehand. You might want to invite steering group members to visit the field at several moments during the project life and interact with field staff and primary stakeholders. This gives them a context, however brief, for the office-based discussions.

8.3.4 Learning from Your Annual Project Review

The overall purpose of an annual project review (APR) is to reach conclusions about achievements and failures in order to improve ongoing programme quality, and to share these conclusions. An APR is a critical learning opportunity. Yet not all projects include an APR in their M&E systems. An APR can also help to:

ensure the overall project goals, results and implementation strategy remain appropriate;
assess progress towards planned impacts;
review implementation to date and analyse reasons for any deviations;
review the operational and management effectiveness and efficiency of implementation;
identify lessons and actions to improve next year�s implementation and performance.

An APR provides a vital input for the annual work plan and budget process. It allows the entire project team and other stakeholders to stop, take stock of what has been happening, look at the monitoring data, look at performance questions and make collective decisions about what each person/group can do to improve the project�s performance next year. The more stakeholders are involved in these reviews (see Box 8-18), the better the picture will be of what has been happening and the more people will have an understanding of what still needs to happen.

In Cuchumatanes (Guatemala), annual evaluations involved interest group representatives and communal banks to analyse the social, economic and production changes generated by project. Interest group representatives were selected using a simple chance model but came from all five project areas with more than one year of partnership with the project. Focus group discussions were held with interest groups and the banks separately to allow open discussions.

Box 8-18. Annual process evaluation for KAEMP, Tanzania (1999/2000)

The annual process evaluation for KAEMP starts at the ward level. It is organised by the ward executive office, the head of administration at the ward level. The village government is aware of the different project groups active in the government and encourages these groups to send a representative to the evaluation. In addition, the village executive officer, the district project facilitator (DPF) and the extension staff at ward level are invited. Usually about 30 to 40 people attend.

At the meeting, a chairperson and secretary are elected by the participants. These two people will attend the related annual review at the district level where they will raise ward issues. The DPF facilitates a review of the action plan with questions such as:

What happened compared to what was planned?
What did not happen according to plans?
Why did these activities not happen?
What are the proposals for improving project implementation?

The group discusses these questions and responses are noted on flipchart paper to be documented later.

The annual project review is an important force for change as these examples show:

At prior evaluations, participants complained that there was little involvement of the village in the project situational analysis and planning processes, resulting in the promotion of "packages" of agricultural technologies that are not understood by the farmers � they do not know why the packages came and how the interventions are decided. As a result of these evaluations (and pressure from IFAD), farmers are included in the planning cycle by involving them in a situational analysis based on participatory rural appraisal (PRA).
The design for road rehabilitation was changed due to several issues raised by the villages. Since the villages wanted local people to benefit as much as possible from the rehabilitation process, they insisted on more local participation. As a result, the contractors must meet with the village leaders to discuss the design, to initiate the process of recruiting youth groups that will contribute labour (paid by the contractor) to the road rehabilitation, and to negotiate the labour rates.
During the process of providing safe water through protecting natural springs, villagers realised that they no longer had places to wash their clothes. Before they would wash their clothes on the rocks next to the spring, but when the springs were protected, the rocks were no longer available. As a result, the project started to provide concrete slabs for washing clothes. This was not in the original project design.

The mosquito nets that were initially distributed as part of the project�s health component were circular and small. This was not convenient as most people�s beds are rectangular and of varying sizes. The net design was changed to a larger, rectangular shape to accommodate variety in bed size and shape.

Step 1. Prepare well for the APR

Boxes 8-20 and 8-21 show two examples of annual reviews that might inspire you. The Tanzania example shows that annual reviews can be appropriate not just for the overall project, but for any level that involves various stakeholders and consensus about next steps needed. The El Salvador example highlights the importance of building an annual review process that encourages constructive criticism to strengthen and empower local organisations.

Besides internal self-evaluation and learning events, all projects also deal with external reviews and evaluations. These events can be very valuable moments for project stakeholders to focus on key issues and strategic changes. Outside perspectives and experience can help by raising questions about what is taken for granted and bringing out issues that might be sensitive yet critical. Those involved in project implementation can benefit from making these events reflective. This section focuses on how a project team can make external missions interesting learning opportunities but does not provide details for the external reviewers on how to undertake such missions.

Typically, a supervision mission will visit the project once a year, with another short, half yearly follow-up. A supervision mission can last from one to two weeks, with a team of three to five people. Supervision missions aim to provide regular external guidance on operations, progress made and technical problems (see Box 8-22), in order to support timely corrective actions. As these missions have a formal status, they can be useful moments for agreeing on changes in project directions.

Supervision missions vary considerably in nature and quality. Many projects experience them as policing exercises rather than collegiate support. It may be worth discussing with the responsible cooperating institution and IFAD how to modify mission styles to make them more collaborative along the lines of the example in Box 8-23.

Significantly for those concerned with M&E, supervision missions rarely pay sufficient attention to M&E from a learning perspective. If they do consider M&E, they tend to focus on the data collection side rather than how information is used and participatory M&E processes. To improve supervision missions and make them part of the overall project learning process, consider undertaking these steps.

8.4.2 Preparing for, Managing and Responding to External Reviews and Evaluations

When external reviews or evaluations work well, project stakeholders will feel that the external reviewers have:

However, problems can occur. Being aware of the key problems (see Box 8-24) can improve the reviews and evaluations. Project managers can work with implementing partners to prepare, manage and respond well to the review.

Once the external mission has arrived, senior management will need to coordinate the visit so it involves stakeholders fairly and appropriately.

The findings of the mission represent opportunities for project implementers to take corrective actions. Not dealing well with the report may mean losing out on an important learning opportunity.

The completion point of a project reflects an understanding among core project stakeholders about the lessons learned and recommendations from the final evaluation, and it expresses an intention to use these not only in the future implementation of this project, but also in designing new projects. Project completion reports do not need to be overly elaborate. For example, four completion reports undertaken in the Asia region looked at only four questions:

These questions can be answered in elaborate ways but also have simpler alternatives.

Impacts will always need to be considered at project evaluation. Many aspects of assessing impacts have been discussed in other sections of this Guide (see Sections 2, 5, 6, 7 and Annex D). Consider the following as you prepare for project completion.

Prepare well ahead of time if the learning process is to include other stakeholders. Particularly in the case of participatory impact assessments that form the basis of project completion, start preparations well in advance. If you wish to involve primary stakeholders in designing the impact study and implementing it, then this will require considerable time to organise perhaps some training events and pre-testing evaluation questions and data-collection methods.

Seek insights via open-ended questions. At project completion, you are expected to show what the impact of the project has been at the level of outcomes and purpose. Many projects try to show impact by using the indicators as set out in the M&E plan (see Sections 4 and 5 and Annex C). However, also seek to include more open-ended questions as this might reveal more accurately what has actually changed in local lives from the perspective of the primary stakeholders. Asking primary stakeholders how their lives are now and what can be attributed to project efforts will allow a broader understanding than by only looking at indicators related to the objective hierarchy.

Avoid overly-ambitious impact assessments. Many completion evaluations are overly-ambitious in terms of what they intend to assess and explain. Some impacts may only emerge well after project completion. So keep focused on the types of changes that are likely to be tangible.

Devote time for cross-cutting issues. Cross-cutting issues, such as impact on gender relations, benefits for the poorest groups and contribution to shared decision making, may lose out if they are not included explicitly in the evaluation questions. Box 8-26 lists the types of information you might seek to understand gender impact.

Do not delay "lessons learned" exercises too much. Undertake any lesson-learning work as promptly as possible at project completion to ensure the findings are relevant and up-to-date. If writing up lessons is left too long, then the information may become outdated and worthless. Furthermore, memories fade quickly, which may pose problems particularly when wanting to understand what is attributable to project efforts.

Document project completion reports including lessons learned. Table 8-1 offers an idea of what to include in the project completion report. Rather than describing what happened during the project, focus instead on distilling impacts and lessons learned.