8.1 An Overview of Reflecting Critically to Improve Actions

8.1.1 What Is Critical Reflection?

Critical reflection in a project means interpreting experiences and data to create new insights and agreement on actions. Without critical reflection, your M&E data will not help you to manage for impact (see Section 2). Active discussion during team meetings and in meetings with primary stakeholders are vital if M&E information is to be shared, analysed and acted upon. Making analysis "critical" means moving beyond collecting, processing and reviewing data. After asking "what is happening", also discuss:

• "Why is it happening?"
• "So what are the implications for the project?"
• "Now what do we do next?"

If you manage to discuss these questions regularly with project stakeholders, then you are well on the road to reflecting critically. Reflecting critically means questioning what is normally taken for granted, particularly project assumptions (column 4 of the logframe matrix, Section 3). This requires reflecting on what did not work or is not working. The project objective hierarchy is based on assumptions about the context and the likely effect of project activities. Only time will tell if these assumptions are valid or not as it will become clear what has and has not worked. Learning from what did not have the desired effect enables you to adjust your mental model of how the project works and work with more valid assumptions.

8.1.2 The Importance of Critical Reflection for M&E

Normally, relatively little attention is invested by projects in critical reflection. Most M&E efforts and resources are spent on collecting data, statistical analysis and basic reporting of activities. This has several causes (also see Section 1). Data collection is more mechanical and requires less capacity. M&E is often seen as a bureaucratic obligation requested in the contract and by government regulations and not as valuable to the project. No one seems to think they gain from frank analysis of under-performing projects. Reversing such views requires building capacity, sensitisation and putting in place effective incentives (see Section 7).

The following dialogue shows how embedded a non-reflective approach can be in M&E. The dialogue took place in Bangladesh between an M&E consultant (C) and a sector coordinator (X):¹

C: "What purpose do the weekly reports serve that you request of the district coordinators?"

X: "It is policy that they must send reports weekly."

C: "Yes but what purpose is the policy serving?"

X: "To encourage them to work harder and to inform higher management."

C: "Higher management does not have the time to read weekly reports and the district coordinators are being paid to work hard. Why do you ask them to give you weekly reports?"

X: "When they know that they are being watched, they will pressure their thana managers to work harder."

C: "They are already pressuring their staff hard. You can find the results in the monthly reports. Why do you need weekly reports?"

X: "To inform higher management."

Focusing on producing data because someone wants it rather than analysing it critically in order to learn from it defeats the purpose of M&E. Project stakeholders can only improve their actions by reflecting regularly on data, planning moments for such reflection and taking time to learn lessons. Decisions need to be documented and shared with those concerned. Managers need to check that decisions have been implemented as agreed. Then you can say that learning has taken place.

8.1.3 Reflective Events in a Learning Sequence

Project M&E involves an extensive series of potentially reflective events – from weekly team meetings and informal sessions to the more formal supervision missions and mid-term reviews/evaluations. These events, whether self-organised or externally initiated, occur alongside data gathering. During these events, project stakeholders can use the data gathered to indicate areas of improvement.

Learning does not happen in one sitting. It evolves, starting with individuals raising important issues and questioning assumptions through group-based analyses (see Box 8-1) that bring out different perspectives and information inputs. So you will need to plan "learning" as a series of events. Knowing how to structure the sequence is important (see Box 8-2). The optimal sequence of learning events follows reporting lines and hierarchies of decision making.

As project manager or M&E officer, you will no doubt find that project stakeholders have different levels of ability and willingness to reflect critically on their practice. You cannot force people to engage in a change-oriented learning process but you can put in place some simple opportunities that encourage it. Sections 8.2 and 8.3 discusses concrete ideas for stimulating individual reflection and reflective events, while 7.3 offers ideas on putting in place incentives.

An important moment in the learning sequences is when lessons are identified. Project and partner staff are continually learning, sometimes unconsciously, what to improve and changing their everyday actions accordingly. Sometimes it is useful to systematise this learning in the form of "lessons learned". A lesson learned can have an internal audience – the project and partners themselves – as well as an external audience consisting of other projects, other funding agencies, IFAD and so forth. Their value for projects that are managing for impact is outlined in Box 8-3.

Lessons for internal learning are particularly important when a project is innovative or for external learning after several years of implementation. PADEMER, in Colombia, is an example of an experimental project. Its appraisal report explicitly states the role of lessons learned: "Given the project’s pilot nature in terms of experience, the evaluation function will place special importance on reviewing lessons of implementation, when informing the expansion of activities into new areas." For TNWDP (India), lessons were identified in its completion report and they focused on nine themes: an enabling environment, group formation and cohesion, NGOs, targeting, training, income-generating activities, marketing, financial operations and executing agency.

Section 8.2.2 describes how to undertake a "lessons learned" exercise.

Back to Top

8.2 How to Encourage Critical Reflection

8.2.1 Starting with Individual Reflection

Learning starts with the individual. One critically reflective and innovative staff member can make a considerable difference in a project (see Box 8-4). Your management style will strongly determine whether or not project staff and staff from implementing partners and primary stakeholders will feel free to initiate and participate in similar types of learning exercises.

If individuals do not reflect during their work on their own, then they will probably find it difficult during group events, such as annual project reviews or monthly meetings with implementing partners. While not everyone is equally capable or interested in developing a reflective working style, everyone can start somewhere.

Simple practical steps can make a significant difference in encouraging individual reflection in project staff and in staff of partner organisations.

• Including the expectation of review and reflection in job descriptions and TOR and memoranda of understanding. You can include a clause such as the following in a job description: "The postholder is expected to review regularly (at least once a year) his/her performance with the direct manager, with the intention of identifying areas for improvement." In a memorandum of understanding with an implementing partner or primary stakeholder group, stipulate that not only will raw data be reported, but also the implications of the information for action are to be given. Make explicit the expectation that partner staff participate in regular workshops to assess implications for action.
• Encouraging reporting that asks staff for their opinions. In the TROPISEC project, Nicaragua, the implementing partner has introduced three monitoring tools: the community promoter’s notebook, the notebook of the assistant soil conservationist and the extension agent’s agenda. Staff elaborate their monthly reports directly in the notebook or agenda. They note down field activities developed, primary stakeholders assisted, achievements, problems and proposed solutions.
• Regularly asking project stakeholders their views. This means not just at formal meetings but also in chance encounters to stimulate people to form and share opinions. Ask what surprises they have encountered, what innovations they have tried out and what changes they would like to see in the project. In TROPISEC, Nicaragua, it is a requirement that "each technician or professional of the [management unit] must have as an annual purpose the capitalisation of at least one experience to exchange with other colleagues".
• Providing constructive feedback. If you are managing others, give positive reinforcement about what you like and constructive criticism about what could be improved (see Box 8-5).

8.2.2 Capturing Lessons Learned with Project Stakeholders

Learning needs to be systematised. Accidental learning happens all the time but is not the most efficient way to learn nor does it necessarily lead to improved actions. Increasing successes and avoiding pitfalls is best when conscious efforts are made to learn lessons.

A lesson learned is defined as "the knowledge derived from experience that is sufficiently well founded and can be generalised so that it has the potential to improve action". Many projects and organisations talk about the importance of identifying such lessons. But in practice, few are able to identify lessons that actually help improve actions. Two examples of lessons from IFAD-supported projects illustrate how to include an action orientation (see Box 8-6).

How to distil lessons learned

"Lessons learned" events are not daily occurrences. The frequency with which lessons are identified will be less than weekly or monthly team meetings. In some projects, the team and partners identify lessons at annual project reviews, at mid-term reviews and again at completion. If you want to have an active learning organisation, then including lessons as part of the annual review is a good idea. Waiting until the end of the project will mean wasting many potential learning opportunities.

A lessons-learned event can take several days of intense discussion and must be prepared well. In the planning, answer these six questions with other key stakeholders before you start:

1. What do we mean by a "lesson learned"?
2. Why do we want to identify lessons?

• For reporting (to primary stakeholders, partners, to funding agencies)?
• To learn for ourselves for a next phase of the work?
• To deal with a crisis?
• As a strategy to support fundraising (by being able to report on lessons)?

3. For whom are these lessons and, accordingly, how are they best shared (written, verbal, video, drama, etc.)?
4. Whose lessons are they – primary stakeholders, field workers, management? And, therefore, who should we involved in identifying the lessons?
5. What are the lessons – per stakeholder group (source and audience)? And how do we prioritise them if there are too many to share (see Box 8-7)? To narrow the discussion around key lessons, first decide on the themes or issues where most learning occurred or is needed.
6. How do we document the lessons and how do we link the lessons into the next phase of planning?

Formulating a lesson learned

To formulate a useful lesson, consider the following.

• Include a generalised principle that can be applied in other situations. Do not write the lesson only as an observation, description or a recommendation that lacks justification.
• Explain the lesson in the context of the project. For it to be useful to others, they need to understand the situation in which it occurred. Otherwise, they will not know if the lesson might be appropriate for them or useful. Relating the lesson to one (or more) assumption(s) on which the project is based can help others understand what exactly has been learned.
• Justify the lesson with proof of why it is valid. But if it is a hypothetical lesson, test it. Do not rely on it without ensuring it is valid.
• Check that the lesson is not too general or too specific to be useful.

Document the "lesson learned" with at least these five elements.

1. Theme of "lessons learned"

The core question that the project asked itself due to a methodological innovation or problem encountered, or because it is a key theme of the project.

2. What was our original understanding or assumption?

A short description of the original understanding of the theme/question. This is what people assumed before the experiences on which they are now reflecting. For example, "We assumed, incorrectly, that there would enough women able to participate in the micro-enterprise training workshops."

3. What is our revised understanding or assumption?

The new thinking about the original question/problem or the reworded assumption. Taking the example above, "We now know that we not only need to create interest in micro-enterprise training but also need to find ways to make it possible for women to attend the course."

4. One or two examples that substantiate the new understanding

To be sure you have a high-quality lesson, you need to provide evidence that supports the proposed lesson learned. Only with multiple sources of proof will it be solid enough to apply in future. The greater the number of sources for a "lesson learned", the more rigorous the supporting evidence.³ The more the evidence supports the lesson learned, the more confidence you can have in the lesson’s significance and meaningfulness. If you have only one type of evidence then you have a "lessons-learned hypothesis". This will need to be tested and verified.

5. How the project came to this insight

   Description of what triggered the project team to be challenged in its views and revise its understanding (Was it a crisis, monitoring data, a field observation that contradicted earlier ones, etc?).

8.2.3 Planning for an Integrated Sequence of Reflective Events

All projects have several routes for learning and decision making. These need to be interlinked if they are to be complementary and not duplicate efforts. In the example from Tanzania, in Box 8-8, each step in the sequence of meetings has its own mandate in terms of types of information shared and discussed and the types of action-decisions that it can make.

Knowing how to construct a useful learning sequence is a skill that must be learned. This may require some trial, error and innovation. The optimal sequence of learning events follows levels of aggregation (see Box 8-9), reporting lines and hierarchies of decision making. Start a sequence at the lowest possible level where decisions are made (Box 8-10). If a decision must be made elsewhere or discussed with others, then link this into a sequence by feeding any lessons or conclusions from one discussion and decision forum to the other.

Sometimes you will need to include special evaluations in a learning sequence in order to fill a knowledge gap. For example, the WUPAP programme (Nepal) planned two human rights assessments as part of its M&E process. These will look at the extent to which the programme has promoted awareness of rights, advocacy, and how to improve the human rights situation for the next phase.

If you have a sequence of learning events in which conclusions are documented and passed to another decision body, provide feedback on any conclusions. If, for example, project management receives progress reports from implementing partners, provide them feedback on the quality of the reports and on how the information has been used. This is important not only to let partners know how information has helped influence decisions, but also to redirect them in what they include in reports. It can help move them away from giving lists of activities to reporting on results, problems and proposed solutions.

Back to Top

8.3 Making M&E Events More Reflective

Critical reflection occurs in everyday planning, implementation and M&E activities. Each person involved in the project filters or changes information, prioritising and rejecting data continuously and often unconsciously. This happens in each informal chat, as well as during formal external missions.

Making a project reflective means planning more consciously when and how to deal critically with information. This is not just about ensuring you have one annual review with primary stakeholders. Box 8-11 illustrates how one IUCN (World Conservation Union) programme in Tanzania uses various elements to encourage ongoing and participatory critical reflection about its innovative collaborative coastal resource management initiative.