I plan to use this document as a beginning. As I continue my work I hope to add to the list of positive examples of D,M&E in COs that others can use as reference points and resources for learning. I also plan to refine, add to or subtract from the positive trends and areas for improvement. If the reader has an interest in discussing any of the issues raised, would like to change an item or make a contribution, or learn more about the examples given, please contact me at westley@uk.care.org.

Positive trends

Data collection: Most country offices I have worked in can mobilise resources to collect vast amounts of quantitative and qualitative data. This is either done by CARE staff or contracted out to consultants, students, or research organisations.

Data entry: Similarly, country offices can access quantitative data entry skills either internally or from external sources. Generally, computer systems are adequate for basic data entry, storage and analysis.

Increasing skills in PRA and facilitation: Gulf of Guinea, Uganda, Zimbabwe and Bangladesh have staff with skills in PRA and facilitation of community meetings.

Field staff with good contextual knowledge: the high levels of knowledge among CARE CO staff of local issues, community priorities, social and cultural constraints and realities, and ecological conditions is a great asset for community assessment, project design and evaluation as well as project implementation. This knowledge should be used more effectively.

Increasing interest and commitment: Throughout the CARE world, interest in acquiring D,M&E skills, developing robust M&E systems, improving the quality of project design, and participating in lesson sharing appear to be increasing. Country offices, projects and CI members are devoting more resources to D,M&E and adding D,M&E staff to teams at local, national, regional and international levels. For example, in the past three years three new M&E positions have been created in the CARE-Bangladesh ANR sector; the Asia Regional Management Unit is recruiting two regional D,M&E advisors; CARE UK has hired a Programme Advisor for D,M&E; project budget allocations to M&E are increasing.

Sincere desire to improve: Most CARE staff I have come across express interest in training, on-the-job coaching, learning through exchanges and other means through which they can improve their own D,M&E skills.

Problem Areas and Creative Solutions

The following section highlights problem areas in D,M&E. Where appropriate, it also provides examples of CO strategies to overcome them. These are listed as a source of lessons learned for others facing similar issues. If others have experiences resolving some of the problem areas described below, it would be helpful if you could let me know, so that I can share your learnings with other COs.

Management structures and support: Most D,M&E staff sit outside typical headquarters, CO and project management structures. They are often working as a one-person unit. Because they have specific skills that are unique with respect to the other members of their departments, they often do not receive substantive feedback or support from their managers and colleagues. Due to the nature of their work they are sometimes treated with suspicion, or accused of undermining project efforts. In such cases, it has proved difficult for them to access the necessary management support to perform their functions or overcome barriers to their work. In some cases, their M&E contributions are seen as peripheral to project activities or as ‘add ons.’ Several M&E staff have been told that "M&E is a luxury;" which makes them feel like their contribution is not valued, and therefore that their job security is low. Another manifestation of low management commitment is the lack of time and resources dedicated to M&E activities. M&E staff members are expected to get their work done without the assistance of the rest of the team.

Many COs have explored the relative merits of having D, M&E staff forming a special unit outside of project structures versus having D, M&E staff within project staff. Of course this depends largely on resources available.

One solution tested in Ghana was to have an M&E specialist based at a sub-office and shared between two projects. This individual, due to their cross-cutting role, and their proximity to the two project teams was able to play an active role in project management decisions.

CARE Mali has a mission level M&E specialist who has been designated to play a key role in capacity building of project M&E staff, while upgrading M&E systems throughout the mission.

CARE-Bangladesh, because of the scale of its projects and the level of resources available has project-based, sector-based and mission-level D,M&E teams.

Weak links between data collection and analysis: There tends to be a physical and temporal gap between data collection and data analysis. The staff involved in collecting data are often not involved in analysing it. Analysis often happens months after the data is collected. Often data is not analysed at all. One M&E staff person joked that when they started their job there was a huge container of paper outside their office that one day simply disappeared. The point is that data is collected, it is not analysed or used in any way, and is in effect entirely disposable.

Over the past year, projects in the ANR sector have been shifting their monitoring activities from the headquarters to the field levels. This initiative grew out of concerns that analysis of monitoring data was not done by those who collected the information and who were involved in the day-to-day running of the project. Also, the time it took for Dhaka based staff to receive monitoring forms, enter data, send forms back to the field for corrections, and so on, meant that it sometimes took over a year for data to be processed.

A simultaneous effort to introduce more participatory methods into the project monitoring systems led to changes in the INTERFISH II monitoring system as well as in NOPEST, and other ANR projects. The reasons for introducing more participatory M&E are multiple and include: 1) increasing the validity of monitoring data by having field trainers and project participants involved in analysis; 2) increasing the quality of data by helping participants become aware of why they are being asked certain questions. The GO-Interfish team are in the process of finalising their participatory M&E forms which are one-page pictoral summaries of production and input data which will be used with farmers. This data will then be entered and analysed at thana and district level. Composite reports will then be sent to Dhaka, where they will be compiled and analysed for the project as a whole.

On the whole, the ANR sector is implementing changes in their M&E systems designed to streamline information collection, make it more participatory both within the CARE team and with project participants, and to analyse the information in a timely manner. One of the guiding principals of this process is that field trainers become aware of the ‘bigger picture’ of what their work is achieving in terms of impact on participants.

You can contact Faheem Khan at carebang@bangla.net for more information. You will need to include his name in the subject heading of the message.

External contracts for key M&E learning opportunities: Baseline studies, mid-term reviews and evaluations are opportunities for team building and lesson learning for project staff. Often, however, external consultants and enumerators conduct them, with little staff involvement. While in some cases this is necessary to maintain objectivity and transparency and indeed required by donors, in other cases it is a lost opportunity for the project team. Nor is it surprising that project managers and others often complain that the quality of baselines and evaluations is disappointing, or that the baseline is ‘useless.’ In one project, the mid-term evaluation was of such poor quality that I asked to see the original questionnaires. They were nowhere to be found and the consultant had ‘moved offices’ and had lost them.

Disconnecting data from the logframe: One of the most common analytical gaps in D,M&E systems is in the process of relating data collected to the logframe. Baseline surveys, in particular, do not make this connection adequately. What this means is that projects measure a great deal but often don’t show how what they are measuring relates to what they set out to achieve. This is one of the main reasons why baseline documents are not useful; while they are often very comprehensive in terms of the data collected, the data may be largely irrelevant.

Participation vs. consultation: Although many D,M&E systems and processes claim to be based on participation, in reality they are more like to consult with stakeholders in limited aspects of implementation, but not involve them in decision-making, design, or resource management.

Several country offices have experience designing and using participatory M&E systems. NOPEST and INTEFISH in CARE-Bangladesh, the PROSPECT project in Zambia and the TEAM project in Lesotho are notable examples. The INTERFISH and NOPEST experiences have been documented. Reports are available from the CARE-Bangladesh ANR sector.

The SCORE project in CARE-Gulf of Guinea provides PRA and action planning training to community members who then carry take leadership in finding community-based solutions to education problems. This experience has been documented. The report is available from Sam Oduro-Sarpong, in Ghana – care@ghana.com. Include his name in the subject heading.

Not monitoring unpredicted outcomes: In general, monitoring systems do not pick up unpredicted outcomes. Field staff have no structure or process for reporting on effects and impacts of the project outside the parameters of the logframe. As a result, course corrections and appropriate adjustments to project implementation will not occur.

In Bangladesh, CARE and DFID have developed a Livelihood Monitoring System that over the next five years will monitor broader project impact, the vulnerability context in which the project is operating and attribution of livelihood changes to project activities. This process is just getting started but you can obtain more information from Phil Sutter at carebang@bangla.net (include his name in the subject heading, or from me: westley@uk.care.org

Only monitoring and evaluating positive change: The majority of logframes and M&E plans focus only on monitoring the positive changes expected from a project. This can be thought of as a ‘systematic bias.’ Risks associated with the project are often captured as assumptions. This puts them outside of the project framework, whereas in some cases they should be addressed by the project directly. Indeed they become ‘killer assumptions’ and undermine the effectiveness of the project. Also, while the project may have a positive impact on a specific target group, few M&E systems address the question of impact on non-targeted groups and individuals: is it positive, negative or negligible? In some cases there may be losers as well as winners. Knowing this and who they are would improve project targeting overall.

The East Africa Regional Management Unit has developed tools for benefit-harms analysis. Contact Jim Rugh at rugh@care.org for further information

The learning environment: In some cases, project staff are not encouraged to critique project activities in an objective way. They are expected to collect monitoring data and meet targets without questioning the effects or relevance of their work. In some cases, evaluations of staff members’ performance is linked to achievement of project targets. The potential for misrepresentation is obvious. In such a situation the organisational culture does not promote learning.

In other cases, CO staff have little time or resources to interact with staff from other projects. Missions run in a ‘tubular’ fashion with little cross-learning.

CARE Bangladesh has set up an Institutional Learning Team. This team addresses gender, partnerships, and D,M&E and promoted learning at the mission level. A livelihoods advisor will soon be joining the IL team.

The Livelihood Monitoring System being launched by CARE-Bangladesh will be looking at learning processes within two CARE projects as well as between CARE and other stakeholders.

Working with partners in M&E: The CAGES project in Bangladesh has produced an excellent document about their experiences with M&E and their NGO partners. In the CAGES project, NGO partners submitted false data about the productivity of fish cage culture, and the contribution of the project on farmers’ income. In some cases the data was biased to either make the impact of the project activities look better or worse, in other cases monitoring forms were just filled in the office and submitted, and clearly were not based on field reality. There are many lessons learned documented by the project. The key one here is that NGO partners were not involved in designing the monitoring systems and forms, nor were they briefed on the relevance of monitoring to their work. It would be naïve to say that fostering a sense of ‘ownership’ of the monitoring systems would in and of itself reduce fraud in reporting, but it is worth a try! Many smaller NGOs do not have an ‘M&E culture.’ More and more projects that work with small NGO partners are finding that M&E is a key area for capacity building of these NGOs. Simply put, projects have found that they cannot expect NGOs to carry out high quality M&E if they do not have the skills. Therefore, the projects need to collect monitoring data themselves, or build the capacity and/or provide the resources for the NGO to do so.

The CAGES project team has written up their experience. The document: ‘Working through NGOs: Lessons Learned from the CARE-CAGES project, Bangladesh’ can be obtained from the Project Coodinator, Kenny McAndrew at carebang@bangla.net. (Remember to include his name in the subject heading.)

Another helpful resource is the INCOME III partnership monitoring system, which is a set of tools developed to monitor the capacity building of local partner micro-finance institutions. This material can be obtained from Carlos Ani, SEAD Co-ordinator at CARE-Bangladesh (carebang@bangla.net – include the name in the subject heading.)

The PEP project in Togo works through two local NGOs. They have dedicated significant resources and time in the start-up phase of the project to developing realistic and mutually acceptable relationships between CARE and the two NGOs. This included a week-long workshop to revise the project document and log-frame as well as developing a participatory capacity building plan for each NGO. They have developed numerous lessons learned, which can be obtained from the Project Manager: Claudine Mensah at care@care.café.tg

Another very common strategy in the health sector is to strengthen the Health Management Information System of district health services, and then to expect to collect project related data through the government HMIS. In some cases, the incentives for maintaining a high quality and reliable HMIS are not considered. One District Health Officer asked me once: ‘why should we keep records for CARE; we don’t even have a computer to keep our own records?’ In another case, the CARE team had invested a significant amount of time in training an MOH clerk, who was then transferred to another district. Unfortunately, in every project that I have worked with that employs this strategy, district health personnel felt that CARE was ‘policing’ their data. The alternative – creating a parallel HMIS for the project – is clearly undesirable. A longer process of negotiation and discussion around these issues may help resolve some of the motivational problems. Another approach would be to change the indicators we expect to measure through the HMIS to ones that are collected from clients and community members.

"Skills set" of M&E staff: Current recruitment of M&E staff focuses primarily on computer and statistical analysis skills. There is a growing trend towards hiring M&E staff with participatory assessment and monitoring skills. However, training, facilitation and communication skills are not emphasised. Nor are M&E staff expected to have design experience. This pattern is symptomatic of the compartmentalisation of D,M&E activities: data entry is separated from data analysis, is separated from decision-making, is separated from lessons learning, is separated from design, and so on ad infinitum. As a result, some of the most dynamic M&E staff leave the organisation. Others are not equipped with, or give the opportunity to acquire, necessary skills that will enable them to relate their data analysis to the broader D,M&E picture.

Who is involved in design: Conventional wisdom says that an Assistant Country Director sitting in a room alone typing up a project proposal, is no longer an acceptable design process. Although this is becoming a common refrain in CARE, it is still a great challenge to involve a range of stakeholders in a design process. Within CARE, we tend not to involve field staff (or anyone below a Project Manager). Involving community members beyond the assessment stage is even more rare. NGO, government, and CBO partners are usually consulted, but don’t play a large role in the design itself. Many projects are now introducing a start-up phase where a more intensive process of consultation leads to adjustments to project design after funding is secured. Sometimes, however, partners feel they are presented with a fait accompli; they are invited to make suggestions, but only within the confines of the project budget, staffing structures and strategies that have already been defined.

NO-PEST design workshop: The NO-PEST project in CARE Bangladesh recently held a design workshop that was based on participatory assessment with communities and a one-week problem analysis workshop. Workshop participants included all levels of project staff, as well as farmer representatives from different regions. More information about this experience can be obtained from the Project Manager: Marco Barzman at carebang@bangla.net. (Include his name in the subject line.)

Application of CARE-Zimbabwe Household Livelihood Assessment: In Zimbabwe results from community assessments have been used to assess the programming gaps in the two Regions and to develop areas for co-ordination of project teams and development of new projects. All levels of staff we involved in this analysis Documentation can be obtained from Kelly Stevenson in CARE Zimbabwe: root@care.icon.co.zw.

Gender is equivalent to counting men and women: More and more M&E systems are designed to collect disaggregated data for men and women; many of them stop at that. Assessing intra-household dynamics, differential impact on different age/ethnic/social groups and potential negative impacts on these different groups is becoming more frequent, but is not yet common practice. Several projects are using special case studies or thematic qualitative studies to address these issues.

The CARE USA gender initiative has worked in several country offices on improving gender in programming. More information can be obtained from martinez@ny.care.org. CARE-Niger has been working on assessing the impact of projects on intra-household dynamics. The country director, Doug Steinberg, can provide more information. His e-mail is c-niger@intnet.ne

MER: (For all those who feel the need to take sides in the cosmic debate on MER, rest assured. There are positive aspects of MER discussed further down.) I have yet to see a project that is actually using MER. What I have seen is several projects struggling to get the system up and running and getting caught up on software issues that should be simple to resolve by somehow are not. The staff using MER in COs are often the most highly trained in computer programming and computer based MIS. These staff say MER is not ‘user friendly.’ More importantly, however, is the lack of support for M&E staff who are working through the quirks to get MER running. While i2K provides support for software issues, and staff receive one-off training in the systematisation of M&E plans for transfer into MER, there is no source of on-going coaching and on-the-job support. Staff report that i2K is often very slow to respond to questions, perhaps because they are meant to provide support on technical software issues only, not on systematising and automatic specific CARE project M&E systems. CARE should consider a mechanism whereby CO staff exchange learnings on the set-up of MER. Staff who have already worked through the software and have project M&E systems running smoothly could provide support to novices through exchange visits or e-mail.

Frustration arises simply because there has been a high level of awareness created about the potential of MER, but there has been little sharing of practical experience. The great strength of MER is that it links M&E data directly to he log-frame – an analytical step that is often overlooked. Therefore, instead of running several different computer programmes for data storage, analysis and presentation, with MER projects can combine all of these steps into a single software.

CARE-Honduras has been using MER for several years now at both the project and programme level.

Conclusions

The diversity and dynamism of D,M&E processes and practice in the CAR world deserves documentation. Many of us re-invent the wheel as we go, not realising how many resources there are to help us. The CARE International Impact Evaluation Guidelines are a clarion call for all CARE COs and D,M&E professionals to help each other learn from mistakes and replicate successes. It should be clear from this document, that good D,M&E is about more than technical skills of D,M&E staff and hiring good consultants. It is about developing an organisational culture that supports learning and enables discussions of the negative and positive results of CARE’s work around the world.