7.1 An Overview of Putting in Place the Necessary Capacities and Conditions

7.1.1 Capacity for People and Their Organisations

When asked why a project M&E system is not working, a common response is "poor" or "insufficient capacity". Capacity is "the ability of individuals and organisations to perform functions effectively, efficiently and in a sustainable manner".¹ According to most people, "capacity" means the human ability knowledge and skills to do a given task. A most common answer to inadequate capacity is "lets send the M&E officer on a training course" where new knowledge can be heard and new skills can be practised. Although a training course can provide valuable input, every course has limitations. In practice, much capacity is built on the job through concrete experience. For an effective M&E system you need skilled people who can, between them, fulfil the M&E functions and tasks. Key tasks include: designing the general outline of the M&E system (see Section 3); setting up and operating supportive computerised systems (see 7.5); facilitating learning in reflective events (see Section 6); and managing the communication of M&E findings (see Section 6). Meeting capacity needs will require that you:

1. Acquire the right people by:

• hiring already trained people;
• training your staff (internally or via external courses);
• hiring external consultants for focused inputs.

2. Ensure capacity of good quality by:

• removing disincentives and introducing incentives for learning;
• being clear about what you expect;
• keeping track of staff performance through regular evaluations;
• outsourcing data verification;
• striving for continuity of staff;
• finding a highly qualified person to coordinate M&E.

3. Build capacity for M&E. Start by developing an M&E training plan for all stakeholders - and with them. This entails agreeing on who is expected to do what and assessing if they have the necessary skills and conditions. You can undertake training using a combination of these three options:

• external courses;
• internal courses, tailor-made for stakeholders and linked to the development of the M&E plan itself;
• on-the-job training/mentoring.

4. Invest in capacity for participatory M&E (PM&E). Work closely with project staff, implementing partner staff and primary stakeholders to identify what is needed to make PM&E work and to develop plans to fill capacity gaps. When working with consultants on PM&E, clearly define her/his responsibilities, hire the same consultant(s) to ensure consistency in approach and build relationships among stakeholders; also, include PM&E in the terms of reference (TOR) and discuss with each potential candidate how she/he sees PM&E.
   

7.1.2 Paying Attention to Incentives

Putting in place incentives for M&E means offering stimuli that encourage project managers, M&E officers and primary stakeholders to perceive the usefulness of M&E, not as a bureaucratic task, but as an opportunity to discuss problems openly, reflect critically and criticise constructively in order to learn what changes are needed to enhance impact. It involves implementing encouragements and removing disincentives.

When thinking about incentives, consider those you can put in place within the boundaries of the organisation, that is, without rocking the boat, and also those that might require structural changes to the way the project is organised and operated. Also consider whom they are meant to stimulate so that they engage with learning-oriented and participatory M&E. This will allow you to fine-tune incentives for particular groups.

Incentive systems should be equitable, applied in a timely manner, be compatible with the projects principles and strategies, and be recognised as part of a projects policy. Incentives need to be context specific and aimed at supporting sustainability of efforts. This is why financial incentives are undesirable in many contexts, as sustaining them beyond the life of the project would be unfeasible.

Good incentives for M&E are closely linked to general management efforts to improve overall project performance. Examples of common incentives include:

• clarity of M&E responsibility in job descriptions and work plans;
• appropriate salaries and other rewards, such as housing and vehicle use;
• support to carry out required project activities, such as making financial and other resources easily available;
• professional development for career advancement.

You do not have as much influence over incentives for implementing partners and primary stakeholders as you have for project staff. Yet it is crucial for them to be as motivated as project staff, when it comes to participatory learning-oriented M&E. See 7.3 for more ideas.

Incentives will change during the life of the project. Keep motivation high by changing incentives. They may vary per project phase as the M&E tasks and issues change and, in some cases, actions taken early on may prove to be incentives at a later stage. For example, in Ghana, potential applicants for project positions had to go through an intense selection process. On top of this, ministry staff that applied also had to be nominated by the head of their department. Because they knew they had passed a tough recruitment process, project staff, including those in M&E, held a high respect for each others professional skills and abilities.

7.1.3 Getting an Optimal Structure for M&E Responsibilities

Getting the basic structure for a projects M&E functions and responsibilities right can avoid major communication bottlenecks, conflicts of power and interest, forgotten or duplicated tasks, and wasted efforts. This saves resources and headaches. Organising responsibilities means considering the most appropriate contribution for project staff, partner organisation staff and primary stakeholders and how to link these.

M&E is part of every single persons job, from the messenger to the project director. Monitoring is a daily and spontaneous activity. Yet it is important that M&E functions also have a clear position in the project structure, whether among project staff, with partners or among primary stakeholders. High visibility and clear positions of authority for those with M&E responsibilities can help link information to its use in decision-making.

To ensure clarity of M&E functions and tasks:

• define the M&E responsibilities of implementing partners and primary stakeholders;
• consider what staffing levels are appropriate for the set of M&E tasks and functions you need to fulfil;
• allocate clear levels of authority to M&E-related staff;
• ensure overlap between project management and M&E;
• use detailed job descriptions for each staff member to coordinate inputs.

All projects use consultants in some form, local or foreign, short-term or long-term, extending big responsibilities or small tasks. Ensure that:

• you are using them strategically for M&E development in ways that build local capacity and build on existing M&E forms;
• when contracting them, you are completely clear about what you expect them to add to the existing systems and expertise, by when and in what manner (particularly vis-à-vis primary stakeholders) they are expected to work;
• you are working with as much continuity of consultants as possible to minimise the need to reconcile conflicting advice.

7.1.4 Thinking through the Information System

In IFAD-supported projects, the quantity of information that is collected and needs to be shared justifies well thought-out information systems that store data and make data accessible to others. This is also vital for a participatory process. Documentation provides the foundation for interactive communication, transparency, consensus-building and continuity.

Storage of two types of information is needed impact-related information to guide the project strategy and progress-related information to track operations. To store this range of information, from survey data to copies of contracts and correspondence, will probably require different information storage systems.

Computers can make a critical contribution to tracking and using data but are no panacea. Achieving impact certainly does not depend on computerising data. Information that needs to be shared can also be photocopied and circulated, with each recipient using a common filing system.

To set up a computerised information system, follow these steps:

1. Define what you want to store in the information system and for what purpose.
2. Define your basic network structure by analysing how, when and by whom the database will be used (see Figure 7-1 as an example of a network structure).
3. Identify how you plan to process the information, who will do it and what forms this will require.
4. Compare options for software and hardware (the network) and decide whether to invest in existing software or contract a specialist for tailor-made software.
5. With your preferred option in mind, undertake a more focused data management analysis.
6. Establish the formats needed for database entry.
7. Provide user training on the system, otherwise it might never get used optimally.
8. Adjust the system regularly by evaluating its use with the users.

7.1.5 Finances and Resources to Do the Job

Solid and systematic learning costs money. Financial resources are needed for the time people spend, for supporting information management systems, training, transport, and so forth.

Key items to include in the budget are:

• contracts for consultants/external expertise (fees and travel expenses);
• physical non-contractual investment costs;
• recurrent labour costs (permanent staff salaries, temporary support staff);
• focused labour inputs, such as technical assistance, TA (short or long term, national or international);
• training and study tours for M&E-related capacity-building;
• non-labour operational costs (e.g., stationery, meetings, allowances for primary stakeholders and project implementers, and external data such as maps).

While there are no fixed rules for this, M&E budgets range from 2% to 15% of all costs. In projects where stakeholders are exploring new ways of working with partners, M&E budgets are likely to be proportionally higher as more time is needed to reflect on what works. Note that each project clusters its M&E costs differently, according to the adopted approach.

Irrespective of how the M&E budget is calculated, it will always overlap to some degree with other project activities. Therefore, do not excessively detail the M&E budget. Much learning occurs through the normal interactions of project implementation. What is most important is to budget for the events, procedures and staff time that support project learning and reflection.

Participatory learning processes are more time intensive than those in which only a few people are involved. More time is needed to organise meetings with larger numbers of people and more diverse groups and to reach agreement on how to proceed with M&E or on what data mean. Consider these budget items for participatory M&E:

• specific training for staff in participatory techniques and participatory M&E;
• extra meetings with stakeholders for designing M&E;
• additional meetings for local-level analysis;
• short training workshops on key steps in designing M&E and specific elements such as indicators and methods (including using the logframe matrix).

Back to Top

7.2 Human and Institutional Capacity

Two sets of skills are critical for M&E to be effective: that for dealing with diverse data and that for dealing with diverse people. But many more skills come into play. Table 7-1 lists key areas of M&E knowledge and skills to which each project should have access. As M&E officer or project director, you can use this as a checklist to know if you have the right mix of skills and understanding available among the project stakeholders.

Depending on the projects size and resources, functions can be combined or be assigned to implementing partners, sub-contractors, primary stakeholders or project staff. No matter who has the competency, they must be available to the project and not duplicate efforts and not work in opposite directions (see Box 7-1).

Table 7-1. Capacities needed among project stakeholders for good M&E

7.2.2 Acquiring the Right People

As project director or coordinator of an M&E unit, you have three options to ensure you have enough of the right kind of capacity on hand.

1. Hire already trained people. This is ideal but very difficult for most projects to achieve. Few people are skilled in conventional M&E, let alone the type of participatory learning processes encouraged in this Guide, which asks for people to be creative, conceptually clear and good communicators. Hiring already trained people means being able to provide enough appropriate incentives to make and keep the job attractive (see 7.3).

2. Train the people you need. Training, on-the-job or through external courses, will always be necessary. Even the most trained M&E professional will need to upgrade skills and understanding. Primary stakeholders will need capacity-building to undertake their own M&E and contribute to the projects. Field staff will require continual skill building as information needs in a project shift and new methods of data collection and analysis are required.

3. Hire external consultants for focused inputs. Consultants are a common source of M&E expertise. This is a relatively expensive option and does not contribute as much to local capacity-building but it is often the only alternative when local expertise is not available and time is short. Consultants are particularly necessary at project start-up when staff may not yet be hired and relationships with implementing partners are weak. The initial workload involves establishing management and M&E processes, staff recruitment, finalising the AWPB, defining reporting procedures, setting up information systems, and providing on-the-job training. To make the most of consultants requires clarity about their expected contribution (see 7.4).

Your personnel strategy will be a mix of these three options. If, as project director, you want to create a learning environment, then you are likely to:

• try to hire the best possible person to guide M&E efforts or seek such a person from among the implementing partners to take on that responsibility;
• seek focused inputs by consultants on specific issues where time and/or skills are lacking, for example, to develop a participatory M&E approach to stimulate self-evaluation among primary stakeholder groups;
• draw up a plan of ongoing M&E training for all stakeholders contributing to M&E.

7.2.3 Ensuring Capacity of Good Quality

"Capacity" is not only a question of sufficient numbers. Good quality is fundamental. Being able to recognise good quality M&E will help keep your learning efforts on track. Box 7-2 shows what good quality M&E meant for one project in India.

Remember that what is essential for one level of staff, with its specific responsibilities, may not be necessary at another level. For example, while you expect extension staff to be excellent communicators with primary stakeholders, the project statistician must be excellent at working with numbers. However, in projects where extension staff input field data, they must have several qualities.

Recognising good quality is one thing, but what can you do to ensure good quality M&E among the project and implementing partner staff? One Tanzanian project attempted this by changing its staff selection procedure to advertise positions outside the government, rather than only within. This offered more chance of finding someone with the right qualifications. In some Latin American countries, contestants for a management job in an IFAD-supported job participate in a training and selection process that includes a two to three day workshop. There they are observed discussing issues, how they deal with groups, make decisions, etc. The selection panel includes government staff, primary stakeholders, and experts. Section 7.3 deals with incentives as a way to ensure good quality.

Below are other suggestions for ensuring good quality.

1. Remove disincentives and introduce incentives for learning. Encourage project and partner organisation staff to be curious and open about learning by providing a range of incentives. Limit those disincentives that may keep them from sharing their mistakes and learning from them. See 7.3 for more on incentives.
2. Be clear about what you expect. Clarify the standards of the M&E capacity you expect and put procedures in place that make sure these standards are reached and maintained. Staff job descriptions (see Annex E) and performance reviews are key mechanisms.
3. Keep track of staff performance through regular evaluations. In most project documents, staff performance is assessed by the degree to which they implement project activities. But also assess what they have been learning from primary stakeholders and colleagues, what learning innovations they are initiating and how they are using any information they have collected.
4. Outsourcing verification of data. Any project can benefit from an external view on what is happening. One way to do this is to sub-contract an organisation periodically to check the quality of data and of data use. This will give you confidence in the methods being used to gather data and can provide additional guidance to project stakeholders on data quality. But avoid making it a policing exercise.
5. Ensuring continuity of staff. Continuity of staff is both very valuable and very difficult. By limiting the number of people who come and go during the project life, you can build a more consistent and less fragmented body of experience. A large part of keeping staff is offering the right incentives.
6. Find a highly qualified person to coordinate M&E. This can give them higher status vis-à-vis the rest of the team and makes it more likely that you have the kind of capacity you need at that level.

7.2.4 Building Capacity for Participatory M&E

Few appraisal reports detail how primary stakeholders should engage with M&E. Therefore, for projects working with participatory M&E (PM&E) there is also little clarity on the capacities needed by staff, by primary stakeholders and by consultants. For many projects that are still getting to grips with basic M&E, the idea of undertaking PM&E may appear overwhelming. While it does require careful thought (see Box 7-3), many small changes can be made that contribute to more interactive forms of learning.

Getting to Grips with the Implications of PM&E

Making M&E a learning process that extends beyond your project team means making M&E participatory. You will need to think through the many implications of including primary stakeholders actively in reflections on progress and impact, as well as in data collection, analysis, selection and updating of indicators, etc. (see Table 7-2).

For PM&E to be worthwhile, stakeholders staff, implementing partners and primary stakeholders must be able to participate meaningfully. This means that project and partner-organisation staff need skills in participatory facilitation techniques plus an appreciation of the importance of seeking other peoples views. Staff need to be committed to making participation happen. Good PM&E also means that primary stakeholders must have the conditions and understanding to make a significant contribution. To do PM&E well will inevitably require capacity-building for everyone.

Table 7-2. How participatory M&E differs from conventional M&E²

1. External courses. The most common courses are those provided in Europe or North America. Increasingly, however, regionally based courses are available. The advantages of external courses are the exchange with participants from other countries, the more focused/intensive use of time, the lower cost to send staff to an existing course rather than design your own training event, and the credibility overseas accreditation can add to a projects M&E and incentives. The disadvantages are that external courses are not focused around project needs, the project M&E system cannot be created as one output of the training course, course material may not always be relevant, and a one-off training is never enough to build sustained understanding.
2. Internal courses. You can hire a consultant to deliver a training course for any of the stakeholder groups. The advantages are that internal courses: are relatively cost-effective if larger numbers of staff are trained, target project needs and skill levels serve to build a common language and provide understanding for all involved. Also, if the course is interactive, then you jointly develop elements of the M&E system en route. Disadvantages include: highly dependent on the quality of the consultant, more expensive and more time consuming if the training is integrated with the actual development of the M&E plan.
3. On-the-job training/mentoring. Most learning occurs through informal sharing of problems and solutions. You can formalise this approach to capacity-building by allocating time for key M&E stakeholders to consult with external M&E mentors. This can work as an incentive for staff, offers the option of timely and problem-focused advice, and allows for ideas that are fine-tuned to the project M&E system. However, finding these mentors may be difficult and they are unlikely to have ideas for all queries. Making such mentors available to primary stakeholders is equally important for their capacity development but may be even more difficult to organise. An alternative is to encourage and arrange for staff and stakeholders to interact with other projects that are more advanced in PM&E processes. The exchange-visit approach can be particularly successful for training primary stakeholders. Some on-the-job strategies can be simple. A project manager in Indonesia maintained a strict procedure of monitoring at monthly intervals as well as quarterly and annually. He used this as a means to train staff in systematic and standardised implementation. As soon as he was confident in their capacity, he would reduce the monitoring frequency to quarterly and half-yearly intervals, being aware that it did not really make sense to monitor at such high frequency.

• Everyone accepts and shares the mission statement.
• The organisational culture and institutional atmosphere encourages good performance.
• The history and traditions of the project and related organisations are respected and known for a focus on learning and improvement.
• There is a supportive, fair and inclusive leadership and management style.
• There is an attractive incentive and reward system, including performance-related incentive plans.
• There is a shared idea of promoting teamwork towards organisational goals.

• prize for "most innovative fieldworker of the month/year";
• study trips and training courses for staff;
• avoidance of staff retrenchment.

• review of project directors performance by project staff;
• "downward accountability", that is reporting to primary stakeholders on progress with implementation and including them in decision making on strategic planning;
• "citizens jury" about project strategy (see Section 8).

• considering where to locate the M&E functions in a project structure (with primary stakeholders and implementing partners);
• being clear about M&E responsibilities.

Finding the Right Place for M&E Functions

It cannot be overemphasised that M&E is part of every single persons job, from the messenger to the project director. The director of the MARENASS project in Peru said, "We did not have a formal M&E unit for the first three years. This did not stop us from monitoring and evaluating our work. Everyone contributed, even the office driver had an M&E function." Everyone, in their own way, keeps track of the operational and sometimes the more strategic aspects of their work whether there is enough petrol in the tank or whether the team is working well and impacts are emerging. Monitoring is a daily and spontaneous activity. Decentralisation to encourage ownership of learning processes further emphasises this.

Nevertheless, experience shows that the location of those responsible for M&E is critical for performance. Linking it directly to project management helps ensure that M&E findings are used to inform decisions. This can be interpreted in different ways. The M&E unit of a Guatemalan project was planned as part of the executive board that supported project management, while in China the M&E unit was located within the responsible ministry. Try to make sure that M&E functions are represented at a high strategic and resource management level, and that they are also incorporated into the approaches and activities of all project implementers.

Simply giving M&E a high visual status in the project is not enough, as was found by one IFADsupported project in Africa. It had been decided that to improve M&E, the M&E unit would be elevated to the status of a project department, similar to a field department. Despite this, the M&E coordinator explained that M&E still:

• made a limited contribution to major decisions in discussions on technical issues, while management decisions were made without M&E participation;
• made a limited contribution to monitoring field implementation, as this was undertaken by technical assistance experts directly under the project manager;
• had insufficient resources allocated to carry out its functions, in particular, inadequate budget support, transport and computer support.

Changes would be required in terms of attitude and in terms of organisational structures and clarity of responsibilities.

Setting up M&E-conducive organisational structures should have been addressed at project formulation (see Section 3). Responsibilities and power can be set up in such a way as to hamper or encourage learning among project stakeholders (see Box 7-14). They are often inadequately addressed but also affected by changes in the operating context that can require negotiations about organisational structures with responsible departments, IFAD or cooperating institutions, and implementing partners.

	Box 7-14. When responsibilities and structures obstruct good M&E: Yemen, Morocco and Colombia The M&E functions of a Yemeni project were carried out by the M&E department of a government agency responsible for M&E in several projects, using national guidelines. This agency had much experience and was able to commence project M&E activities at an early stage. However, the agency did not have direct access to the projects M&E resources and had limited funds. Obtaining authorisation for activities and resources was a lengthy procedure, having to pass through a hierarchy of project management and government staff. Furthermore, as the relationship was sometimes tense, this affected M&E budgeting, incentives and adoption of M&E recommendations by the project. Project staff would often check data themselves without involving the agency, especially for information critical for management decisions. Being that the project M&E system had been based on the existing government system, it had limited relevance for the project. Also, the government agency did not prioritise M&E for this project as they also had several other projects to attend to. Overall, this organisational structure was hindering effective M&E. Two projects in Morocco were built directly into the provincial-level department of agriculture and hence shared many of the same staff, including the person responsible for monitoring agricultural activities for the entire province. While incorporating project activities within government structures may encourage sustainability, staff found that both projects had to compete with ongoing government work and political pressures. This resulted in a conflict between ensuring good management of the project and maintaining the normal work of the department. The model of project execution in a Colombian project appraisal report would have led to scarce interaction and knowledge of field issues by the project management. Information was to be sent from the implementing partner to the project in the form of a general progress report, the project itself having no direct contact with the partner NGOs developing the fieldwork or with the rural micro-enterprises of the primary stakeholders. Furthermore, the appraisal report stipulated that the project pay for an external evaluation unit to undertake evaluation functions. There was a danger that this would lead to duplicating activities, with the project M&E officer assuming monitoring functions and the implementing partner guiding the evaluation. Clarity of M&E Responsibilities Clarity of M&E functions and tasks (see Figure 4-2, Section 4 and Annex E) is essential. In a Colombian project, where monitoring had been separated from evaluation, confusion arose. Evaluation was the responsibility of the evaluation unit. Although monitoring staff did not have evaluation in its mandate, it still needed evaluation information for its own reporting. The evaluation unit had only undertaken extremely limited field visits since project start-up, so the monitoring staff ended up having to take on some of the evaluation activities. Keep monitoring and evaluation tasks and functions connected, rather than splitting them among different people as is common particularly when evaluation is sub-contracted. Several key lessons have emerged from project experiences in terms of ensuring clarity of functions and tasks. Clarify the M&E responsibilities of implementing partners. In Colombia, a project reallocated the M&E responsibility of reporting on impacts to project management. It had been in the hands of an implementing partner but this had not worked well. The project had learned an important lesson: always arbitrate services after making it clear in the agreement the commitments and responsibilities of each party and, above all, stipulate the information that should be reported in terms of effects and impacts, establishing concrete and clear mechanisms to guarantee this. Clarify the M&E responsibilities of primary stakeholders. Clarity of functions with primary stakeholders can be laid down in memoranda of understanding and also in contracts. Although community members may start with a simple monitoring role, other roles, such as stakeholder responsibility for managing contractors (see Box 2-20, Section 2), can increase efficiency and effectiveness in implementation, and local ownership. Consider what staffing levels are appropriate for the set of M&E tasks and functions that you need to fulfil. M&E performance is affected by staffing that is top-heavy, too light or distributed in a way that obstructs good communication and coordination. The M&E system for a very extensive income-generating project in Benin was designed with as light an overall structure as possible. One person was responsible for synthesising all periodic reports, checking with the field, entering basic data, dealing with multiple requests from project coordinators and so on. Much monitoring was carried out on field activities and impacts, but this was too much for one person to analyse, and there was insufficient time for partner performance assessments and impact studies. The project director described the situation as being "like a city having only one point of contact to the outside world, only to be cut off every time a problem comes up on the bridge." Allocate clear levels of authority to M&E-related staff. M&E-related staff need sufficient recognition to undertake functions that others in the project may perceive as intrusive. For example, if one person needs to coordinate progress reports, then she/he must have the publicly given status to ensure that other staff communicate the necessary data. Where M&E functions are decentralised and shared among different organisations, clear lines of communication and authority are equally pertinent but may require more negotiation. Ensure overlap between project management and M&E. This will encourage the much-needed interchange to guarantee that M&E findings inform decisions. For example, in the RADP project in Yemen, the M&E coordinator would act as interim project director in the latters absence. Linking M&E responsibilities and activities with technical project departments also increases the overall capacity for learning. Job descriptions for each staff member are crucial management tools. They help clarify expectations. See Box 7-15 for items to consider when elaborating job descriptions (also see Annex E).   Box 7-15. M&E items to consider when drafting job descriptions Which M&E responsibilities does the position include? What is the responsibility: design, data gathering, data inputting, reporting, facilitating, use of M&E data in decision-making? What type of issues will the position need to track progress with activities, quality of process, etc.? What are the minimal reporting requirements you need from the position (e.g., raw "data", analysed data, lessons learned, actions to be undertaken)? What quality standard do you expect the position to fulfil (e.g., timeliness or type, product/service)? What are the deadlines for the products/services?   7.4.2 How to Make the Most of Consultants All projects contract consultants at some point in some form local or foreign, short-term or long-term, extending large responsibilities or small tasks. Strategic Use of Consultants for M&E Development Hiring a consultant for a particular part of the M&E process usually means either you need extra expertise to cover for project staffs lack of time or you need specific expertise or a particular view (e.g., on component activities or methodology) that project or partner staff may not have. Consultants can be hired at any point of the project life. They can be hired as one-offs or hired for a longer-term input over several years. Working with consultants has both advantages and disadvantages (see Table 7-8). With these in mind, consider well what you would like them to contribute, particularly in terms of building project capacity. Project staff in Benin saw that review missions were making very positive inputs through their systematic analysis and identification of pressing issues. The staff realized that the M&E system itself should be producing such an output but it had only ever posted generalists without such capacity to coordinate M&E work. To increase internal professionalism, they took to hiring experienced consultants for specialist expertise. Table 7-8. Advantages and disadvantages of consultants Advantages of Working with Consultants Disadvantages of Working with Consultants - Give quicker output (perhaps). - Able to stand back from the project and ask questions that staff arent able to see. - Bring interesting ideas from other projects. - Can raise sensitive issues that project staff fear raising. - Can provide on-the-job training if they work closely with project staff. - May be trusted by funding agencies for the impartiality of their views. - Can increase the professional level of M&E. - Can provide focused inputs that are only needed once. - Dependency on outside expertise diverts budget and attention away from investing in local staff. (This is reduced if working with local consultants.) - If working with different consultants, then project staff may need to reconcile contradictory or different views and ideas on how to tackle M&E. - If not well recruited, they may only provide a piece of the missing expertise. - If they are not committed long term to the project, then this may make their suggestions unfeasible and fit poorly in the project. - If they do not have local experience, then advice or ideas may be inappropriate. - If they work in isolation, then local capacity wont be built. - If not living locally, then they cannot help adjust and adapt their recommendations as they are being implemented. Clarity When Contracting Consultants A project in Benin contracted a consultant to design its M&E system but did not include follow-up in the contract. The consultants TOR stopped at design. He helped review initial indicators and ideas for the baseline survey, the project team was not yet sufficiently settled to learn optimally from his input. As the consultant was not involved during the crucial phase of setting up the M&E system he had designed, additional technical know-how on data collection and analysis was lost. Clarity is vital: first in terms of the projects needs and then in the terms of reference about the expected contribution to M&E. Consider the items below. What contribution do you expect them to make, what specific outputs should they deliver, and what are the limits of responsibilities? Team leader of external evaluation, database designer, developer of participatory annual review process, etc. When do you expect them to deliver the first draft and the final report? All in one intense period or in a series of shorter inputs? Before start-up or after? To whom is the consultant responsible and with whom is she/he expected to work? How is the consultant expected to work (alone, with project staff, in a more or less participatory style, minimum number and types of stakeholders to consult, etc.)? Will there be an opportunity for staff/primary stakeholders to respond to the consultants report? Who owns the final report? Who will use it and how? How much is this going to cost, also in terms of other support that the consultant may need from the project (e.g., transport or translation)? This will enable you to assess whether you are getting your moneys worth. The project coordinator may not know the exact areas of technical expertise needed, so may find it difficult to write out the terms of reference. Asking the opinions of those who will be directly affected by the consultants input is good to avoid miscommunication about purpose and expected outputs. This will also help ensure a close fit between the gap in staff skills that the consultant is expected to fill and the needs of staff (project or implementing partner). Annex E provides some examples of TORs for consultants input. Reconciling Conflicting Advice The importance of continuity of perspective for M&E should not be under-estimated. The more different consultants have a say in project M&E, the more problematic this may become, as each consultant has a theory of M&E, a set of methods, different priorities, etc. The more these are mixed, the more confusing this can be for a project. Many projects have faced the situation of working with different consultants over the years due to an inability of the consultant to commit him/herself to the project in an ongoing manner. A common effect is that the project is left picking up the pieces of partial and sometimes contradictory advice (see Box 7-16).   Box 7-16. Conflicting concepts from consultants in Mali The PDR-San project in Mali has worked with two different consultants on developing its M&E framework. Both have recommended a different way of dealing with the idea of impact. One has focused on a list of indicators, another has abolished that and focused on household case studies. To make matters more confusing, the cooperating institution has a different opinion yet again. Just after developing an elaborate questionnaire for tracking impact on several case study households, the cooperating institution told the project that the project should not invest in assessing impact. This should be done only by an impartial external evaluation team or organisation. It is no wonder that project management is left wondering whose ideas to follow.   To limit conflicting advice, ask the consultant to: read all key material produced by previous consultant(s); spend time with project and partner staff to clarify what was found useful from the previous consultant(s) and what is now needed; make sure that the consultants contribution builds on what has been found useful so far and that she/he explicitly states how the existing elements of M&E fit with her/his contribution. But it may already be too late. Before you know it, you are looking at vastly different sets of advice. First and foremost, use your own judgement on what will work for you in your context. You might find it helpful to come up with a cost estimate and technical capacity assessment for the more complicated recommendations consultants make. Second, refer to relevant sections in this Guide for guidance on what is considered good practice. Project Sustainability and Technical Assistance Working with technical assistance (TA), even more than with consultants, requires careful thought to avoid TA leaving a lack of sustainability. TA is often appointed for several years to implement and be responsible for the work plan. In one project, M&E staff were experiencing problems integrating M&E into the project. A TA team arrived and was able to connect the M&E staff with project management effectively. But the TA team also took on many M&E tasks and reports written without much input from local M&E staff. Applying a long-term perspective, the TA input might have been organised more effectively to build local capacity. Back to Top 7.5 Organising the Projects System for Managing Information 7.5.1 Why an Information System Is Vital Typical IFAD-supported projects are large initiatives, in terms of numbers of primary stakeholders to be reached, geographic coverage, types of activities, budgets, time frame and sometimes number of staff. Keeping track of relevant information means that memory and handwritten notes alone will never be enough for those responsible for management or M&E. Furthermore, several people/partners may wish to use the data at different moments for different aspects of the work. The quantity of information that is collected and shared justifies some form of information system that stores data and makes it accessible to others. A participatory process is also vital. Documentation provides the foundation for interactive communication, transparency, consensus-building and continuity. Information storage is needed at two levels impact-related to guide the project strategy and progress-related to track operations. The focus of most M&E-related information systems is on registering indicator-related information to assess progress with implementing the logframe. In addition, AWPBs often include formats to track information on operational aspects, such as personnel issues, vehicle use and accounts (see Box 7-17). To store this range of information, from survey data to copies of contracts and correspondence, will probably require different information storage systems (see Box 7-18).   Box 7-17. Information for clarity of responsibility The PDR-San project in Mali keeps track of all correspondence in which the project requests something of its implementing partners in order to monitor the decision-making process and its timing. This has been useful in analysing where bottlenecks were occurring and correcting them. It has also been very helpful during supervision missions to show that in several significant cases delays were not caused by project management but by one of the partners. This included a delay of more than one year in contracting the M&E consultant that the project badly wanted.       Box 7-18. Aspects to include in the management information system of WUPAP, Nepal (extract from project document) Within the first six months, the programme will establish a comprehensive MIS (management information system) to provide: intranet with email facility and a database application for storing M&E data, reports and records; a filing and documentation system for efficient tracking of documents and a library to store programme documents and written and audio-visual records for reference; software and hardware support.     7.5.2 Setting Up a Computerised Information System The information system of your project will consist of paper-based archives as well as computerised databases. As the latter are often not as well structured, this section focuses on how to go about setting up a computerised information system. Computers can make a critical contribution to tracking and using data but are no panacea (see Table 7-9). Much time and effort is wasted in many projects on computerising data that is then never used. In Jordan and Ecuador, projects spent substantial amounts of resources and efforts to develop a computerised database system but relatively little effort in ensuring that it became functional. On the other hand, as a project in Tanzania learned, handling information coming in from several project components may be usefully computerised to extract higher-level findings on, say, cost effectiveness or efficiency of an intervention. Be clear about what will be stored on paper. For example, what will you do with interview notes? With copies of maps or matrix rankings (if you use participatory diagramming methods)? With geographic maps? These cannot always be placed on computers, although software options for this are growing. The larger the number of people who need to use the same data for different kinds of analysis, the more useful it is to computerise it on a network. But this is only true if they have easy access to the network and the information on it. In deciding when, where and what information needs to be computerised, this should be located as close to the field as possible to avoid mishaps with or loss of raw data, to enable data aggregation but above all to facilitate access for data inputting and use. Achieving impact certainly does not depend on computerising data. Information that needs to be shared can also be photocopied and circulated, with each recipient using a common filing system. Further, some projects may find themselves in a culture that doesnt keep written records of activities, such as the KAEMP project in Tanzania. The project team there had to start by training participants in keeping farm records and writing notes, rather than investing in systems and training for computerising data. Other projects have resisted computerisation as it means having to become more transparent and undergo more checking. Changing this requires sensitivity. When developing computerised system, it is paramount that this is not divorced from management information needs. This can occur if software is used with which management staff are not comfortable and which relies on more junior level expertise. Table 7-9. Benefits and drawbacks of incorporating computer systems as part of project M&E 6 Benefits Drawbacks - Makes it necessary to define M&E indicators and variables in highly precise terms. - Provides precise information with different levels of aggregation. - Reduces the amount of time usually required to process data, so increases time available for analysis. - Facilitates timely access to information by various groups and creates conditions necessary for improving those groups analytical capabilities. - Systems are inflexible to changes in indicators and units of analysis; methods are not straightforward to integrate. - Quantitative measures may dominate over qualitative measures. - Analysis may be very mechanical if it is limited to producing reports via standardised computer programmes, limiting reading to these existing report formats.   The process of setting up the information system has eight steps. These steps will be iterative, particularly in situations where M&E is being set up with project stakeholders. A system needs to be adapted as information needs in particular become clearer for the stakeholders, which is rarely possible in one go to the degree needed for a computerised database. Define what you want to store in the information system and for which purpose. Section 5 discusses in detail how to decide what to monitor and evaluate. Define your basic network structure (see Figure 7-1 as an example), analysing how and when the database will be used and by whom. This will strongly determine the design and timing of data inputting and analysis. Ask all project and partner staff: "Who needs to have access to what information and who needs to input data?" Identify how you plan to process the information, who will do it and what forms you need for this. As previously mentioned, locate this at the lowest possible level to make it easier for people collecting the data to analyse them. This also limits distortions in data analysis. Compare and decide on options of existing software and hardware (network). Keep in mind that data inputting should be relatively easy to do for those responsible, that data access must be in the right place at the right time in the right form for the users, and that the system needs to be within the financial and technical means of project and implementing partners. This means that a decision must be arrived at with all involved. You have two options, each with advantages and disadvantages (see Table 7-10): (1) basic existing spreadsheets and database structures, such as Excel and Access (both part of Word Office) and (2) custom-designed software. Figure 7-1. Example of defining the network and its connections (from NWUDP, Nepal) Table 7-10. Comparing existing and custom-made software Existing Software Custom-Designed Software Pro: - Relatively cheap (purchase of licences, purchase of hardware, training, supplies) - Easily available - Easily adaptable by the project and often relatively easy to master - Interchangeable/Compatible with others - More well-known Con: - Less flexible and less suited to specific project needs Pro: - Specific to the data needs of the project - Can be made accessible to all levels, e.g., by working with icons rather than words Con: - More expensive (design personnel, purchase of licences, purchase of hardware, specialist training sessions, supplies) - Needs time and expertise to develop, test, refine - The more specific, the more difficult to adapt at later stages Largely due to budgetary restrictions, the PADEMER project in Colombia does not use an automated information system or a beneficiary database. Instead, it simply extracts information from implementation and impact reports and stores this on Excel spreadsheets. A project in India established a simple data-handling system without a central network. Instead, district units were equipped well enough to provide the monthly progress reports on diskette or by email. This information was used to track project progress throughout the region. If you decide to contract a consultant to custom make your system, be sure the purchased or designed software can be used flexibly during the project life, since information needs are guaranteed to change. One project in Mali is planning its computer system around the assumption that data to be stored will stay the same for many years. This will pose increasingly serious problems for M&E officers as time proceeds and information needs change. Box 7-19 describes one software programme to aid with monitoring annual work plans.   Box 7-19. ASISTENTE - a programme from Latin America for annual work plan monitoring ASISTENTE is software that monitors activities and results set out in the logframe and annual work plan (AWP). It developed out of a project in Guatemala that needed to handle: a large, isolated and ethnically diverse region; servicing many stakeholder groups; diverse stakeholder involvement in project implementation and receipt of information; and decentralised services to farmer organisations. The current version of ASISTENTE has been adapted for use in all Latin American projects. Specific characteristics of ASISTENTE are: Versatile: it can be used in diverse analysis contexts of the project AWP, including project components, results of operational plans, beneficiary services and users of the monitoring system. Multi-purpose: it operates on various local/satellite computers where monitoring information is inputted and local reports produced. A central computer consolidates information sent by the satellite computers to produce consolidated reports. Information is available for all stakeholders. User-friendly: information is inputted once to be processed to obtain diverse reports. This requires no specialised training and is relatively quick, since the programme uses icons and legends that are activated with a mouse. Reports produced are numbered and with graphics and so are accessible to many people. Reports can first be checked on screen prior to printing or filing and they can be reformatted for insertion in other documents or reports. Safe: restricted access profiles defined by the supervisor avoid mishandling of inputted data. Each user is assigned a name, password and access profile, and is only able to save and modify her/his own information and generate her/his own reports. Users do not have access to information saved by others nor can they generate reports by other users without authorisation. The system supervisor has the right to configure terminals and users, input and modify information, generate all reports and assign passwords. Complementary software, also emerging from the Guatemala project and based on the same elements as above, is called DIRECTORIO (Programme for the Registration and Monitoring of Primary Stakeholders, Social Organisations and Implementing Organisations). It is for information on identification of and impact on primary stakeholders, their local organisations and implementing organisations. The majority of projects held this information, but usually not organised in a useful way. DIRECTORIO stores information related to: number of primary stakeholders and their productive activities; number of men and women; place of residence and age; services offered and number of primary stakeholders receiving each service; co-executing agencies offering services; effects and impact achieved. DIRECTORIO can hold a complete register of the primary stakeholders, rural organisations and implementing organisations, incorporate a gender focus and the concept of stakeholder services, input and monitor indicators on effect and impact on the primary and other stakeholders, generate various types of numerical and graphic reports, and classify data by project and by region.   With your preferred option in mind, undertake a more focused data management analysis, talking with everyone from the secretaries to the responsible ministry about information needs and uses and what peoples roles are in data management. This communication is needed to ensure that the system you have in mind is capable of dealing with the precise information and reporting needs. Establish the formats needed for database entry and keep them in line with those for gathering information. Think about how you will store qualitative and quantitative information. Do not develop precise data formats until you are completely clear on what needs to be computerised. Provide user training on the system, otherwise it might never get used optimally. The ongoing storing, updating and accessing of computerised information take skills that require user training and follow-up support from an accessible professional. Training includes those personnel involved in designing field data collection materials. In a project in India, for example, training for data entry was given at the start of field implementation. However, staff responsible for monitoring at the overall-project level were not included in the training. This, together with problems in data quality and data checking, meant the impact studies requested by the supervision missions still werent available by the time of the mid-term review. In another project, in Malawi, many M&E functions were carried out at the national level, such as an agricultural survey and household survey, the results of which were computerised. However, when it came to analysing the results, the M&E sections at the divisional level did not have the documentation on the software used, and thus analysis was delayed. Adjust the system regularly by evaluating critically with the users what information is being used by whom, what is not, what problems exist, and whether other people need to or would like to have access to information. Back to Top 7.6 Finances and Resources to Operate the M&E System 7.6.1 Budget Items to Consider To be effective, the M&E functions need to be supported with a realistic and clear budget. Knowing what to include in M&E budgets is not always as clear cut as in other areas of the project, as many M&E functions and activities overlap with implementation and management activities. For example, management meetings where M&E information is discussed and analysed is this a management cost or an M&E cost? While it does not matter where such costs are allocated, it is critical to include them, make clear that they are for M&E purposes and be clear about who can decide how to use the M&E budget. However, do avoid putting all M&E under the heading "project management" as this makes it very unclear what is available for the M&E. The M&E budget wherever it might be allocated needs to cover the items listed in Table 7-11. Table 7-11. Items to consider in an M&E budget Categories Specific Items Contracts: consultants/external expertise (fees and travel expenses) - Developing a detailed M&E plan - Establishing monitoring mechanisms and formats - Establishing information management systems - Facilitating review workshops - Conducting specific survey or monitoring work - Assisting with capturing and documenting lessons learned - Training and capacity-building (workshops, courses) - Impact assessment studies (e.g., baseline studies) - Mid-term and final external evaluations Physical non-contractual investment costs - Equipment for monitoring - Communications and presentation equipment - Establishment of M&E offices (e.g., furniture, equipment) - Publication materials - Computers and software - Vehicles Training and study tours for M&E-related capacity-building External and on-site training courses: - Training of primary stakeholders to build capacity in M&E - Training of selected implementing partners concerned with community development on introducing and supporting participatory monitoring - Training selected M&E staff of service providers (government and NGO) on relevant M&E aspects - Training M&E officers and key management staff on M&E, including computer training - Financial management training, as appropriate - Promoting exchange of experiences with other projects, among the different stakeholder groups - Course fees Labour costs: - Recurrent permanent staff salaries, temporary support staff - Investment technical assistance (short- or long-term, national, international) - Planning and developing the M&E system - Conducting regular monitoring - Report writing and analysis - Participating in review processes and events - Information management - Capturing and documenting lessons learned - Disseminating M&E findings - Supporting community based/participatory M&E processes Non-labour operational costs - Vehicle fuel and maintenance and other transport - Office running costs (overheads, maintenance) - Stationery - Meetings - Allowances for primary stakeholders and project implementers - External data, such as maps - Communication and publication costs printing/copying documents, editing, layout and publication of key documents Specific evaluation events (M&E planning workshops, review workshops, field survey work, stakeholder consultations, specific monitoring activities, mid-term and final external evaluations): - Venue costs - Advertising materials - Accommodation - Attendance fees and course fees 7.6.2 Levels of Allocation While there are no fixed rules for this, M&E budgets range from 2% to 15% of all costs. In projects, such as MARENASS in Peru and FODESA in Mali, where the stakeholders are exploring new ways of working together, M&E budgets are likely to be proportionally higher since more time is needed for reflection on what works. Table 7-12 shows the budget line for five Latin American projects, including MARENASS. Note that each project clusters its M&E costs in different ways due to the different approaches adopted. M&E in MARENASS is decentralised and sub-contracted. Therefore the project coordination itself does not need vehicles or materials for M&E. However, clustering all M&E costs under "studies" makes it necessary for the budget to be accompanied by a detailed M&E plan to ensure clarity about how the funds are to be allocated. Regardless of how the M&E budget is calculated, it will always overlap to some degree with other project activities. For example, about 78% of total annual funds for the MARENASS project goes directly to farmers management of their own community development plans, with the remainder going to two components: project management and M&E. Yet, even within these two components, many costs were related to community development such as festivals, communication bulletins and so on, raising this figure to 82-85%. Therefore, do not detail the M&E budget excessively. Much learning occurs through the normal interactions of project implementation. What is most important is to ensure inclusion in the budget of the events, procedures and staff time that support project learning and reflection. Many projects experience funding delays at start-up. To deal with this, financial advances are available from IFAD through the Special Operations Facility (SOF). These grants have proven to be extremely useful in expediting project start-up and implementation. The Philippine government, for example, administered such a grant to facilitate staff interview and recruitment, the start-up workshop and purchase of equipment. Other uses of SOF grants include engaging a project expediter to assist with fulfilment of conditions for loan effectiveness and financing training in management systems. Many projects use the SOF to cover initial M&E costs, such as M&E training for staff and partners and preparing baseline studies. Table 7-12. M&E budgets for five projects in Latin America, compared to total project budget, from appraisal reports (in United States dollars) Item Country and Project Venezuela Ecuador Peru Chile PROSALAFA PRODECOP SARAGURO MARENASS PRODECOP-IV Investments: jeep 4x4, computer equipment, furniture and office equipment 20,800 28,830 27,950   24,329 Running costs: salaries, travel, materials, maintenance, insurance 144,900 346,485 213,010   227,532 Studies: baseline, evaluation studies, documentation, monitoring, publications, taxes 205,000 540,000 119,916 1,770,000 330,815 Total M&E 370,700 915,315 360,876 1,770,000 582,676 Total project 26,742,700 24,365,000 16,753,781 19,142,800 34,491,969 Share of M&E 1.4% 3.8% 2.2% 9.2% 1.7% 7.6.3 Costs and Resources for Participatory M&E and Selected M&E Activities From a budget perspective, participatory learning processes are more time intensive than those in which only a few people are involved. Time is needed to organise meetings with larger numbers of people and more diverse groups and to reach agreement on how to proceed with M&E or on what data mean. Perhaps it will be necessary to invest in training both project and implementing partner staff in facilitation skills and also in training primary stakeholders in, for example, impact assessment methods and developing local indicators. Some projects have found that capacity-building on participatory M&E tasks can be undertaken on a cost-sharing basis, but usually the project bears the full cost. Specific budget items to consider for more participatory forms of M&E are: specific training for staff in participatory techniques and participatory M&E; extra meetings with stakeholders for designing M&E; additional meetings for local-level analysis; short training workshops on key steps in designing M&E and specific elements such as indicators and methods (including using the logframe matrix). Participatory projects also require ongoing planning processes to determine what will be implemented for whom and with whom. In cases where project activities emerge, the M&E budget needs similar flexibility. Therefore the budget could include a contingency line for significant but unanticipated opportunities. Often, the items in an M&E budget are sufficiently broad that they can be used for diverse types of M&E activities. For example, including a budget line "participatory impact assessment" leaves the project with much flexibility about when and how to undertake this. Boxes 7-20, 7-21 and 7-22 show examples of how to go about calculating costs for specific M&E activities: a baseline study, participatory impact monitoring and the management information system. Note that these costs are not intended to be indicative. Your costs will depend on context-specific factors, such as the size of the geographic area, local labour costs, sample size, type of methodology, use of external expertise, etc.   Box 7-20. Provincial baseline study in China (2000) Total cost = USD 13,173 Activity 1,200 household interviews in 37 project areas in seven counties; data entry cost = USD 10.98 per interview Personnel For interviews: two full-time staff per project area = 74 staff For data management: two full-time staff per county = 14 staff For managing/reporting: one province-level consultant Time schedule Interviews: two weeks Data entry: ten days Data checking, aggregating, analysing: two weeks Reporting: two weeks       Box 7-21. Introducing participatory impact monitoring (PIM) in Vietnam (2000) Total cost of introducing PIM per project = USD 15,000. This includes: consultants fees, training workshop materials and follow-up assistance. This cost can be reduced by: training and employing local experts to act as facilitators and consultants; integrating PIM activities into ongoing planning, extension and M&E activities; cleaning up existing data "cemeteries" and substituting more expressive impact-related information for fewer columns of figures; accepting reliable statements on trends rather than insisting on statistical accuracy.       Box 7-22. Costs for setting up the management information system in Nepal (2001) Total cost of setting up the management information system: NPR (Nepalese rupees) 1,719,239, spread over four years, with 43% of this for the first year, when procurement costs are high. Total expenditure, per item is: Server software 255,500 Email server software 73,000 Proxy server software 43,800 Network hub and cables 21,900 Installation 51,100 Internet dial-up charges 169,859 Dial-up networking phone charges from districts and project unit 636,970 Purchase of publications and media for library 212,323 Network maintenance contract 254,788 At the time of writing the document, the exchange rate was 1 USD per 75 NPR (April 2001).   Back to Top Further Reading Gubbels, P. and Koss, C. 2000. From the Roots Up: Strengthening Organizational Capacity through Guided Self-Assessment. (second edition). Oklahoma City: World Neighbors. Order ISNAR. 2001. Framework and Methodology on Training Needs and Organizational Constraints Assessment in Kenya. Download Lamy, E. and Lessard, P. 2001. Capacity Building: A Manual for NGOs and Field Workers. Quebec: CECI. Also available in French and Spanish. Order Lusthaus, C., M. Adrien. G. Anderson and F. Carden. 1999. Enhancing Organizational Performance. A Toolbox for Self-assessment. Ottawa: IDRC. Also available in French and Spanish. Order via: pub@idrc.ca. Lynton, R. and Pareek, U. 1990. Training for Development (second edition). Connecticut: Kumarian Press. Order 1/ UNDP. 1998. "Capacity assessment and development in a systems and strategic management context". Management Development and Governance Division, technical advisory paper No.3. New York, N.Y.: United Nations Development Program. 2/ Adapted from Guijt, I. and Gaventa, J. 1998. "Participatory Monitoring and Evaluation: Learning from Change". IDS Policy Briefing 12. Brighton: IDS. 3/ Abbot, J. and Guijt, I. 1998. "Changing Views on Change: Participatory Approaches to Monitoring the Environment". SARL Programme Discussion Paper No. 2. London: IIED. 4/ Lusthaus et al. 1999. See Further Reading. 5/ Adapted from Jackson, E. 1999. "The Strategic Choices of Stakeholders: Examining the Front-End Costs and Downstream Benefits of Participatory Evaluation". Presented to the World Bank Conference on Evaluation and Poverty Reduction. Washington, D.C.: World Bank. 6/ Furman, R. 2001. "Computerised Monitoring and Evaluation Systems: Design and Implementation Guidelines". In: A. Ocampo (ed.). Reflections on Project Monitoring and Evaluation Methodology. Rome: IFAD. Download PDF Version (263 KB) About Evaluation Programme of Work Strategy, Process & Methodology Evaluation Knowledge System A Guide for Project M&E
Back			Home		Next