Cuchumatanes Highlands Rural Development Project
Ministry of Agriculture, Livestock and Food – Government of Guatemala
(FIDA 296-GM, NL 296-GM, OPEP 580-P)
Ing. Agr. Jorge C. Piña Puig
International Coordinator – Cuchumatanes Project
October 2000

Introduction

The purpose of this document is to systematize the experience accrued in the Cuchumatanes Highlands Rural Development Project Monitoring and Evaluation System. This document describes the Monitoring and Evaluation System’s main characteristics and its contribution to the adjustment of project’s intervention strategy.

The manner in which the Monitoring and Evaluation System was set up was very important for the execution of the Cuchumatanes Project, more so in view of the fact that this project had complex execution characteristics; some of the most important are described below:

• Territorial: : the project covered a total area of 2 000 km^2. Since the vial network is poor, the area is poorly connected to the main centers of the country. As a result of this situation, during the first stage of the project, a main office and 5 secondary offices including a housing area for the technicians was set up in order to guarantee the Project’s territorial coverage. Under these conditions, the information system (result of the monitoring and evaluation system) gained a great importance.
• Agro ecological: the project has been implemented between 1 500 to 3 700 m.a.s.l in areas that have major differences in temperature and rainfall. Eight distinct production systems could be find ranging from coffee up to sheep breeding and different types of basic grains, fruits and vegetables. In this frame, the project had to implement a variety of proposals involving a diversity of technological, commercial and economic indicators.
• Ethnical and Social and Economic: the inhabitants of the area live in poverty, and in extreme poverty and register low indexes of human development. Six different ethnic-linguistic groups could be distinguished. The information system of the project had to reflect the ethnical diversity and levels of poverty of the target population.
• Institutional: During the execution of the project, public sector institutional changes were implemented as part of the structural adjustment process geared towards decentralizing public services. As a result of the foregoing, extension services were transferred to the beneficiaries´ grassroots organizations. Under this new framework, the need for a participatory and decentralized monitoring and evaluation system became undoubtedly evident.

The project

The Cuchumatanes Highlands Rural Development Project (Cuchumatanes Project) was executed by the Ministry of Agriculture, Livestock and Food (MAGA) of the Government of Guatemala. The Executing Unit of the Project (EUP), created for this purpose, was in charge of the project implementation.

The project area is located in the North West highlands of Guatemala. The Cuchumatanes Project is an initiative of the Government of Guatemala executed through the Ministry of Agriculture, Livestock and Food. The Project covers nine rural municipalities that belong to the Department of Huehuetenango. Target population live in poverty, extreme poverty and have low levels of human development. The main offices of the project were located in the Municipality of Chiantla.

The funds for the Cuchumatanes Project were provided by: the International Agricultural Development Fund – FIDA 296/GM; the Government of The Netherlands – grant NL 296/GM; the OPEP International Development Fund - OPEP 580/P; the World Food Program and, the Government of Guatemala.

The components of the project were as follows:

• Support for Production: involving the following sub components: validation and extension, rural credit, marketing, small irrigation, soil conservation and gender development.
• Social Organization: involving the following sub components: training, organization and social communication.
• Infrastructure: involving rural roads, soil protection and civil engineering construction.
• Management and Administration
• Monitoring and Evaluation.

The Overall Objective of the project was " to contribute to the generation of a dynamic self sustainable development process to boost production and the productive capacity of the peasant production in the area".

The Specific Objectives were: i) to expand cultivated areas and to raise productivity; ii) to diversify agricultural crops; iii) to improve the management of sheep breeding; iv) to facilitate access to productive resources, especially credit; v) to assist in setting up storage and marketing systems for products and inputs, vi) to provide adequate management of water and soil resources; vii) to encourage the participation of local inhabitants in the development process, particularly women, and; viii) to improve and expand the vial network.

As specified in the ex ante Evaluation (EEA) the strategies of the Project were restricted to improving agricultural and livestock production and productivity through services offered by governmental Agricultural and Livestock entities, although that at that moment, this entities had no presence in the area of the project. As a result of the experience gained by the execution process in the third year, the strategies of the project were reformulated as follows:

• Identifying and strengthening grassroots peasants organizations representing the production systems of each locality. The reinforcement of grass-root organizations implied, inter alia, the implementation of a training program directed to local human resources.
• Promoting the sustainable development process fostered by the project, by transferring the project’s services to the producers organizations.
• Promoting participatory and ongoing processes of diagnosis, planning and monitoring and evaluation-
• Translating users demands into integrated plans for the main products of each production system (technology, funding, marketing, agroindustry, environmental protection, roads and highways infrastructure).

The monitoring and evaluation system

The objectives of the Monitoring and Evaluation System in document EEA were as follows:

To provide information to top management regarding the fulfillment of the Project’s goals and objectives, in order to introduce the necessary modifications and justify the annual operational plans.

To provide the Project Management with reliable and updated information concerning the fulfillment of its goals and the results of its actions, to assist in decision making and, to adopt the corrective measures as deemed necessary.

To provide the participating institutions with useful information and experiences in order to enhance their managerial capacities.

To facilitate an inter.-institutional coordination as required for the execution of the project.

To ensure that beneficiaries of the Project participate in the analysis of project results and impacts..

To support national efforts geared towards consolidating a monitoring and evaluation system for the agricultural and livestock sector.

The Monitoring and Evaluation activities began in 1994 and, although in general terms they apparently responded to the EEA guidelines, the approach of the Monitoring and Evaluation System did not reap adequate results.

Functions foreseen for this initial period were: i)  to establish and maintain an information system, ii) to define performance an frequency of data collection, iii)   to design formats and other instruments to compile information, iv) to establish systematic procedures to prove the truthfulness and reliability of the information, v) to process and consolidate the information, to draft reports for the Government and financial entities, vi) to conduct evaluation surveys, in coordination with the MAGA Planning Unit and, vii) to establish consultation mechanisms with the beneficiaries.

The general approach of the Monitoring and Evaluation System was basically oversight and no participatory methodology was adopted. This resulted in the non fulfillment of the objectives proposed. As concerns the former, an external evaluation requested by the Executive Management concluded that the monitoring and evaluation component: i) did not satisfy the management information requirements, ii) the working methodology was not participatory and therefore hardly known, and partly overlooked by several UEP levels, iii) an oversight nature prevailed instead of a facilitator role, iv) the evaluation was geared towards an analysis of the staff’s performance, v) the information generated did not adequately reveal the progress made by the Project, and vi) the staff responsible for the monitoring and evaluation did not participate in drafting the physical and financial progress reports and the drafting of other reports was scarce and of a limited scope.

Despite the foregoing and its slack contribution regarding the performance of the Monitoring and Evaluation System, during this stage the following results were obtained:

• Initial Logical Framework: As a starting point of a logical monitoring and evaluation, a planning process was designed with ZOPP methodology. A Logical Framework Matrix was elaborated. This matrix defined the indicators required in order to monitor and evaluate the Project. However, the usefulness of this instrument was limited, since the logical framework indicators were not appropriate for monitoring purposes. Despite this situation, the drafting of this first version of the Logical Framework helped to review the EEA proposals and to acquire a much clearer understanding of the expected results regarding the project’s intervention.
• Baseline Study: This study was drafted by contracting a national NGO. It had a marginal benefit, both for the project as well as for the Monitoring and Evaluation Unit, basically because of the following factors: i) the monitoring and evaluation team had a poor understanding of the structure and contents that the report of the study should contain and this became evident in the Reference Terms drafted for the technical proposal bids, ii) the final product –the Baseline Study – is a statistical report made up of five volumes that basically contain quantitative data that is difficult to read and interpret; its usefulness to analyze the changes brought about throughout the execution of the project has not been confirmed yet.

One partially positive aspect was the attempt to identify the current family production systems in the area, despite the fact that only a preliminary identification was carried out and used as a basis for subsequent studies. The reasons for this limited result were, amongst others, a lack of preliminary information to design the survey’s sample and questionnaire.

• Reporting System: conducted by implementing and operating a computer system called Automatic Monitoring System -SAS-, especially developed for the project.

The SAS did not operate efficiently since the program had certain defects and it did not generate useful information for decision making purposes due to the fact that the indicators basically referred to the execution of activities (the activities undertaken were registered in a daily or weekly manner. Within this former framework, the reports generated by the SAS were not an input to draft the Project’s Progress Reports.

Despite the inefficiency of the SAS, its forward looking perspective was an important contribution to subsequently develop the Project Monitoring System. The version that the project implemented later (NSAS) salvaged and adjusted various characteristics from the first version.

• Participatory Evaluation: This evaluation was geared towards controlling the performance of the staff instead of the performance of the Project; the beneficiaries consulted gave their opinions regarding the attitudes and practices of the extentionists rather than about project results.

The technique of Focus Groups was adopted and the information obtained was presented in a sole and voluminous report entitled "Performance Evaluation". This study, clearly reflected the oversight nature of the monitoring system. Due to its characteristics and approach, the evaluation had a limited use to improve the project’s execution.

In order to supplement the "findings" of the Performance Evaluation, an additional evaluation was carried out to "verify the data" but, it actually accentuated the oversight nature of the monitoring and evaluation actions and, aggravated the reluctance of the project’s staff to participate in generating and processing data.

In practice, the working approach that had been adopted spurred a rejection by the different levels of the Project’s Executing Unit to monitoring and evaluation practices. The experience of this first stage revealed the ineffectiveness of monitoring and evaluation and its orientation towards control.

Reorientation of the monitoring and evaluation system

The Directorate of the project decided to reorient and reorganize the Monitoring and Evaluation System in 1996, in view of the poor performance up to date. The team in charge of reorienting the system was made up of a new coordinator of monitoring and evaluation and an international expert in this field.

The reorientation included defining a conceptual framework, adjusting the system’s objectives and redefining strategies and working methodologies. The reorientation process involved all the project staff, as well as the beneficiaries and peasant promoters. Information demands from financial institutions and Cooperating Institution (UNOPS) were also incorporated into the new monitoring and evaluation system.

The objectives of the new Monitoring and Evaluation System reflect a perspective that goes beyond the need to provide information for the project management levels. The approach and working method of the new system involved different stakeholders (beneficiaries, grassroots organizations, financial organizations, etc.) in compiling and analyzing information of the project’s strategies and results.

General Objective: To contribute towards achieving the project’s objectives by generating a systematic process of analysis concerning the project’s approach, its results and limitations.

Specific objectives

• To contribute to reinforce managerial capacities by implementing a monitoring and evaluation system that would help to establish mechanisms to regularly review and improve intervention strategies and decision making processes.
• To generate a system to compile, process, analyze information as concerns the progress made by the project, its effects and impacts.
• To create channels to allow different stakeholders related to the execution of the project to participate in systematic processes of review and reformulation of strategies.
• To strengthen capacities for monitoring and evaluation at a local level, by generating self evaluation processes geared towards contributing to the sustainability and effectiveness of the project.
• To lay the foundations for mid term and completion evaluations of the project in order to systematize the experience of the execution and extract lessons that will help to improve the formulation of future rural development projects.
• As has been mentioned previously, the objectives of the Monitoring and Evaluation System intend to go beyond managerial support, normally interpreted as the generation and supply of timely and relevant information. Managerial support, whose importance is undoubted, is a specific objective, among others, proposed by the Monitoring and Evaluation System.
• Two specific objectives of the monitoring and evaluation system, mentioned above, that are worthwhile remarking are those that propose the participation of the population and the strengthening of local skills for Monitoring and Evaluation. The latter bears a particular importance, since the formal organizations of produces are those that intermediate the services offered by the project (credit, marketing, technical assistance). Bearing in mind the role that these organizations play, they have to develop skills to produce, analyze and use information that will enable them to adopt their own decisions.

Strategies and Methodologies: As a counter balance to the predominating approach of the previous phase, during this period, the strategies and methodologies responded to a participatory and useful perspective for all the stakeholders of the project’s intervention process. Actions comprised in the system implied a two way feedback process. The new system took into account the comprehensiveness of the different components of the project. The working guidelines, axis of this new proposal were:

• Support to project components: in order to respond to the information needs of: the field staff, the coordinators of components, the Management and Directors Committees as well as other stakeholders, the monitoring and evaluation orientated its actions to support the implementation of other components of the project. This facilitated the execution and the feedback of field activities.
• Participatory Approach: since its inception the system proposed the active participation of all the stakeholders. The system conceived a decentralized scheme to compile and consolidate the data. Through this system, the information generated from the users was to be consolidated at higher levels. Thanks to this process, it was possible to know the origin of the information and the informants at all times. Participation was not restricted to the intervention of the beneficiaries in the generation and gathering of data, but also in feedback and decision making processes.
• Formative and Facilitating Orientation: geared towards developing and strengthening the capacity of the local stakeholders, so that in the short term, they become involved in the Monitoring and Evaluation System; and, in the long term they could improve their management and negotiation capacities. The facilitating role is interpreted as a support to compile, record, analyze and use data by means of instruments that are easy to use and interpret.
• Project Management Support: the system became the main source to draft reports for the Project’s managerial levels and also for the financial entities.
• Quantitative Qualitative Aspect -: in order to guarantee a comprehensive analysis of the information. This aspect is of utmost importance to have a balanced flow of information that facilitates an understanding of the progress made concerning the goals and achievements vis a vis the annual operational plans or global objectives of the project.
• Flexible and Multiple Techniques: due to the fact that the Project must adapt itself to a changing reality, derived from the external factors beyond the control of the Project, the Monitoring and Evaluation System should also have been dynamic in order to match the changes of the Project.

The Current Monitoring and Evaluation System

The Monitoring and Evaluation System of the Cuchumatanes Project is made up of the following elements:

1. Logical Framework and Formulation of Indicators

   Two Logical Frameworks have been prepared since the monitoring and evaluation project was reformulated in 1996. The first, in 1996 included a list of monitoring indicators, effects and impact. Later with the technical assistance of PREVAL and PROCASUR (1998), this Logical Framework was updated to include changes in the Project’s strategy, which in turn were generated by changes in the MAGA policies. The indicators were reviewed each year when the annual planning of the project was carried out.

2. Annual Operational Plan

   The Technical Sub Management was in charge of drafting the Annual Operational Plan. The monitoring and evaluation staff accomplished a facilitating and feedback role, derived from the Monitoring and Evaluation actions.

3. Monitoring System

This was made up of a system of reports, whose results are basically three: i) Monthly Reports, ii) Semester Reports and, iii) reports for Governmental entities. All these reports adopted as their main source of reference the annual operational plans of the components and of the co executing entities (cooperatives, peasant associations and other public and private institutions).

The Monitoring System involved i) the system’s users, ii) the data collection instruments, iii) a computer program to file and process the data and issue the reports (A New Automated Monitoring System (NSAS), iv) a data base of the beneficiaries, listed per gender, v) the training and updating activities of the users and, vi) the reports.

One important aspect to be stressed is that despite the fact that the System was innovative, it started to issue reports as of the second month of its initial design; a key factor were the different levels of training and awareness activities.

The problems faced by the project to regulate and optimize the functioning of the Monitoring System were varied: maintenance of the computer equipments, distinct levels of identification with the system, among others.

Users of the System and Data Collection Instruments. All the stakeholders of the Project were the users of the system They actively participated in generating and recording data and in drafting and using the reports. The main users were, among others, peasants promoters, the extensionists, the managers of the co executing entities, the coordinators of the components and sub components and, the Monitoring and Evaluation Unit. Each user had an ad-hoc instrument to compile information, specially tailored in correspondence with their working methodology and technical capacity. The sequence of generating and collecting data became a chain in which the users acted as links or integral parts.

• The peasant promoters (the first link in the chain): is a community member and has the responsibility of providing technical assistance to the beneficiaries (restricted to their knowledge and experience). Their role has changed over time in keeping with the skills they have acquired. The promoters received training and supervision from the Project and are currently contracted by the co executing peasant organizations. (remunerations and incentives).

The Promoter is the main link of the chain of the monitoring system. The promoter registers data concerning visits, training, demonstrations, and number of beneficiaries and/or assistants of each activity. These data are recorded in a NOTEBOOK designed by the project. The original information is left in the community for its use and a copy is transferred to the extensionist (the second link in the chain)

The Promoters Notebook is made up of the following parts: i) General information: geographic data, conversion tables, etc., ii)  Information on individuals, municipalities, the co executing organization and communities , iii)  a monthly schedule of activities, iv) a monthly record of activities, that includes a list of the beneficiaries per community, listed by gender and activity and, v)  a report on problems and observations.

• Extensionist (second link): The extensionist (originally contracted by the project; currently he/she is a staff member of the co executing peasant organizations), consolidates the information recorded in the Promoters Notebooks.

These records are kept in the Extensionist´s Notebook that was initially designed in a simple manner, but, afterwards it evolved into an agenda of activities whose format gradually was adapted to the recording needs for information..

The information was consolidated and recorded by the extensionist and inserted in the NSAS in each co executing organization which was made possible because each co executor has a computer. It could be remarked that extensionists are capable of generating their own reports and/or adding and processing supplementary indicators to those of the Monitoring System.

The Monitoring and Evaluation System envisioned a constant review and updating of the data recording instruments. This encouraged the participation of different stakeholders and overcame the oversight approach of the first stage.

• The Coordinators of Components and Sub Components (the third link): The Coordinators were in charge of recording monthly information, as specified in their working program; this was done in the computer of each coordinator, because the NSAS was installed in the UEP LAN network.

At one training course, practical demonstrations were included and the participants were able to prove that the average time used by each Coordinator to record the data was less than half an hour per month, including printing the reports.

• The Monitoring and Evaluation Unit (the fourth link): This Unit was in charge of: i) picking up the floppy disks with the data recorded by the extensionists and coordinators, ii) consolidating the data and generating the reports, iii) drafting and presenting the monthly and semester project progress reports to the Executive Committee, iv) evaluating the problems that arose after the monitoring system was launched, in a regular and participatory manner.

The NSAS Computer Program-: The NSAS was developed with the purpose of correcting certain programming problems of the original SAS and also to make it more "user friendly", in view of the fact that it must be used by different stakeholders, some of whom have scarce knowledge and experience in managing a computer. The NSAS operates in Windows and can be installed in a LAN network, but, it allows information to be recorded from several terminals (inside and outside the LAN network) and their consolidation in a main computer (multi-user).

This program was installed in the LAN network of the Project and in the cooperatives or associations that co execute the project services. It contains: i)  a planning module used to feed the system with indicators and goals of the operational plans ii) a monitoring module to regularly include the progress of each indicator, iii) a report module that allows to print or see the screen of the progress made by the operational plans and, iv) a consolidation module used to aggregate information of the period, generate back up files, transfer data base files to the floppy disks and send it to the monitoring and evaluation unit.

The continuity of the computer data processing facilitated accessing data for all those involved in the project, and it eliminated the need for written reports, at least for the three first links of the information chain, in addition to improving the capacity of the peasants organizations to administrate resources and make decisions.

Beneficiaries Data Bank and Gender Perspective: One of the key aspects approached when designing the Monitoring System was the need to obtain information broken down by gender of the beneficiary; another key aspect was the need to know the amount of users, men and women, who received specific services, avoiding a duplicity in the accumulated values.

This is why a decision was made to implement a computer data base to include the records of each beneficiary of the project’s services. The program was entitled Beneficiaries Data Bank; this allowed the users to record general data (name, gender, age, marital status, and identity document), location (municipality, community, sub headquarters), social organization and classification per production system, aside from their participation in the different services provided by the Project.

At present, the Beneficiaries Data Base is installed in fourteen co executors grassroots organizations.

Activities for raising awareness and training: These activities were of a great importance for the performance of the monitoring system because it helped to detect the users demands and difficulties in managing data and, on this basis, to define the contents of the training activities. Aside from the foregoing, these activities also helped the users to clearly understand the change of approach of the Monitoring and Evaluation System. The activities were as follows:

• Data management workshops: the purpose of these workshops was to learn about the user’s information demands in the field, to harmonize criteria concerning data recording and to strengthen the skills of the users as concerns managing formats to consolidate and analyze data. During the initial stage of the new system, these workshops fostered a closer contact with the field staff to disseminate the objectives of the System, to inform about its functions and, to raise awareness of the stakeholders concerning the need for and importance of their participation.

These workshops helped to establish a diagnosis of the monitoring and evaluation conditions at different levels of work as well as mechanisms to update the demands. During the implementation of the system, these workshops guaranteed that the monitoring indicators and evaluation contents of the different working instruments, matched users demands at all levels. A result of these workshops was the Monitoring notebooks for peasant promoters was designed.

• Monitoring visits: were scheduled to the sub offices of the Project and later, to the co executing organizations in order to secure the feedback of the data recording processes, to learn about the problems, qualitative aspects regarding the progress made by the Project and, to establish a continuous validation process of the instruments. This activity was very intensive during the first years in which the system was implemented but, later, it dropped when the staff learned how to use these instruments much better and improved their skills.

1. Reports: the main characteristics and the destination of the three reports mentioned above are described below:

• Monthly Monitoring Reports: also entitled Executive Report, submitted each month. The report (4 pages) explains the progress achieved in a series of chosen indicators on each component of the project. For each indicator the following was registered: annual goal and unit of measurement; prior absolute and relative progress; absolute and relative progress in the period reported and the absolute and relative accumulated progress. The report also presents the financial execution, broken down per component and funding source in absolute values and percentages.

This report also includes a section of the achievements and limitations extracted from the "monthly qualitative reports". The quantitative and qualitative flow of information to draft the report helped to obtain the consolidated data for the monthly progress report, between 5 to 8 days after the end of each month. The report was distributed in house the Project and also sent to MAGA and the Cooperating Institution. The report was studied each month by the members of the Executive Committee to learn about the progress made as well as the situation of each component.

• Physical and Financial Semester Progress Reports: these reports were part of the contractual agreements with the financial entities. Their structure evolved in keeping with the availability of information of the system, the index of the report is as follows: i) General aspects, that include the period of the report and background of the project, ii) Activities of the period: planning activities, organization of the project, major achievements of the project, achievements by expected results, financial execution and missions received, iii)  problems encountered and final comments that correspond to the Executive Management iv) Charts and Figures and , v) Annexes.

These reports were widely distributed and also delivered to the different institutions related to the Project.

• Reports for Governmental Institutions these reports were sent to the MAGA Monitoring and Evaluation Unit and the SEGEPLAN¹ Project Management each month.

Evaluation

The evaluation activities covered the following major lines:

• Community Participation Evaluation;
• Participatory Evaluation at the project team level,
• Special Evaluations (by contract),
• Occasional Evaluations (not programmed),
• Evaluation Reports

Community Participation Evaluation (conducted annually):

Self Evaluating/Participatory Workshops with Formal² Producers Organizations: its main purpose was to provide a space for thought and analysis concerning the project’s intervention results. In addition, the workshops proposed to facilitate the annual programming process at the level of formal organizations (those with legal status).

The initial methodology of the workshops envisaged group discussions on the support received by the Project, the relevance of this support; the achievements obtained; and an analysis of problems and recommendations in order to improve project interventions. The workshops focused on the institutional strengthening of the organizations by improving their management skills, decision making capacities and skills for the administration of technical and financial services. The workshops adopted a methodology directed and facilitated by the project; the methodology involved the presentation of a methodological guideline for group discussions. Group discussions were organized in two moments. In the first moment groups of beneficiaries and promoters, separately were conformed. In the second moments beneficiaries and promoters worked in mixed groups.

After a revision, the project adjusted the initial methodology of the workshops in order to facilitate its use by the organizations. Thus, a decision was made to adopt the SWOT³ technique. SWOT analyses were performed during the workshop in order to address the different aspects of the project intervention strategy. A special SWOT determining the situation of each of the organizations participating at the workshop, was also developed.

At present, workshops are held grouping the organizations according the type of service they offer to producers: i) financial intermediation organizations; ii) those that intermediate other services excluding financial intermediation; and iii) organizations that do not intermediate services or that have recently incorporated this working modality.

• Focus Groups: this activity was carried out in Women’s Groups (of the Community Banks Program) and with representatives of "Interest Groups" (non formal organizations that benefit from the project’s services , but do not intermediate these services).

In view of the large number of organizations that fall under this category, the project chose a sample based on three criteria : production system, time of relationship with the project, type of productive activities.

Through this technique, the monitoring and evaluation unit (the leading team of the focus groups) approached, in general terms, the same subjects and moments specified in the description of the Self Evaluating Workshops. The questions guideline adopted as a reference the strategy designed by the project for non formal organizations.

Unlike the Self Evaluating Workshops with the formal organizations, this activity was not transferred to the organization. This due to the fact that focus interviews require skilled personnel with knowledge of social research techniques and participatory facilitating methods.

The Workshops and Focal Groups were evaluated by the participants as specially useful for their work. During interviews with representatives of formal organizations, it was possible to detect that the information generated at the workshops is available in the files of the organizations and that it is being used in meetings and for decision making purposes.

One aspect that should be stressed is that the monitoring and evaluation unit in its effort to respect and not "contaminate" the participatory nature of these evaluation processes, restricted its role to organizational aspects: establishing working groups, delivering questions, summoning plenaries, etc. Under this framework,, group discussions were developed "without" external interference.. As a result, the monitoring and evaluation team did not encourage a major discussion concerning the key aspects of the project,

Participatory Evaluation at the project level: Held annually as a joint activity between the monitoring and evaluation unit and the Technical Sub Management of the Project. These were called "Self Evaluating Work Days" and made up of the following parts/moments:

• First part/moment: presentation and discussion of; I) the results obtained through the Self Evaluating Workshops and the Focus Groups, ii) the progress of the project, among other aspects.
• Second part/moment (group works and plenaries): Evaluation of institutional aspects: I) coordinating mechanisms among the components; ii) administrative aspects of the project and; iii) the monitoring and evaluation system. The SOWT analysis was used as the main technique at this moment. The results of a poll that the monitoring and evaluation unit carried out before the meeting among the field staff of the project was used as an input.
• Third part/moment: Guidelines to draft the Annual Operational Plans (POA's). Discussion based on a document drafted jointly by the Technical Sub Management and the monitoring and evaluation unit.

At the beginning, performing this activity exclusively involved the team of the project professionals. But, after the reformulation of the strategies of the project involving the transfer of services to formal organizations and the establishment of the Users Committee⁴, new participants were included⁵. This generated more and better contributions to the planning process, because the representatives of the grassroots organizations made remarkable contributions to define the planning guidelines. The participation of the beneficiaries at this level of decisions and the democratization of decision making based upon the actions of the Project, also strengthened the processes of understanding and adopting methodologies and instruments, not only as concerns monitoring and evaluation, but also those promoted by other components

Special Evaluations (by contract): Under this item an analysis of special subjects considered to be of importance for the project was included. The special evaluations carried out were:

• Environmental Impact Assessment
• Case Study on formal and non formal organizations
• Evaluation on the spans of roads and highways
• Credit Evaluation
• Evaluation on the adoption of technologies.

Occasional Evaluations (not programmed): These were quick evaluations conducted in response to a concrete need detected by USE or upon request of the Executive or Technical Committees. The occasional evaluations carried out were:

• Evaluation of the Community Stores (1996)
• Evaluation of the Sheep Auction (1997)
• Evaluation of the Nutritional Activities in the sub offices of Agua Alegre (1997)
• Evaluation of Community Banks (1997)
• Performance Evaluation of the Soloma sub office (1998)
• Efficiency Evaluation of the Todos Santos Cuchumatán Irrigation Systems (1998)

Evaluation Reports: these reports should have consolidated all the activities evaluated each year, including an analytical section describing the conclusions and recommendations. However, due to the constant work overload these were never prepared. The author considers that this activity should be outsourced through an external consultant who could also add contributions to improve the project’s intervention strategy in addition to the Monitoring and Evaluation System’s performance.

There are however, specific reports per activity (workshops, focus groups, etc.).

Monitoring and Evaluation Unit Organization, Staff and Logistics

The Monitoring and Evaluation Unit reported to the Executive Management of the Project. It was considered an advisory unit advisory unit.

The EEA document proposed allocating six people for monitoring and evaluation: i) a Coordinator, ii)  an Agricultural Economist, iii) a Agrarian Technician, iv) a Social Worker, v) a computer operator and, vi) a Secretary.

However, all the staff proposal was never contracted, and during the first stage ( from 1994 to 1996), the monitoring and evaluation team was comprised of:

• A Coordinator (a Veterinarian)
• A Social Technician (with a degree in Social Work)
• A Computer Technician (a program analyst)
• A Graphic Designer
• Half of this staff was dedicated to computer activities
• As of 1996 the composition of the team was changed and the computer functions were separated, thereby establishing a Computer Unit to provide the services for all the project. The monitoring and evaluation team was organized as follows:
• A Coordinator (Architect)
• An Agrarian Technician (Agrarian Engineer)
• A Social Technician (with a degree in Social Work)
• A Secretary

The functions of each team member were distributed according to their expertise, but special geographic areas of attention were also assigned thereby enabling each team member to fully learn the work of the Project and all the technical assistance needed as concerns monitoring and evaluation.

In view of the monitoring and evaluation unit work burden in relation to the staff assigned, several evaluation activities were carried out by consultants, particularly special evaluations. In these cases, the USE drafted the terms of reference and supervised the consultancies. The design of the computer programs was also outsourced.

As concerns the equipment, the USE permanently provided a jeep, two computers and on laptop aside from a full set of office furniture and equipment; during the peak period of field work, another vehicle was temporarily provided.

Lessons Learned

The following are some of the lessons learned from the experience of the Cuchumatanes project.

1. The dissemination and general acceptance of participatory evaluation methodologies apparently has minimized the importance of quantitative data. However, an integral knowledge of intervention effects requires the combination of qualitative and quantitative information. Moreover, the results of participatory evaluations are better analyzed and weighed if they are complemented with quantitative data. In this frame it can be concluded that M&E systems must integrate multiple techniques and methodologies in adequate proportions if its aim is to improve not only practices but also knowledge.
2. One aspect that is usually not foreseen in project design is that M&E activities, if appropriately addressed, could contribute to strengthen peasant’s grassroots organizations. This effect (or impact?) highly achieved through participatory monitoring and evaluation at the Cuchumatanes, was evaluated as a not foreseen but important outcome of the overall project..

No doubt that organizational strengthening through monitoring and evaluation is not easy to anticipate at the design stage, specially because prior knowledge about beneficiary organizations is needed. On the other hand, this outcome can not be achieved through a specific intervention or through one sole activity, but as a result of several S&E activities clearly geared towards achieving organizational strength.

1. The success of monitoring and evaluation depends mostly upon the participation of all the stakeholders in the design and implementation of the system. By applying a modern criteria, it could be ascertained that the users of monitoring and evaluation systems are all the participants in project implementation including, beneficiaries, field workers, management team, governmental and financial institutions, etc.

---

1/ The General Planning Secretariat of the Presidency of the Republic.

2/ UIT a legal status (cooperatives and associations)

3/ Strengths, Opportunities, Weaknesses and Threats.

4/ Made up by three representatives of formal organizations, two from interest groups and two from community banks.

5/ 14 UEP professionals and 28 peasant leaders participated in the 1999 UEP Self Evaluating Meeting