Toolbox Evaluation & Impact Assessment

Evaluation & Impact Assessment

TOOLBOX

Evaluation & Impact Assessment

This toolbox offers you training materials and good practice experiences on evaluation and impact assessment. We focus in particular on

We strive to include a wide variety of sources. Are your favourite materials not yet included? We would be delighted if you would share them with us.

Filter

569

Sort by:

Direction:

Download (88)

Monitoring, Evaluation and Learning for Resilience: A Companion Guide

Elsa Febles Carmona

Sebastian Thomas; Helen Jeans; Gina Castillo; Alexia Pretari; Claudia Geraets (contrib.)

Oxford: Oxfam (2018), 32 pp.

"This companion guide accompanies The Future is a Choice: The Oxfam Framework and Guidance for Resilient Development, providing practical guidance on how to design and implement resilience-sensitive monitoring, evaluation, accountability and learning (MEAL) systems within our interventions that aim ... more

Download (118)

Evaluating the impact of information campaigns in the field of migration: A systematic review of the evidence, and practical guidance

Jasper Tjaden, Sandra Morgenstern, Frank Laczko

Geneva: International Organization for Migration (IOM); Global Migration Data Analysis Centre (GMDAC) (2018), 50 pp.

"Information campaigns designed to raise awareness of the potential risks of (irregular) migration have attracted much attention and investment across the world in recent years. Studies have repeatedly shown that many migrants start their journeys with limited or biased information and end up in vul ... nerable situations. In response, information campaigns have increased in number and the type, format, messages and strategy of such campaigns have diversified. This report presents the results from a systematic literature review of evaluations of such information campaigns in the field of migration. The study reveals that the evidence base available for programming and policymaking in this area is strikingly limited. We find that the uptake in the use of information campaigns has far outpaced any rigorous assessment of the effects that different campaigns may have on their respective target groups. In the absence of reliable evidence, the debate on the potential of this policy tool often relies on largely anecdotal evidence. Better evidence can show how information campaigns can be designed to best achieve their intended effects given the particular circumstances. The current lack of evidence limits the impact of future campaigns. While rigorous assessment of campaign impact can be difficult and costly, better evidence is clearly needed – wherever feasible and appropriate. Based on an extensive, systematic literature review, 60 relevant evaluations of information campaigns that targeted potential migrants and traffickers, as well as communities at large, were identified from a pool of 3,600 records. Only 30 of the selected campaign evaluations had publicly available results; the rest were collected through expert referrals. Two studies were published in peer-reviewed journals. Among the campaigns featured in the 60 studies, the most popular communication tools were workshop-type activities and cable TV programmes/advertisements. Most campaigns focused on trafficking, followed by irregular migration and, more generally, smuggling. A common issue is the lack of a clearly defined campaign objective and/or target group. This hampers any rigorous evaluation of programme effects. Whenever an objective is defined, it is most often aimed at “awareness-raising” and “knowledge generation.” The majority of the campaign evaluations claimed that the campaign under study was “successful” in inducing a change in knowledge, perceptions, attitudes and – to a lesser degree – (intended) behaviour. However, most of the evaluations reviewed provided relatively little evidence of the full impact of information campaigns. While many of the evaluations reported the number and profiles of campaign recipients or beneficiaries, impact was not directly measured. (In the evaluation literature, impact is defined as a change in outcome that is directly attributable to the programme and not any other factor.) Most of the evaluations identified did not meet minimum standards for robust evidence on programme effects. The large majority of evaluations were based on based on cross-sectional surveys of small numbers of participants (N) sampled at convenience, limiting the generalizability of the results. Only a few large-N studies employed a control-group design or involved pre- and post-measurements. None employed a (quasi-)experimental method for causal inference (e.g. randomized controlled trial, which is considered the “gold standard” for measuring impact)." (Executive summary) more

External Link (334)

Dejar huella: Historias de evaluaciones que marcaron la diferencia

Esteban Tapella

San Juan (AR): Editorial Universidad Nacional de San Juan (UNSJ) (2018), 188 pp.

"En los casos trabajados resulta muy relevante la capacidad técnica y competencias del equipo evaluador. Las diversas dimensiones que se encuentran al interior de dicho tema trascienden la imprescindible rigurosidad técnica en el desarrollo de la evaluación misma, sin la cual no habría impacto ... positivo alguno. Por ejemplo, en la evaluación del programa de México resultó clave el acercamiento de tipo antropológico de los evaluadores, mientras que en el caso de UNICEF en Argentina fue muy importante el carácter interdisciplinario del equipo de evaluación. Más allá de la rigurosidad técnica, cobra importancia creciente la comunicación de los resultados de la evaluación a los diferentes actores relevantes involucrados. Esta instancia de comunicación presta atención al tipo de reporte utilizado, la adecuación del lenguaje para distintas audiencias y la capacidad de generar, cuando corresponda, lecciones aprendidas aptas para articularse con la realidad organizacional respectiva. Vinculado a ello, aparece una tensión recurrente en la evaluación de programas: la que presenta el binomio participación / autonomía de la evaluación. A menudo los evaluadores se esfuerzan por mantener una cierta distancia sobre el programa evaluado a fin de tratar de proteger su independencia, incrementando de ese modo las posibilidades que la evaluación resulte distante e irrelevante para aquellos que actuarán sobre sus resultados a fin de realizar mejoras y facilitar el cambio. La cercanía con el programa evaluado y sus actores permite aprovechar oportunidades para marcar una diferencia a través del uso de procesos. Esto apunta a reconocer que los beneficios e impactos de la evaluación se desprenden tanto, o aún más, del modo en cómo ella se realiza (utilidad del proceso) que simplemente en relación a los hallazgos de la misma (utilidad de los resultados). Esta situación la ilustra el proceso de autoevaluación de la Universidad Nacional de Lanús (UNLa), el cual favoreció el involucramiento de una multiplicidad de actores de la comunidad universitaria, lo que permitió el acceso democrático a la información, la interacción novedosa entre muchos de ellos, la discusión y el debate, así como el fortalecimiento de capacidades. Las historias de evaluación relevadas en el proyecto ilustran a su vez el valor de un enfoque positivo. Vale decir, ponen el énfasis en la evidencia de aquello que funciona y/o que puede funcionar y que amerita ser continuado, ampliado o modificado. Esto contrasta con evaluaciones que acentúan principalmente los problemas técnicos y deficiencias varias presentes en todo programa. Un enfoque de este tipo ha mostrado ser altamente relevante a fin de trascender las actitudes de defensa y sospecha que suelen asociarse a toda práctica evaluativa, al movilizar la discusión hacia el campo de las soluciones. Ello no implica obviar la necesidad de cambio, sino que más bien apunta a profundizar las recomendaciones al señalar el tipo de cambio necesario y su porqué. Por ejemplo, la evaluación de la iniciativa Fortalecimiento de las capacidades de las mujeres indígenas para concertar e incidir en la implementación de políticas públicas (Colombia) validó muchos de los enfoques adoptados y señaló las potencialidades que permitiría desarrollar el contar con una línea de capacitación adicional a los objetivos planteados originalmente en el proyecto. Junto a ello, las evaluaciones con este enfoque positivo suelen caracterizarse por actitudes de acompañamiento, comprensión y cercanía a los actores de la intervención, ya que se evita limitar su tarea al señalamiento de cuestiones a modificar, lo que es muchas veces leído por los actores locales como “buena disposición” de los evaluadores. La participación de los actores en la misma recopilación y el uso de datos de la evaluación son dos poderosas maneras de involucrar a los usuarios y beneficiarios. Ello permite que los participantes aumenten su apropiación de los datos, situación que los empodera para responsabilizarse de sus resultados y el uso de los mismos. Una evaluación caracterizada por un enfoque colaborativo lleva a los participantes a hacerse cargo de la misma y, desde allí, del cambio o transformación que le sigue. De este modo, la participación activa en el proceso de evaluación ayuda a desarrollar una mejor comprensión de la evaluación, lo que contribuye a su utilización. Esto lo ilustra también la evaluación participativa en Costa Rica, en la que los equipos técnicos regionales se involucraron y tuvieron profundo interés en conocer cómo funcionaba el programa evaluado en su territorio, mientras que las autoridades de mayor nivel se limitaron a brindar su visto bueno a la evaluación. De este modo, las recomendaciones a nivel de territorio o región se pudieron aplicar al poco tiempo de terminada la evaluación, mientras que aquellas recomendaciones generales -dependientes de las autoridades de más alto nivel- aún no se aplicaron." (páginas 17-21) more

External Link (249)

Google Scholar (263)

WorldCat (253)

„Mehr Wert für eine bessere Welt?“ Evaluation und Controlling der Kommunikation von Nichtregierungsorganisationen (NGO)

Michael Bürker

In: Handbuch NGO-Kommunikation

Nadine Remus, Lars Rademacher (eds.)

Wiesbaden: Springer VS (2018), pp. 461-487

"Zunächst werden theoretische Konzepte und Modelle sowie die empirische Forschung zum Einsatz von Evaluation und Controlling für die NGO-Kommunikation gesichtet. Dabei werden auch Konzepte für die Evaluation von Organisationen sowie für die Kommunikation von Nonprofit-Organisationen (NPO) als al ... more

External Link (257)

Google Scholar (267)

WorldCat (267)

Vertrauensforschung für NGOs: NGO-Trust Index und NGO-Trust Map

Christine Viertmann, Julia Wölfer

In: Handbuch NGO-Kommunikation

Nadine Remus, Lars Rademacher (eds.)

Wiesbaden: Springer VS (2018), pp. 443-460

"Dieser Beitrag widmet sich dem medienvermittelten Vertrauen in NGOs. Ein auf Basis der Theorie des öffentlichen Vertrauens (Bentele 1994) entwickeltes, inhaltsanalytisches Messinstrument, der NGO-Trust Index (NGO-TI) und die Ergebnis-Visualisierung in der NGO-Trust Map (Wohlgemuth et al. 2013), we ... more

Google Scholar (269)

WorldCat (251)

Monitoring Media Sustainability: Economic and Business Revisions to Development Indicators

Robert G. Picard

In: What is Sustainable Journalism? Integrating the Environmental, Social, and Economic Challenges of Journalism

Peter Berglez, Ulrika Olausson, Mart Ots (eds.)

New York et al.: Peter Lang (2018), pp. 243-256

"Robert G. Picard describes the evolvement of UNESCO's media development indicators. The chapter describes a growing focus on economic, financial and managerial dimensions, since, it argues, they pave the fundament to any sustainable, commercial or non-commercial journalistic venture. What Picard cr ... more

Google Scholar (268)

WorldCat (272)

Evaluation Practice for Collaborative Growth: A Guide to Program Evaluation with Stakeholders and Communities

Lori L. Bakken

New York: Oxford University Press (2018), xvi, 236 pp.

"Evaluation Practice for Collaborative Growth highlights the approaches, tools, and techniques that are most useful for evaluating educational and social service programs. This book walks the reader through a process of creating answerable evaluations questions, designing evaluation studies to answe ... more

Download (88)

Journalism Impact Primer & Journalism Impact Toolkit
Key Guides

Walton Family Foundation (2018), 25 pp., 21 pp.

"This report puts forth a Walton Family Foundation (WFF) Media Impact Framework and identifies key indicators and research methods that can be used to assess journalism investments across program areas, based on best practices in the field of media strategy and research. The companion WFF Media Impa ... more

External Link (265)

Leaving a Footprint: Stories of Evaluation That Made a Difference

Pablo Rodríguez Bilella, Esteban Tapella

San Juan (AR): Editorial de la Universidad Nacional de San Juan (UNSJ) (2018), 186 pp.

"The evaluation stories revealed in the project illustrate the value of a positive approach. They emphasize the evidence of what works and/or might work and is worthy of being continued, enlarged or modified. This can be contrasted with evaluations that are focused mainly on the technical problems a ... nd deficiencies present in most interventions. This type of approach has been shown to be highly relevant in getting beyond defensive and suspicious attitudes, and instead promoting a constructive focus on possible solutions. For example, the evaluation of the initiative Strengthening the Abilities of Indigenous Women to Set up and Have a Bearing on the Implementation of Public Policies (Colombia) validated many of the approaches adopted, and indicated the potential that could be developed by having an additional training to the objectives outlined originally in the project. In addition to that, in evaluations with a positive approach, evaluators usually develop a close relationship with the actors of the intervention, understanding them and supporting them. The evaluators' task in these cases is not limited to indicating what to be modified. Many times local actors regard this as evaluators demonstrating ‘commitment’ to the project and its future. Participation of the actors in the collection and use of the evaluation data is a powerful way of including users and beneficiaries. This allows participants to get involved and understand the data better. An evaluation characterised by a collaborative approach leads participants to take responsibility for the evaluation and then for the change and transformation that follows. In this way, active participation in the evaluation process helps to develop better understanding of evaluation and contributes to commitment and use. This is illustrated by the participatory evaluation in Costa Rica, in which regional technical teams were involved and deeply interested in getting to know how the evaluated program worked in their area. In contrast, the higher authorities limited their participation to approving the evaluation. In this way, recommendations at regional and local levels were applied soon after the evaluation finished, whereas general recommendations —dependent upon the higher authorities— have not yet been applied." (Introduction, page 21-22) more

External Link (280)

Evaluating C4D Resource Hub

RMIT University; University of Hyderabad; UNICEF Communication for Development (C4D) (2018)

"The Evaluating C4D Resource Hub houses a growing collection of the best available guides, toolkits, tools and methods to use for research monitoring and evaluation (R,M&E) of Communication for Development (C4D) initiatives. The Hub is structured around two combined frameworks: C4D Evaluation Framew ... more

External Link (307)

Google Scholar (317)

WorldCat (252)

Evaluating Public Communication: Exploring New Models, Standards, and Best Practice

Jim Macnamara

London; New York: Routledge (2018), xix, 392 pp.

"Chapters feature: A review of 30 frameworks and models that inform processes for evaluation in advertising, public relations, health communication and promotion, government communication and other specialist fields including the latest recommendations of industry bodies, evaluation councils, and re ... more

Download (79)

Civic Media Practice: Identification and Evaluation of Media and Technology That Facilitates Democratic Process

Eric Gordon, Gabriel Mugar

MaxArthur Foundation (2018), 34 pp.

"Based on over 40 interviews with practitioners, this report identifies “civic media practice” as media and technology used to facilitate democratic process. It focuses specifically on those practitioners using media tools to form relationships and build trust - a practice that sometimes runs co ... more

Download (74)

NGO-IDEAs Toolbox for Participatory Monitoring of Outcomes and Impacts

Eberhard Gohl, Rosa Mendoza, Dagny Skarwan

Lima: Proyecto ONG-IDEAs Latinoamérica (2018), 104 pp.

External Link (186)

Participatory Evaluation

INTRAC (2017), 3 pp.

"A participatory evaluation actively involves key stakeholders, especially the intended beneficiaries of a project or programme, in the design and implementation of the evaluation. Participatory evaluations can be carried out for many reasons. The two most common are to empower beneficiaries to bett ... more

External Link (252)

Most Significant Change

INTRAC (2017), 4 pp.

"The Most Significant Change (MSC) technique is a form of participatory monitoring and evaluation. It involves the collection and selection of stories of change, produced by programme or project stakeholders. MSC can be used in projects and programmes where it is not possible to precisely predict de ... more

External Link (191)

Process Tracing

Nigel Simister, Vera Scholz

INTRAC (2017), 5 pp.

"Process tracing is a qualitative analysis methodology. The main purpose of process tracing is to establish whether, and how, a potential cause or causes influenced a specified change or set of changes. This is done by applying formal tests to examine the strength of evidence linking potential cause ... more

External Link (186)

Impact Grids

Nigel Simister

INTRAC (2017), 3 pp.

"An impact grid is a tool that helps stakeholders involved in a development intervention to analyse multiple changes by mapping those changes onto a grid. A typical grid charts the extent of change alongside the degree of contribution of the intervention. Impact grids can be used for many different ... more

Download (127)

The Step-by-Step Guide to Evaluation: How to Become Savvy Evaluation Consumers

Kellogg Foundation (2017), ix, 251 pp.

"At the W.K. Kellogg Foundation, we believe that evaluation is an effective management tool to both inform strategy development and track the progress and impact of strategy implementation. We have long been committed to supporting our grantees’ ability to derive and share lessons learned from the ... more

External Link (255)

Developmental Evaluation

Nigel Simister

INTRAC (2017), 3 pp.

"Developmental evaluation involves long-term relationships between evaluators and project or programme staff. Evaluation is ongoing, which means that feedback can be provided on a continuous basis. Development evaluation is particularly appropriate for projects or programmes working in complex or un ... more

Download (85)

Impact Evaluation of Development Interventions: A Practical Guide

Howard White, David A. Raitzer

Manila: Asian Development Bank (ADB) (2017), ix, 202 pp.

Journalism Impact Primer & Journalism Impact Toolkit Key Guides

Journalism Impact Primer & Journalism Impact Toolkit
Key Guides