Toolbox Evaluation & Impact Assessment

Evaluation & Impact Assessment

TOOLBOX

Filter

194

Sort by:

Direction:

Download (109)

How to Perform Impact Assessments: Key Steps for Assessing Communication Interventions

James Dennison

International Centre for Migration Policy Development (ICMPD); EUROMED Migration (2020), 29 pp.

"This report brings together disparate terminology, findings and recommendations from the private and public sectors and academia to synthesise a set of five general steps for practitioners when performing impact assessments. Impact assessment (IA) compares a variable of interest after an interventi ... on (e.g. a communication campaign) to what it would it be if that intervention had never happened. Accompanying each of the five steps are examples of how migration communicators can assess the impact of campaigns to, first, change attitudes and, second, change behaviour. Step 1: Set objectives for the intervention that define what the desired effect is. Ideally, this should (1) meet the SMART criteria of specific, measurable, attainable, relevant and timebound; (2) focus on ultimate outcomes, rather than only outputs; (3) avoid a number of common mistakes in setting objectives and (4) carefully consider what the—as specific as possible— target audience is. Step 2: Identify a measure of the impact(s) that the intervention has as its objective. These should be (1) clearly defined and unambiguous; (2) externally valid (they act reliably if used for other interventions) and (3) internally valid (they measure what they claim to measure). There already exist long-lists of measures (also known as indicators) to choose from. Step 3: Estimate what would have happened had there been no intervention, known as a ‘counterfactual’. Ideally, but not necessarily, this requires performing a pre-intervention measurement, also known as baseline assessment. However, this is not always possible. This should not dissuade practitioners but instead assumptions of any IA, as discussed below, should be made openly. A long list of methods for measurement, including sampling considerations, are provided. Step 4: Perform the intervention. This may involve separating a randomised sample into a treatment group(s)—that receives the intervention(s)—and a control group—that does not. If this is not feasible, other forms of counterfactual are possible that ‘construct’ a control group. This section also overviews: (1) types of communication campaigns; (2) the MINDSPACE checklist of behavioural influences for interventions; (3) key recommendations on persuasive interventions from ICMPD’s previous work; (4) recommendations on campaigns deterring irregular migration. Step 5: Post-intervention measurement and analysis: (1) those receiving the intervention should be measured post-intervention (ideally, as well as, either a genuine or constructed control group); (2) the impact must be calculated: usually the difference in the variable of interest pre- and post-intervention and/or between the treatment and control group; (3) theoretical consideration of why and what aspect of the campaign caused an impact; are the findings likely to be universalizable?; (4) creation (and, in some cases, incorporation) of recommendations based on combination of impact and theoretical considerations." (Executive summary) more

Download (116)

Decoding Media Impact: Insights, Advice and Recommendations

Katie Donnelly

Philadelphia, Penns.: Media Impact Funders (2020), 31 pp.

"This guide is meant to serve as a practical resource for funders who want to understand where to start. Informed by feedback from our network, it represents a synthesis of the past seven years of work we’ve done in the impact space, and includes examples of successful media impact evaluation, too ... more

External Link (273)

Measuring Beyond Money: The Media Viability Indicators (MVIs)

Laura Moore, Ann Hollfield, Nadine Jurrat, Gerwin de Roy

Bonn: Deutsche Welle DW Akademie (2020), 7 pp.

"This paper presents a new set of indicators to assess the viability of news media at a time when the discussion about the survival of quality media is omnipresent, while systematic, fact-based strategies to tackle the problem are omni-absent. As digital transformation is sweeping through the media ... more

Google Scholar (258)

WorldCat (232)

Public Diplomacy: Messung, Entstehung und Gestaltung von Landesimages

Diana Ingenhoff, Alexander Buhmann

Köln: Herbert von Halem Verlag (2019), 269 pp.

"Diana Ingenhoff und Alexander Buhmann führen in den aktuellen Forschungs- und Wissensstand zu Public Diplomacy und insbesondere zu Landesimages ein. Sie reflektieren dabei Fragen der Messung, Entstehung und Gestaltung von Landesimages und geben Antworten auf die folgenden Leitfragen: Welche Aspekt ... more

Download (115)

Guía práctica y herramientas para el impacto: Del arte al impacto
Key Guides

Sahar Driver

London et al.: Doc Society (2019), 237 pp.

"El objetivo de esta guía va más allá de presentar herramientas y recursos, pretende crear comunidad. Igual que sucede con el proceso de evaluación, vimos una oportunidad para aprender cómo se produce el impacto y hacer un seguimiento de cómo las películas están catalizando el cambio social. ... more

Download (110)

O guia de campo: Kit der ferramentas para o impacto. Da arte ao impacto
Key Guides

Sahar Driver

London et al.: Doc Society (2019), 238 pp.

"Nós oferecemos ideias e abordagens para que você não tenha que lidar com um obstáculo que alguém já ultrapassou e, com isso, concentrar-se nos desafios próprios de seu projeto. Disponibilizamos todas as nossas colaborações, conversas, acordos e desentendimentos com equipes de filmagem incr ... more

Download (115)

UNESCO’s Internet Universality Indicators: A Framework for Assessing Internet Development
Key Guides

David Souter, Anri van der Spuy

Paris: UNESCO (2019), 198 pp.

"The four principles identified as key to Internet Universality are summarised as the R-O-A-M principles, and are fundamental to the development of the Internet in ways that are conducive to achieving the Sustainable Development Goals with no one left behind. These principles are: R – that the int ... more

External Link (284)

Google Scholar (295)

WorldCat (297)

Evaluations and Impact Assessments in Communication for Development

Lauren Kogen

In: Handbook of Communication for Development and Social Change

Jan Servaes (ed.)

Singapore: Springer (2019), 17 pp.

"Within the development field, project evaluations and impact assessments are essential. Donors are increasingly requiring rigorous evaluations in order to (1) ensure that aid dollars are spent on projects that are having positive impacts and not being wasted on projects that are ineffective and (2) ... more

Download (104)

The Impact Field Guide and Toolkit: From Art to Impact
Key Guides

Sahar Driver

London et al.: Doc Society, 2nd ed. (2019), 381 pp.

"This guide is about more than just tools and resources, it’s about building community. As with the process of evaluation, we saw an opportunity to learn about how impact happens and to track how films are catalysing social change. That’s the impetus behind the case studies that make up our Impa ... more

Download (87)

El buen-convivir y bien-transformar en la comunicación: Sistematización de prácticas innovadoras para el Buen Vivir

Adalid Contreras

Quito: Signis América Latina y Caribe (2019), 204 pp.

"Hemos estructurado el documento en tres grandes partes. La primera es una lectura del Vivir Bien / Buen Vivir desde las prácticas de las experiencias sistematizadas. Es un esfuerzo por conceptualizar la Comunicación para el Vivir Bien / Buen Vivir desde las prácticas vividas. Siguiendo las narra ... more

Download (89)

The Impact Field Guide and Toolkit: From Art to Impact [in Arabic]
Key Guides

Sahar Driver

London et al.: Doc Society, 2nd ed. (2019), 238 pp.

Download (118)

Evaluating the impact of information campaigns in the field of migration: A systematic review of the evidence, and practical guidance

Jasper Tjaden, Sandra Morgenstern, Frank Laczko

Geneva: International Organization for Migration (IOM); Global Migration Data Analysis Centre (GMDAC) (2018), 50 pp.

"Information campaigns designed to raise awareness of the potential risks of (irregular) migration have attracted much attention and investment across the world in recent years. Studies have repeatedly shown that many migrants start their journeys with limited or biased information and end up in vul ... nerable situations. In response, information campaigns have increased in number and the type, format, messages and strategy of such campaigns have diversified. This report presents the results from a systematic literature review of evaluations of such information campaigns in the field of migration. The study reveals that the evidence base available for programming and policymaking in this area is strikingly limited. We find that the uptake in the use of information campaigns has far outpaced any rigorous assessment of the effects that different campaigns may have on their respective target groups. In the absence of reliable evidence, the debate on the potential of this policy tool often relies on largely anecdotal evidence. Better evidence can show how information campaigns can be designed to best achieve their intended effects given the particular circumstances. The current lack of evidence limits the impact of future campaigns. While rigorous assessment of campaign impact can be difficult and costly, better evidence is clearly needed – wherever feasible and appropriate. Based on an extensive, systematic literature review, 60 relevant evaluations of information campaigns that targeted potential migrants and traffickers, as well as communities at large, were identified from a pool of 3,600 records. Only 30 of the selected campaign evaluations had publicly available results; the rest were collected through expert referrals. Two studies were published in peer-reviewed journals. Among the campaigns featured in the 60 studies, the most popular communication tools were workshop-type activities and cable TV programmes/advertisements. Most campaigns focused on trafficking, followed by irregular migration and, more generally, smuggling. A common issue is the lack of a clearly defined campaign objective and/or target group. This hampers any rigorous evaluation of programme effects. Whenever an objective is defined, it is most often aimed at “awareness-raising” and “knowledge generation.” The majority of the campaign evaluations claimed that the campaign under study was “successful” in inducing a change in knowledge, perceptions, attitudes and – to a lesser degree – (intended) behaviour. However, most of the evaluations reviewed provided relatively little evidence of the full impact of information campaigns. While many of the evaluations reported the number and profiles of campaign recipients or beneficiaries, impact was not directly measured. (In the evaluation literature, impact is defined as a change in outcome that is directly attributable to the programme and not any other factor.) Most of the evaluations identified did not meet minimum standards for robust evidence on programme effects. The large majority of evaluations were based on based on cross-sectional surveys of small numbers of participants (N) sampled at convenience, limiting the generalizability of the results. Only a few large-N studies employed a control-group design or involved pre- and post-measurements. None employed a (quasi-)experimental method for causal inference (e.g. randomized controlled trial, which is considered the “gold standard” for measuring impact)." (Executive summary) more

External Link (249)

Google Scholar (263)

WorldCat (253)

„Mehr Wert für eine bessere Welt?“ Evaluation und Controlling der Kommunikation von Nichtregierungsorganisationen (NGO)

Michael Bürker

In: Handbuch NGO-Kommunikation

Nadine Remus, Lars Rademacher (eds.)

Wiesbaden: Springer VS (2018), pp. 461-487

"Zunächst werden theoretische Konzepte und Modelle sowie die empirische Forschung zum Einsatz von Evaluation und Controlling für die NGO-Kommunikation gesichtet. Dabei werden auch Konzepte für die Evaluation von Organisationen sowie für die Kommunikation von Nonprofit-Organisationen (NPO) als al ... more

External Link (257)

Google Scholar (267)

WorldCat (267)

Vertrauensforschung für NGOs: NGO-Trust Index und NGO-Trust Map

Christine Viertmann, Julia Wölfer

In: Handbuch NGO-Kommunikation

Nadine Remus, Lars Rademacher (eds.)

Wiesbaden: Springer VS (2018), pp. 443-460

"Dieser Beitrag widmet sich dem medienvermittelten Vertrauen in NGOs. Ein auf Basis der Theorie des öffentlichen Vertrauens (Bentele 1994) entwickeltes, inhaltsanalytisches Messinstrument, der NGO-Trust Index (NGO-TI) und die Ergebnis-Visualisierung in der NGO-Trust Map (Wohlgemuth et al. 2013), we ... more

Google Scholar (269)

WorldCat (251)

Monitoring Media Sustainability: Economic and Business Revisions to Development Indicators

Robert G. Picard

In: What is Sustainable Journalism? Integrating the Environmental, Social, and Economic Challenges of Journalism

Peter Berglez, Ulrika Olausson, Mart Ots (eds.)

New York et al.: Peter Lang (2018), pp. 243-256

"Robert G. Picard describes the evolvement of UNESCO's media development indicators. The chapter describes a growing focus on economic, financial and managerial dimensions, since, it argues, they pave the fundament to any sustainable, commercial or non-commercial journalistic venture. What Picard cr ... more

Download (88)

Journalism Impact Primer & Journalism Impact Toolkit
Key Guides

Walton Family Foundation (2018), 25 pp., 21 pp.

"This report puts forth a Walton Family Foundation (WFF) Media Impact Framework and identifies key indicators and research methods that can be used to assess journalism investments across program areas, based on best practices in the field of media strategy and research. The companion WFF Media Impa ... more

External Link (280)

Evaluating C4D Resource Hub

RMIT University; University of Hyderabad; UNICEF Communication for Development (C4D) (2018)

"The Evaluating C4D Resource Hub houses a growing collection of the best available guides, toolkits, tools and methods to use for research monitoring and evaluation (R,M&E) of Communication for Development (C4D) initiatives. The Hub is structured around two combined frameworks: C4D Evaluation Framew ... more

External Link (307)

Google Scholar (317)

WorldCat (252)

Evaluating Public Communication: Exploring New Models, Standards, and Best Practice

Jim Macnamara

London; New York: Routledge (2018), xix, 392 pp.

"Chapters feature: A review of 30 frameworks and models that inform processes for evaluation in advertising, public relations, health communication and promotion, government communication and other specialist fields including the latest recommendations of industry bodies, evaluation councils, and re ... more

Download (79)

Civic Media Practice: Identification and Evaluation of Media and Technology That Facilitates Democratic Process

Eric Gordon, Gabriel Mugar

MaxArthur Foundation (2018), 34 pp.

"Based on over 40 interviews with practitioners, this report identifies “civic media practice” as media and technology used to facilitate democratic process. It focuses specifically on those practitioners using media tools to form relationships and build trust - a practice that sometimes runs co ... more

External Link (252)

Evaluation Approaches for Mental Health Prevention and Early Intervention Programs

Nicole K. Eberhart, Jennifer L. Cerully, Amy L. Shearer, Sandra H. Berry, M. Audrey Burnam, Patricia A. Ebener

Santa Monica, Calif.: RAND Corporation (2017), ix, 81 pp.

"The goal of this handbook is to provide guidance to practitioners and organizations interested in evaluating mental health prevention and early intervention (PEI) programs. In the course of developing an evaluation plan for a series of PEI programs implemented statewide by the California Mental Hea ... more

Guía práctica y herramientas para el impacto: Del arte al impacto Key Guides

O guia de campo: Kit der ferramentas para o impacto. Da arte ao impacto Key Guides

UNESCO’s Internet Universality Indicators: A Framework for Assessing Internet Development Key Guides

The Impact Field Guide and Toolkit: From Art to Impact Key Guides

The Impact Field Guide and Toolkit: From Art to Impact [in Arabic] Key Guides

Journalism Impact Primer & Journalism Impact Toolkit Key Guides

Guía práctica y herramientas para el impacto: Del arte al impacto
Key Guides

O guia de campo: Kit der ferramentas para o impacto. Da arte ao impacto
Key Guides

UNESCO’s Internet Universality Indicators: A Framework for Assessing Internet Development
Key Guides

The Impact Field Guide and Toolkit: From Art to Impact
Key Guides

The Impact Field Guide and Toolkit: From Art to Impact [in Arabic]
Key Guides

Journalism Impact Primer & Journalism Impact Toolkit
Key Guides