Toolbox Evaluation & Impact Assessment

Communication for Development: An Evaluation Framework in Action
Top Insights

Jessica Noske-Turner (ed.)

Rugby: Practical Action Publishing (2020), xiii, 124 pp.

"This book presents five cases that reflect on the experiences of using practices consistent with the 'Evaluating C4D framework' [published by June Lennie and Jo Tacchi in 2013]. Case studies are important to help move from a set of ideal principles to an understanding of how the framework may be op ... more

Process Tracing as a Practical Evaluation Method: Comparative Learning from Six Evaluations

Alix Wadeson, Bernardo Monzani, Tom Aston

[authors] (2020), 31 pp.

"This paper presents comparative learning from the evaluation of six international development initiatives that applied various forms of Process Tracing. While these initiatives span across diverse contexts and pursued different aims, they are connected by a common thread: all six case studies centr ... more

Decoding Media Impact: Insights, Advice and Recommendations

Katie Donnelly

Philadelphia, Penns.: Media Impact Funders (2020), 31 pp.

"This guide is meant to serve as a practical resource for funders who want to understand where to start. Informed by feedback from our network, it represents a synthesis of the past seven years of work we’ve done in the impact space, and includes examples of successful media impact evaluation, too ... more

External Link (12)

"Big Changes Start Small": Stories of People Making a Difference. An Impact Report

Bonn: Deutsche Welle DW Akademie (2020), 97 pp.

"DW Akademie collects and documents the impacts of its projects through a range of methods. There are three approaches that are currently standard in development cooperation: Applied studies, which enable new insights on impacts, impact-oriented monitoring performed with local partners in the course ... more

A Valuing Impact Toolkit for ICT Investment

NEF Consulting;Digital Impact Alliance (dial) (2020), 124 pp.

"ICT4D projects are known for having social, economic, and environmental impacts that are difficult to capture. Investments in digital infrastructure and innovation often have high up-front costs with largely medium-to-long-term benefits, not quick, short-term outcomes desired by policy makers. The ... more

How to Perform Impact Assessments: Key Steps for Assessing Communication Interventions

James Dennison

International Centre for Migration Policy Development (ICMPD);EUROMED Migration (2020), 29 pp.

"This report brings together disparate terminology, findings and recommendations from the private and public sectors and academia to synthesise a set of five general steps for practitioners when performing impact assessments. Impact assessment (IA) compares a variable of interest after an interventi ... on (e.g. a communication campaign) to what it would it be if that intervention had never happened. Accompanying each of the five steps are examples of how migration communicators can assess the impact of campaigns to, first, change attitudes and, second, change behaviour. Step 1: Set objectives for the intervention that define what the desired effect is. Ideally, this should (1) meet the SMART criteria of specific, measurable, attainable, relevant and timebound; (2) focus on ultimate outcomes, rather than only outputs; (3) avoid a number of common mistakes in setting objectives and (4) carefully consider what the—as specific as possible— target audience is. Step 2: Identify a measure of the impact(s) that the intervention has as its objective. These should be (1) clearly defined and unambiguous; (2) externally valid (they act reliably if used for other interventions) and (3) internally valid (they measure what they claim to measure). There already exist long-lists of measures (also known as indicators) to choose from. Step 3: Estimate what would have happened had there been no intervention, known as a ‘counterfactual’. Ideally, but not necessarily, this requires performing a pre-intervention measurement, also known as baseline assessment. However, this is not always possible. This should not dissuade practitioners but instead assumptions of any IA, as discussed below, should be made openly. A long list of methods for measurement, including sampling considerations, are provided. Step 4: Perform the intervention. This may involve separating a randomised sample into a treatment group(s)—that receives the intervention(s)—and a control group—that does not. If this is not feasible, other forms of counterfactual are possible that ‘construct’ a control group. This section also overviews: (1) types of communication campaigns; (2) the MINDSPACE checklist of behavioural influences for interventions; (3) key recommendations on persuasive interventions from ICMPD’s previous work; (4) recommendations on campaigns deterring irregular migration. Step 5: Post-intervention measurement and analysis: (1) those receiving the intervention should be measured post-intervention (ideally, as well as, either a genuine or constructed control group); (2) the impact must be calculated: usually the difference in the variable of interest pre- and post-intervention and/or between the treatment and control group; (3) theoretical consideration of why and what aspect of the campaign caused an impact; are the findings likely to be universalizable?; (4) creation (and, in some cases, incorporation) of recommendations based on combination of impact and theoretical considerations." (Executive summary) more

Sida's Evaluation Handbook: Guidelines and Manual for Conducting Evaluations at Sida

Stockholm: Swedish International Development Cooperation Agency (Sida) (2020), 41 pp.

A Toolkit for Monitoring and Evaluation of Gender-Based Violence Programming in Restricted Environments

Women's Refugee Commission (2020), 210 pp.

CEDIL Guidelines on Impact Evaluation that are Sensitive to Disability

Centre of Excellence for Development Impact and Learning (CEDIL) (2020), 5 pp.

"These are simple recommendations on the inclusion of data on disability for researchers conducting evidence syntheses and primary studies." (Page 1)

Using Big Data for Evaluating Development Outcomes: A Systematic Map

Francis Rathinam, Sayak Khatua, Zeba Siddiqui, Manya Malik, Pallavi Duggal, Samantha Watson, Xavier Vollenweider

Centre of Excellence for Development Impact and Learning (CEDIL) (2020), vi, 104 pp.

"Spatially and temporally relevant ‘big data’ that does not require data collection in the field has the potential to provide insights into people’s economic, social, behavioural and political lives, and hence could be used in measuring key development outcomes. Big data consists of humangener ... more

Measuring Results and Impact in the Age of Big Data: The Nexus of Evaluation, Analytics, and Digital Technology

Pete York, Michael Bamberger

Rockefeller Foundation (2020), ix, 72 pp.

How to perform impact assessments: Key steps for assessing communication interventions [Arabic]

James Dennison

International Centre for Migration Policy Development (ICMPD);EUROMED Migration (2020), 26 pp.

Comment réaliser des évaluations d’impact : étapes clés pour évaluer les actions de communication

James Dennison

International Centre for Migration Policy Development (ICMPD);EUROMED Migration (2020), 30 pp.

"Ce rapport réunit une terminologie hétérogène, des conclusions et des recommandations émanant des secteurs privé et public et du monde universitaire pour définir un ensemble de cinq étapes générales à suivre par les professionnels lors de la réalisation d’une EI. L’EI compare une va ... riable étudiée après une action (par ex. une campagne de communication) à ce qu’elle serait si cette action n’avait jamais eu lieu. Accompagnant chacune des cinq étapes, nous présenterons des exemples de la façon dont les communicants en migration peuvent évaluer l’impact des campagnes pour, d’abord, modifier les attitudes et, ensuite, changer les comportements. Étape 1 : Fixer des objectifs pour l’action qui définissent l’effet souhaité. Dans l’idéal, ils devraient (1) correspondre aux critères SMART : spécifique, mesurable, atteignable, pertinent et temporellement défini; (2) se concentrer sur les résultats finaux, plutôt qu’uniquement sur les produits; (3) éviter certaines erreurs courantes dans la définition des objectifs et (4) déterminer de façon aussi spécifique que possible l’identité du public cible. Étape 2 : Identifier une mesure de l’impact ou des impacts de l’objectif de l’action. Ces impacts devraient être (1) clairement définis et non ambigus; (2) valides d’un point de vue externe (ils restent fiables s’ils sont utilisés pour d’autres actions) et (3) valides d’un point de vue interne (ils mesurent ce qu’ils prétendent mesurer). Il existe déjà de longues listes de mesures (également connues sous le nom d’indicateurs) parmi lesquelles il est possible de choisir. Étape 3 : Estimer ce qui se serait passé si l’action n’avait pas eu lieu, ou situation « contrefactuelle ». Idéalement, mais pas nécessairement, cela implique la réalisation de mesures en amont de l’action, également connues sous le nom d’évaluation de base. Néanmoins, ce n’est pas toujours possible. Cela ne devrait pas dissuader les professionnels, en revanche, les hypothèses relatives aux EI devraient faire l’objet d’une discussion ouverte, comme évoqué ci-dessous. Une longue liste de méthodes de mesure, y compris des considérations liées à l’échantillonnage, sont fournies. • Étape 4 : Réaliser l’action. Cela peut impliquer de séparer un échantillon aléatoire en un ou plusieurs groupes traités, qui font l’objet de l’action/des actions, et un groupe témoin, qui n’en fait pas l’objet. Si cela n’est pas faisable, d’autres formes de situations contrefactuelles sont possibles pour « construire » un groupe témoin. Cette section aborde également : (1) les types de campagnes de communication; (2) la check-list MINDSPACE d’influences comportementales pour les actions; (3) les recommandations clés portant sur les actions persuasives issues des travaux précédents de l’ICMPD; (4) les recommandations relatives aux campagnes de dissuasion de la migration irrégulière. • Étape 5 : Mesures et analyses en aval de l’action : (1) ceux qui ont bénéficié de l’action doivent faire l’objet de mesures en aval de l’action (idéalement dans le cadre un groupe témoin authentique ou construit); (2) l’impact doit être calculé : généralement la différence entre la variable étudiée en amont et en aval de l’action et/ou entre le groupe traité et témoin; (3) réflexion théorique sur les raisons et les aspects précis de la campagne qui ont causé un impact; les conclusions sont-elles transposables ?; (4) création (et, dans certains cas, incorporation) de recommandations basées sur la combinaison de l’impact et des réflexions théoriques." (Résumé) more

The Key to Evaluation: 10 Steps. "Evaluiert" in Brief

Zollikofen: Swiss Federal University for Vocational Education and Training (SFUVET) (2020), 23 pp.

"In this brochure – and in the ‘evaluiert’ textbook to which it refers – evaluation is understood as a scientific service that describes and valuates objects of evaluation (especially programmes and measures) for clarified purposes and on the basis of justified criteria. Moreover, the valuat ... more

Feminist Monitoring, Evaluation, Accountability and Learning

Ottawa: Oxfam Canada (2020), 8 pp.

Estimated Costs for Evaluation Services Depending on the Scale of the Program Being Evaluated

George Argyrous

Institute of Public Policy Governance (2020), 1 p.

"The resource provides a guide for the expected costs of various evaluative activities in an Australian context. The main purpose for commissioners of evaluative activity to have a sense of the budget they should set aside for these activities. However, the guide also alerts potential users to the r ... more

Digital Literacy Key Performance Indicators for Sustainable Development

Danica Radovanovic, Christine Holst, Sarbani Banerjee Belur, Ritu Srivastava, Georges Vivien Houngbonon et al.

Social Inclusion, volume 8, issue 2 (2020), pp. 151-167

"The concept of digital literacy has been defined in numerous ways over the last two decades to incorporate rapid technological changes, its versatility, and to bridge the global digital divide. Most approaches have been technology-centric with an inherent assumption of cultural and political neutra ... more

Managing Outcomes: A Practitioner's Manual for Analysis, Planning, Monitoring and Self-Evaluation Based on Outcome Mapping

Christian Kuijstermans et al.

Cologne: AGIAMONDO (2020), vi, 114 pp.

"Outcome Mapping was developed 20 years ago by the International Development Research Centre and for many in the development sector it has become a staple in planning, monitoring and evaluation toolkits. Countless organisations across the world have applied and adapted Outcome Mapping for their own ... more

Wozu Non-Profit-Journalismus? Ein Report zu gemeinnützigem Journalismus – mit Wirkungslogiken, Qualitätskriterien und Tipps für Förder*innen
Key Guidance

Charlotte Buttkus, Igor Ryabinin, Florian Hinze

Berlin: PHINEO gemeinnützige AG (2020), 63 pp.

"Der Report „Wozu Non-Profit-Journalismus?“ macht sich für eine dritte Säule im Mediensystem stark: Der gemeinnützige Journalismus hält der Krise Antworten entgegen und leistet als Ergänzung zum öffentlich-rechtlichen und privaten Verlagsjournalismus einen Beitrag zur Medienvielfalt. Er is ... more