"This working paper aims to synthesize current efforts to develop comparable evaluation methods for social issue documentary films. Authored by two researchers who have been jointly documenting the field’s transformation over the past five years, this paper offers a framework for planning and eval
...
uating the impact of these films in a networked media environment [...] Drawing insights from the design thinking field—a user-focused creation process that has emerged from the commercial design field and is now being applied to create and improve social sector projects—this working paper examines state-of-the art methodologies for strategic design and evaluation of documentaries. The report’s recommendations are informed by lessons from six case studies of representative films: A Lion in the House; The Line; Lioness; Not in Our Town; Out in the Silence; and State of Fear. These are all award-winning projects featuring compelling documentary films at the center of multiplatform strategic outreach campaigns. All have been screened in traditional broadcast and/or film festival settings, as well as venues designed to engage publics and mobilize advocates relevant to the issue being addressed. These producers utilize a variety of technologies and both online and offline organizing tactics. At their most powerful, they catalyze and support issue-based networks that connect individuals with relevant organizations and empower participants not only to learn about and discuss shared problems, but to organize for action and respond to breaking developments. In this way, documentaries feed both social movements and the broader public sphere." (Introduction)
more
"This book off ers an accessible introduction to the topic of impact evaluation and its practice in development. Although the book is geared principally toward development practitioners and policy makers, we trust that it will be a valuable resource for students and others interested in impact evalu
...
ation. Prospective impact evaluations assess whether or not a program has achieved its intended results or test alternative strategies for achieving those results. We consider that more and better impact evaluations will help strengthen the evidence base for development policies and programs around the world. Our hope is that if governments and development practitioners can make policy decisions based on evidence—including evidence generated through impact evaluation—development resources will be spent more eff ectively to reduce poverty and improve people’s lives. The three parts in this handbook provide a nontechnical introduction to impact evaluations, discussing what to evaluate and why in part 1; how to evaluate in part 2; and how to implement an evaluation in part 3. These elements are the basic tools needed to successfully carry out an impact evaluation. The approach to impact evaluation in this book is largely intuitive, and we attempt to minimize technical notation. We provide the reader with a core set of impact evaluation tools—the concepts and methods that underpin any impact evaluation—and discuss their application to real-world development operations. The methods are drawn directly from applied research in the social sciences and share many commonalities with research methods used in the natural sciences. In this sense, impact evaluation brings the empirical research tools widely used in economics and other social sciences together with the operational and political-economy realities of policy implementation and development practice.
From a methodological standpoint, our approach to impact evaluation is largely pragmatic: we think that the most appropriate methods should be identified to fit the operational context, and not the other way around. This is best achieved at the outset of a program, through the design of prospective impact evaluations that are built into the project’s implementation. We argue that gaining consensus among key stakeholders and identifying an evaluation design that fits the political and operational context are as important as the method itself. We also believe strongly that impact evaluations should be candid about their limitations and caveats. Finally, we strongly encourage policy makers and program managers to consider impact evaluations in a logical framework that clearly sets out the causal pathways by which a program works to produce outputs and influence final outcomes, and to combine impact evaluations with monitoring and complementary evaluation approaches to gain a full picture of performance." (Preface)
more
"Developmental evaluation offers a powerful approach to monitoring and supporting social innovations by working in partnership with program decision makers. In this book, eminent authority Michael Quinn Patton shows how to conduct evaluations within a developmental evaluation framework. Patton draws
...
on insights about complex dynamic systems, uncertainty, nonlinearity, and emergence. He illustrates how developmental evaluation can be used for a range of purposes: ongoing program development, adapting effective principles of practice to local contexts, generating innovations and taking them to scale, and facilitating rapid response in crisis situations. Students and practicing evaluators will appreciate the book's extensive case examples and stories, cartoons, clear writing style, "closer look" sidebars, and summary tables." (Publisher description)
more
"Agency-based Program Evaluation: Lessons from Practice, by Stephen A. Kapp and Gary R. Anderson, serves as a core textbook in the advanced undergraduate and graduate social work program evaluation courses. It combines the methodology of program evaluation with the reality of working with agencies a
...
nd organizations to describe the effectiveness of their services and programs. Students will gain an understanding of the political and social context and pressures in which a program is developed, implemented and evaluated. This book offers a practice-oriented approach to evaluation. While many program evaluation methods texts often add a or brief sections that describe organizational and political factors, this book begins with the context of an agency-based evaluation and describes the method within that context. Students will gain a more complete understanding of this contextual challenge and will learn techniques for operating in the face of these challenges." (Publisher description)
more
"United Nations agencies currently do not have commonly agreed tools for assessing the impact of Communication for Development (C4D) programmes. This paper is designed to help them explore the key issues surrounding C4D, consider case studies and best practice methodology and propose a set of draft
...
indicators for monitoring and evaluation (M&E). Nevertheless, this publication is also of value for anybody interested in M&E. It presents in a clear and succinct way the principal tools and approaches, from conventional, quantitative research techniques to more intricate, participatory processes, and it summarises the challenges to communication for development M&E. Case studies are used to illustrate how initiatives have used a variety of approaches to gather informative results and feed these back into programme design. The authors emphasise that a tailored toolkit approach is necessary in order to deal with the many complexities of C4D evaluation. They highlight the need for flexibility within any evaluation and discuss the strengths of Most Significant Change, Outcome Mapping and the Logframe as state of the art evaluation techniques. Qualitative and quantitative tools should be used as complementary strategies for providing the most comprehensive, clear and pragmatic assessment and reporting of C4D." (CAMECO Update 4-2009)
more
"Die anvisierte Leserschaft dieser Landkarte ist die entwicklungspolitische Fachwelt, die mit der Analyse von tatsächlichen, das heißt empirisch nachprüfbaren (de facto) Wirkungen befasst ist und die einen schnellen, informativen Zugriff auf das Vorhandene benötigt. Dies sind in erster Linie Pro
...
jekt- und Programmverantwortliche in den Entwicklungsorganisationen in Geber- und Empfängerländern, die gehalten sind, die Wirkungen von Entwicklungsmaßnahmen nachzuweisen und in ihren Berichten zu dokumentieren; Mitarbeiter/-innen von Evaluierungs- und Qualitätsmanagementeinheiten, die Wirkungsanalysen zumeist als Teil von (Querschnitts-)Evaluationen in Auftrag geben; Wissenschaftler/-innen, die bei dem Design von empirischen Wirkungsstudien beraten oder diese selbst durchführen; sowie Gutachter/-innen und Berater/-innen, die vor der Aufgabe stehen, Wirkungsanalyseverfahren anzuwenden oder in einer Organisation einzuführen. In der vorliegenden Landkarte wurde versucht, alle, insbesondere im deutschsprachigen Kontext bekannten und in der Praxis erprobten, gut beschriebenen Wirkungsanalyseverfahren aufzunehmen, wobei ein Anspruch auf Vollständigkeit nicht erhoben werden kann. Dabei liegt der Schwerpunkt nicht nur auf qualitativen und partizipativen Verfahren, die selbst oft zusätzliche quantitative Elemente enthalten, sondern es werden auch quantitative experimentelle und nicht-experimentelle Designs beschrieben, die mitunter als „rigorose“ Verfahren bezeichnet werden. Dabei werden hier keine Bewertungen im Sinne von gut oder schlecht ausgesprochen. Ziel ist es stattdessen, den Nutzern dieser Landkarte die Möglichkeit zu geben, Verfahren nach rein sachlichen und pragmatischen Erwägungen für einen bestimmen Untersuchungszweck oder -auftrag auszuwählen." (Vorwort)
more
"This is a practical and well structured manual aiming to use self-evaluation for organisational learning. The book consists of four parts. "The evaluation context" introduces the role of monitoring, evaluation and impact assessment as part of the project cycle logic. "The evaluation process" descri
...
bes steps to be taken in designing and implementing an evaluation. The third part, "evaluation tools", gives a practical insight to major evaluation methods like SWOT analysis, questionnaires (and their design), focus groups or case studies. The fourth and main part provides evaluation guidelines for training courses, newsletters, websites, small libraries and resource centres, online communities, rural radios, databases and selective dissemination of information services." (CAMECO Update 1-2011)
more
"Explores the application of systems ideas to investigate, evaluate, and intervene in complex and messy situations. The text serves as a field guide, with each chapter representing a method for describing and analyzing; learning about; or changing and managing a challenge or set of problems. The boo
...
k is the first to cover in detail such a wide range of methods from so many different parts of the systems field. The book's Introduction gives an overview of systems thinking, its origins, and its major subfields. In addition, the introductory text to each of the book's three parts provides background information on the selected methods. Systems Concepts in Action may serve as a workbook, offering a selection of tools that readers can use immediately. The approaches presented can also be investigated more profoundly, using the recommended readings provided. While these methods are not intended to serve as "recipes," they do serve as a menu of options from which to choose." (Publisher description)
more
"Billions of US dollars are invested each year by the public, NGO and private sectors in information and communication technologies for development (ICT4D) projects such as telecentres, village phone schemes, e-health and e-education projects, e-government kiosks, etc. Yet we have very little sense
...
of the effect of that investment. Put simply, there is far too little impact assessment of ICT4D projects. In part that reflects a lack of political will and motivation. But in part it also reflects a lack of knowledge about how to undertake impact assessment of ICT4D. This Compendium aims to address that lack of knowledge. It presents a set of frameworks that can be used by ICT4D practitioners, policy-makers and consultants to understand the impact of informatics initiatives in developing countries. The Compendium is arranged into three parts: overview – explains the basis for understanding impact assessment of ICT4D projects, and the different assessment frameworks that can be used; frameworks – summarises a series of impact assessment frameworks, each one drawing from a different perspective; bibliography – a tabular summary of real-world examples of ICT4D impact assessment." (Introduction, page iii)
more
"The symposium Measuring Change. Planning, Monitoring, Evaluation in Media Development focused on the utilisation aspect of evaluation1: The adding of “Planning” to “Monitoring and Evaluation” in the subtitle indicates that emphasis was laid on learning from monitoring and evaluation experie
...
nces, to facilitate the improvement of existing projects and programmes at all levels, from planning to implementation and follow-up." (Executive Summary)
more
"What is dialog, and how can it be measured in a meaningful way? In this article, Jacobson presents an approach to assessing participatory communication based on communication in the form of dialog as conceptualized by Jurgen Habermas." (Abstract)
"Este libro recupera parte de lo ya escrito y dicho sobre la evaluación en diferentes textos y espacios de reflexión y de diálogo. Si se nos pregunta sobre su aporte diríamos que, sumada a la convencional y necesaria presentación de argumentos conceptuales y herramientas metodológicas que sust
...
enten cómo hacer evaluación –quizá incorporando énfasis y estilos de alguna manera particulares–, su valor está en juntar en una misma propuesta cuatro propósitos de cambio para la evaluación, que si bien se encuentran aún en proceso de consolidación, tienen una presencia emergente en la reflexión actual sobre el tema. Estos propósitos son los siguientes: transitar de una evaluación centrada en la observación de actividades (qué y cuánto se ha hecho), a una que pone énfasis en la observación del cambio (qué ha cambiado con la intervención); transitar de una evaluación entendida como un servicio externo a los proyectos a cargo de especialistas también externos, a una concebida como una experiencia participativa, basada en la confianza y la corresponsabilidad en el manejo de los procesos, la emisión de juicios y la toma de decisiones; transitar de una evaluación operada como mecanismo de control y enfocada en la observación de defectos, a una entendida y organizada como espacio de aprendizaje y de desarrollo de capacidades; Transitar de una idea de la evaluación como práctica instrumental neutra (libre de valores), a entenderla como una práctica intencional, cargada de ideología y utopía, que no se limita a la verificación del cambio sino que apuesta a facilitarlo, haciendo jugar concepciones e intereses particulares." (Introducción)
more
"This paper discussed the possibility to improve public communication campaign theory, by making use of data obtained through mass media health communication campaign evaluations. The idea of an ‘engineering’ approach to campaign design, where theory and scientific findings are systematically us
...
ed and adopted for practical problems, plays an important role in the discussion [...] In the third part, a sample of 33 evaluation reports for mass media health communication campaigns was analyzed. 32 of these reports have not been published in an scientific journal. The evaluations were conducted in 22 different countries. The analysis of the reports focused on the campaign goals, evaluation outcome measures, research design and methods, and on questions of validity. The findings suggest that theory is not widely and consequently used to inform health mass communication campaigns or their evaluations – with notable exceptions. While there is a large number of outcomes measured, they seem to be taken out of theoretical context. Neither the campaign goals nor the evaluation measures reflect the large number of possible communication strategies that the various communication or behavior-change models and theories imply. Unintended campaign effects were mostly ignored. In very few cases the campaign designers or evaluators make use of an effects model or program logic model. This is one of the areas where I see the possibility of an important improvement. The methodology of campaign evaluation is relatively homogenous across the 33 cases in regards to data collection method. Standardized questionnaires are the dominating data collection instrument. Non-reactive observation or tracking methods are very rare. A surprising two thirds of the evaluations did not use multivariate analysis, and the reliance on self-reports raises questions of reliability." (Summary, page 120-121)
more
"The 29 indicators in the Guide measure the reach, usefulness, and use, as well as the collaboration, and capacity building engendered through information products and services. The innovative “Conceptual Framework for Monitoring and Evaluating Health Information Products and Services” shows how
...
they contribute to the initial, intermediate, and long-term outcomes of health development efforts—something which perhaps we all instinctively recognize but have failed to track in the past. Such a track record will go a long way to making information products and services an integral part of future global health development efforts. What makes this guide special is that it brings together knowledge about monitoring and evaluating information products and services from dozens of health organizations—all members of the HIPNET community of practice." (Foreword)
more
"The purpose of the manual is to introduce peacebuilding practitioners to the concepts, tools, and methods needed to incorporate better design, monitoring, and evaluation practices into peacebuilding programming. As an introductory volume, the target audience is front-line peacebuilding practitioner
...
s from around the world with minimal formal training in design, monitoring, and evaluation. It assumes the audience has experience, training, and access to resources on conflict assessments, which are a prerequisite to participating in conflict transformation program design [...] The manual offers general information on learning and change in addition to chapters dedicated to specific issues such as baselines, indicators, monitoring, and evaluation. It is organized so that readers can easily jump from one chapter to another. However, we strongly urge jumpy readers to start with the chapter on understanding change because it frames the thinking for most of the discussions in the other chapters." (Page 2)
more