"This working paper aims to synthesize current efforts to develop comparable evaluation methods for social issue documentary films. Authored by two researchers who have been jointly documenting the field’s transformation over the past five years, this paper offers a framework for planning and eval
...
uating the impact of these films in a networked media environment [...] Drawing insights from the design thinking field—a user-focused creation process that has emerged from the commercial design field and is now being applied to create and improve social sector projects—this working paper examines state-of-the art methodologies for strategic design and evaluation of documentaries. The report’s recommendations are informed by lessons from six case studies of representative films: A Lion in the House; The Line; Lioness; Not in Our Town; Out in the Silence; and State of Fear. These are all award-winning projects featuring compelling documentary films at the center of multiplatform strategic outreach campaigns. All have been screened in traditional broadcast and/or film festival settings, as well as venues designed to engage publics and mobilize advocates relevant to the issue being addressed. These producers utilize a variety of technologies and both online and offline organizing tactics. At their most powerful, they catalyze and support issue-based networks that connect individuals with relevant organizations and empower participants not only to learn about and discuss shared problems, but to organize for action and respond to breaking developments. In this way, documentaries feed both social movements and the broader public sphere." (Introduction)
more
"This book off ers an accessible introduction to the topic of impact evaluation and its practice in development. Although the book is geared principally toward development practitioners and policy makers, we trust that it will be a valuable resource for students and others interested in impact evalu
...
ation. Prospective impact evaluations assess whether or not a program has achieved its intended results or test alternative strategies for achieving those results. We consider that more and better impact evaluations will help strengthen the evidence base for development policies and programs around the world. Our hope is that if governments and development practitioners can make policy decisions based on evidence—including evidence generated through impact evaluation—development resources will be spent more eff ectively to reduce poverty and improve people’s lives. The three parts in this handbook provide a nontechnical introduction to impact evaluations, discussing what to evaluate and why in part 1; how to evaluate in part 2; and how to implement an evaluation in part 3. These elements are the basic tools needed to successfully carry out an impact evaluation. The approach to impact evaluation in this book is largely intuitive, and we attempt to minimize technical notation. We provide the reader with a core set of impact evaluation tools—the concepts and methods that underpin any impact evaluation—and discuss their application to real-world development operations. The methods are drawn directly from applied research in the social sciences and share many commonalities with research methods used in the natural sciences. In this sense, impact evaluation brings the empirical research tools widely used in economics and other social sciences together with the operational and political-economy realities of policy implementation and development practice.
From a methodological standpoint, our approach to impact evaluation is largely pragmatic: we think that the most appropriate methods should be identified to fit the operational context, and not the other way around. This is best achieved at the outset of a program, through the design of prospective impact evaluations that are built into the project’s implementation. We argue that gaining consensus among key stakeholders and identifying an evaluation design that fits the political and operational context are as important as the method itself. We also believe strongly that impact evaluations should be candid about their limitations and caveats. Finally, we strongly encourage policy makers and program managers to consider impact evaluations in a logical framework that clearly sets out the causal pathways by which a program works to produce outputs and influence final outcomes, and to combine impact evaluations with monitoring and complementary evaluation approaches to gain a full picture of performance." (Preface)
more
"This paper explores why measuring the impacts of information and communication technology (ICT) is important for development – and why it is statistically challenging. Measuring impacts in any field is difficult, but for ICT there are added complications because of its diversity and rapidly chang
...
ing nature. A number of impact areas are identified in section 1, and their relationships explored, in the context of their place in the social, economic and environmental realms. The result is a complex web of relationships between individual impact areas, such as economic growth and poverty alleviation, and background factors, such as a country’s level of education and government regulation.
Existing measurement frameworks are described in section 1, and relevant statistical standards examined. The latter includes internationally agreed standards for the ICT sector, ICT products and ICT demand. The contribution of the Partnership on Measuring ICT for Development and its member organizations to ICT measurement, and its goals for measuring ICT impacts are outlined. Methodologies used in the measurement of ICT are discussed and compared in section 2 of the paper, and empirical evidence reviewed, in section 3. Most research conducted has found positive effects of ICT in the impact areas investigated. However, research has tended to focus on positive, rather than negative impacts; therefore, the latter tend to be indicated by anecdotal evidence. There is relatively little evidence from developing countries and there are indications that findings in respect of developed countries may not apply to developing countries. In respect of both developed and developing countries, there are few studies that provide internationally comparable evidence. The difficulties of ICT impact measurement, major data gaps and the lack of clear statistical standards suggest several issues for consideration. These are presented in the final section of the paper." (Abstract)
more
"This book reviews quantitative methods and models of impact evaluation, presenting an analysis of the quantitative research underlying recent programme evaluations and case studies prepared for a series of impact evaluation workshops in different countries, sponsored by the World Bank Institute (WB
...
I). The handbook also details challenges and goals in other realms of evaluation, including monitoring and evaluation, operational evaluation, and mixed-methods approaches combining quantitative and qualitative analyses. It is designed to put theory on evaluation methods and practices into practice in a hands-on fashion for practitioners, especially researchers new to the evaluation field and policymakers involved in implementing development programmes worldwide." (Publisher description)
more
"This Working Paper from DFID’s Evaluation Department offers a menu of Voice & Accountability (V&A) indicators, and suggests steps for building monitoring and evaluation frameworks for V&A interventions. It provides a check list of management issues, and some ideas for data collection. It does not
...
attempt to make any judgement about the indicators identified, as this work will be further developed under DFID’s ‘Suggested Indicators for Governance’ work, due in 2010. The paper is intended for DFID advisers and managers working on V&A initiatives at the country level. It will also be of use to people outside DFID who are interested in understanding how V&A work contributes to development outcomes; or who want more information about data collection methods for V&A measurement. V&A interventions range from work with governments on policy and reform processes, to activities at community level on civic education and rights awareness. DFID supports a significant amount of V&A work through government and non-state actors, in sector programmes and in work with civil society organisations including the media. We now need to establish the evidence base to show what change has resulted." (Executive summary)
more
"Die anvisierte Leserschaft dieser Landkarte ist die entwicklungspolitische Fachwelt, die mit der Analyse von tatsächlichen, das heißt empirisch nachprüfbaren (de facto) Wirkungen befasst ist und die einen schnellen, informativen Zugriff auf das Vorhandene benötigt. Dies sind in erster Linie Pro
...
jekt- und Programmverantwortliche in den Entwicklungsorganisationen in Geber- und Empfängerländern, die gehalten sind, die Wirkungen von Entwicklungsmaßnahmen nachzuweisen und in ihren Berichten zu dokumentieren; Mitarbeiter/-innen von Evaluierungs- und Qualitätsmanagementeinheiten, die Wirkungsanalysen zumeist als Teil von (Querschnitts-)Evaluationen in Auftrag geben; Wissenschaftler/-innen, die bei dem Design von empirischen Wirkungsstudien beraten oder diese selbst durchführen; sowie Gutachter/-innen und Berater/-innen, die vor der Aufgabe stehen, Wirkungsanalyseverfahren anzuwenden oder in einer Organisation einzuführen. In der vorliegenden Landkarte wurde versucht, alle, insbesondere im deutschsprachigen Kontext bekannten und in der Praxis erprobten, gut beschriebenen Wirkungsanalyseverfahren aufzunehmen, wobei ein Anspruch auf Vollständigkeit nicht erhoben werden kann. Dabei liegt der Schwerpunkt nicht nur auf qualitativen und partizipativen Verfahren, die selbst oft zusätzliche quantitative Elemente enthalten, sondern es werden auch quantitative experimentelle und nicht-experimentelle Designs beschrieben, die mitunter als „rigorose“ Verfahren bezeichnet werden. Dabei werden hier keine Bewertungen im Sinne von gut oder schlecht ausgesprochen. Ziel ist es stattdessen, den Nutzern dieser Landkarte die Möglichkeit zu geben, Verfahren nach rein sachlichen und pragmatischen Erwägungen für einen bestimmen Untersuchungszweck oder -auftrag auszuwählen." (Vorwort)
more
"This guide outlines a step-by-step process for developing a performance story report in the context of natural resource management. The guide can be used in a range of contexts by evaluators and others who want to use participatory evaluation approaches to report on outcomes. Users may include cons
...
ultants, government and non-government organisations, community groups, students, natural resource management organisations, industry bodies and academics. The guide will enable them to evaluate and report on program outcomes and continually improve their programs based on lessons learned through participatory evaluation approaches. Performance story evaluation is useful for both internal and external evaluations and learning. Performance story reports are effective for reporting annually on achievements from one year to the next." (Introduction, page 5)
more
"This report investigates the impact media and ICTs can have on the lives of the poor, based on the experiences of nine donors and NGOs forming part of the "Building Communication Opportunities (BCO)" alliance. It suggests that radio will have the most influence on social and political change where
...
it is widely accessible, trusted by listeners, and open to inclusive participation. ICTs can help make markets work for the poor, but the surrounding circumstances are highly influential in determining in how far they make a difference. Communication networks appear to be particularly effective in building communities of activists where they enable the pooling of resources and expertise and leverage wider influence on decision-makers. However, the report concludes that evidence of the impact of ICTs is still weak. More debate is needed about how ICTs are best deployed. This requires learning how people really use the tools, as well as a more effective assessment of past and current experiences." (CAMECO Update 1-2009)
more
"Having a vibrant media scene is a necessary prerequisite to human development and good governance. But, the time has come for us, media practitioners and support organisations, to accept and recognise that this is too complex to bring about on our own. It would be prudent to recognise the limitatio
...
ns of our sector, and create appropriate evaluation and impact assessment tools. The existing tools and methodologies are devised to give a macro picture of the overall environments but fail to clearly demarcate the roles played by various actors: State, Judiciary, Executive, Civil Society and Media. Media is just one contributing factor, albeit an important one at that. Hence, it is imperative to track the spheres of influence wielded by the sector so that support organisations are not misled into tracking and measuring overall environments while attempting to quantify the impact that media support organisations have in the process of change." (Page 2)
more
"Billions of US dollars are invested each year by the public, NGO and private sectors in information and communication technologies for development (ICT4D) projects such as telecentres, village phone schemes, e-health and e-education projects, e-government kiosks, etc. Yet we have very little sense
...
of the effect of that investment. Put simply, there is far too little impact assessment of ICT4D projects. In part that reflects a lack of political will and motivation. But in part it also reflects a lack of knowledge about how to undertake impact assessment of ICT4D. This Compendium aims to address that lack of knowledge. It presents a set of frameworks that can be used by ICT4D practitioners, policy-makers and consultants to understand the impact of informatics initiatives in developing countries. The Compendium is arranged into three parts: overview – explains the basis for understanding impact assessment of ICT4D projects, and the different assessment frameworks that can be used; frameworks – summarises a series of impact assessment frameworks, each one drawing from a different perspective; bibliography – a tabular summary of real-world examples of ICT4D impact assessment." (Introduction, page iii)
more
"Teil I erläutert Begriffe und Hintergründe wirkungsorientierter Projektplanung und verortet Wirkungsorientierung im Projektzyklus der Welthungerhilfe. Teil II stellt eine praxisorientierte Handreichung bereit, in der das wirkungsorientierte Vorgehen in den einzelnen Projektphasen Schritt für Sch
...
ritt anhand eines konkreten Projektbeispiels erläutert wird. Mögliche Methoden, die – in Abhängigkeit vom Projekttyp – bei der Durchführung des jeweiligen Schrittes angewandt werden können, sind nur aufgeführt, aber nicht im Einzelnen näher erläutert. Teil III enthält eine Sammlung von Instrumenten und Methoden zur Wirkungsorientierung und benennt Vorteile und Grenzen der jeweiligen Werkzeuge. Die Auswahl basiert zum überwiegenden Teil auf den Erfahrungen, die in den Projekten der Welthungerhilfe gemacht wurden." (Seite 4)
more
"Does USAID’s democracy promotion program work? Although some prior studies have examined specific projects in individual countries, no prior effort has studied the question on a world-wide basis, and no prior study has encompassed the entire post Cold-War period. [...] In the first phase of that
...
research, we found that the answer to that question was “yes.” That is, on average, in the period 1990-2003, USAID’s investments in democracy promotion produced significant increases in the national level of democracy as measured by Freedom House and Polity IV indicators. [...] The current report presents the results of the second phase of the project “Cross-National Research on USAID’s Democracy and Governance Programs.” [...] In the current effort, the data set is extended from 14 years to cover 15 years (1990-2004) and 165 countries, yielding 2,416 observations (country-years). [...] USAID civil society and media assistance have a significant positive impact directly on their respective sectors, and USAID human rights assistance has a significant negative impact on the human rights outcome." (Executive summary, page 2, page 5)
more
"This guide aims to provide practitioners with a broad framework for carrying out project level Participatory Impact Assessments (PIA) of livelihoods interventions in the humanitarian sector. Other than in some health, nutrition, and water interventions in which indicators of project performance sho
...
uld relate to international standards, for many interventions there are no ‘gold standards’ for measuring project impact. For example, the Sphere handbook has no clear standards for food security or livelihoods interventions. This guide aims to bridge this gap by outlining a tried and tested approach to measuring the impact of livelihoods projects. The guide does not attempt to provide a set of standards or indicators or blueprint for impact assessment, but a broad and flexible framework which can be adapted to different contexts and project interventions. Consistent with this, the proposed framework does not aim to provide a rigid or detailed step by step formula, or set of tools to carry out project impact assessments, but describes an eight stage approach, and presents examples of tools which may be adapted to different contexts. One of the objectives of the guide is to demonstrate how PIA can be used to overcome some of the inherent weaknesses in conventional humanitarian monitoring evaluation and impact assessment approaches, such as; the emphasis on measuring process as opposed to real impact, the emphasis on external as opposed to community based indicators of impact, and how to overcome the issue of weak or non-existent baselines. The guide also aims to demonstrate and provide examples of how participatory methods can be used to overcome the challenge of attributing impact or change to actual project activities. The guide will also demonstrate how data collected from the systematic use of participatory tools can be presented numerically, and can give representative results and provide evidence based data on project impact." (Introduction, page 6)
more
"Hauptfunktion des wirkungsorientierten Monitoring ist es, die Steuerung von Entwicklungsmaßnahmen zu unterstützen sowie den Dialog über die eingeschlagene Strategie im Management- bzw. Projektteam und deren Aufsichtsgremien anzuregen – insbesondere im Hinblick auf die regelmäßige Operationsp
...
lanung. Es fördert darüber hinaus Lernprozesse (Wissensmanagement) und legt die Grundlage für eine verlässliche Rechenschaftslegung [...] Das Monitoring organisiert die Prozesse der Sammlung, Analyse- und Bewertung von Daten zu den Wirkungen eines Projektes und den Veränderungen in seinem Umfeld. Es liefert kontinuierlich Hinweise darüber, ob der eingeschlagene Weg Erfolg verspricht, die Strategie angepasst werden sollte oder die gesetzten Ziele zu überprüfen sind. Ebenso geht es um die Beobachtung der indirekten Wirkungen. Sie stellen den eigentlichen Grund für die Durchführung des Projektes dar. Sie sind in der Regel Ziele nationaler oder sektoraler Strategien und Programme des Partners, des Schwerpunkts der deutschen EZ in einem Land oder ähnlicher Zielsysteme, z. B. von „country assistance“-Programmen der EU oder von Multi-Geber-Programmen. Wirkungen auf dieser Ebene können zwar einer Entwicklungsmaßnahme nicht mehr eindeutig, d. h. kausal zugeordnet werden, jedoch trägt sie durch ihre Leistungen plausibel zu deren Zielerreichung bei." (Aufgaben des wirkungsorientierten Monitoring, Seite 9)
more