"It has been almost two years since the 'Practice Guides on Doing Evaluation in Service of Racial Equity' were published. Since then, evaluators continue their commitment to advance racial equity through their practice as evident in the number of pre-conference workshops at the 2022 American Evaluat
...
ion Association conference (almost one-third of total workshops), activities and publications by the Equitable Evaluation Initiative, and the growing network of culturally responsible equitable evaluation practitioners. Since the practice guides’ release, Community Science, with support from the W.K. Kellogg Foundation, conducted many workshops and webinars about the information in the guides. Questions were collected as part of the registration and during the events. The questions shed light on the challenges we still face in supporting and doing evaluation in service of equity. In general, people are still working to get concrete about how to implement such evaluations and overcome the resistance — intentional or not — to engaging in courageous conversations about racial equity and shifting current practices in evaluation, community engagement, strategy development, and grantmaking.
The questions reaffirm that evaluators alone cannot advance the practice of doing evaluation in service of racial equity. The guides discuss the importance of recognizing this point. The larger systemic issues at play contribute to racial inequity, and social injustice requires all of us in philanthropy, government, nonprofit, and for-profit sectors to work differently. Community Science compiled common questions from evaluators, funders, executive directors, and program staff, organized them into themes, and attempted to answer them in this tool kit. Some questions cannot be answered easily because the solutions aren’t the evaluation profession and evaluators’ responsibility alone. For instance, questions about the distribution of wealth, role of philanthropy in closing the wealth gap, levers of change to prioritize public and philanthropic investments in communities of color, ways to manage and disrupt power differences between foundations and organizations that receive funding, use of power and narratives to center equity and justice in philanthropy, and building leadership that isn’t resistant to racial equity. There were also questions about shifting power, engaging communities, creating space in federal agencies for discussions about community engagement and racial equity, and changing mindsets and behaviors of leadership. In addition, there were questions we can only answer through a broader dialogue with social scientists from different disciplines with various philosophical approaches to research and evaluation (e.g., is community-based participatory research equitable, how to stop relying on pre- and post-outcome data).
This tool kit isn’t intended to repeat the original practice guides. It compiles new information in slide decks, tip sheets, and blogs. Some of the blogs have been written by Daniela Pineda and her colleagues at RTI International in support of the Practice Guides. This tool kit also doesn’t contain all the answers. We invite you to take the guidance farther and share your experiences. This tool kit, in our humble opinion, is merely another way to approach evaluation in service of racial equity and a starting place for people interested in this work." (Introduction)
more
"From the founder of empowerment evaluation (EE), a framework uniquely suited to advancing social justice causes, this book explains the theories, principles, and steps of conducting EE from scratch or within a preexisting evaluation or work plan. David M. Fetterman describes how EE enables program
...
planners and participants to define their mission or purpose, take stock of how well they are doing, and plan for the future to achieve self-determined goals. EEs of two large programs (Feeding America and USAID/REACH) are discussed in depth; other EE case examples address such topics as raising test scores in impoverished and rural schools and bridging the digital divide in communities of color. User-friendly features include chapters on conducting EE remotely and frequently asked questions, as well as illuminating sidebars and glossaries of acronyms and concepts/terms." (Publisher description)
more
"The Toolkit is built on a foundation of conceptual clarity, emphasizing the importance of clear definitions and context-specific understanding. It underscores the need for comprehensive disaggregated data that reveals the true extent of racial inequities and enables evidence-based policy-making. Th
...
roughout, the Toolkit focuses on the inclusion of affected communities and stakeholders, fostering meaningful engagement and dialogue. This Toolkit serves as a practical guide for policy-makers, activists, and organizations committed to advancing anti-racist policies. Drawing from global case examples and the latest research, it provides a step-by-step approach to developing, implementing, monitoring, and evaluating policies that combat racism and discrimination." (Short summary)
more
"Most advocacy work is carried out in complex environments where there are many different influences on change. This has important consequences for how objectives and indicators are developed. CSOs need to understand the immediate and short-term changes resulting from different advocacy activities i
...
n order to understand whether they are having the desired effect." (Introduction)
more
"Monitoring and evaluating advocacy work involves unique challenges. These can usually be overcome provided expectations are realistic. Ongoing monitoring and evaluation carried out during advocacy interventions should enhance and support real-time decision-making. Advocacy evaluations should includ
...
e critical and informed assessments of how organisations contributed to change." (Introduction)
more
"Monitoring and evaluation (M&E) systems for advocacy interventions usually include a number of common elements. The nature of advocacy work means M&E systems need to be designed and implemented differently than in conventional projects. M&E systems for advocacy work should be simple and time-effici
...
ent, and should be designed to support ongoing advocacy processes." (Introduction)
more
"Many different tools and methodologies can be used to support advocacy work. Some are designed specifically for monitoring and evaluation (M&E) purposes. Others are designed to support the planning and management of advocacy interventions, but can be used or adapted for M&E purposes. CSOs often emp
...
loy several different tools and methodologies over the course of an advocacy intervention." (Introduction)
more
"The aim of this paper is to analyse participatory video as a participatory action research method through the lenses of the capability approach. In order to do this, we used a participatory video experience that took place in the municipality of Quart de Poblet (Valencia, Spain) from February to Ma
...
rch 2014. The participants were 11 young people between 16 and 24 years of age, severely affected by the economic crisis that has hit Spain in recent years. To develop our analysis, we introduced the participatory video as a technique and a process within the participatory action research methods. Then, we analysed the participatory process to verify the extent to which it had contributed to expanding the capabilities and agency of the participants. The evidence revealed a significant expansion of the awareness capability and, in some cases, of the capability for voice. In contrast, the capability to aspire and the agency of the participants were not expanded, due to contextual factors and the limitations of the process itself." (Abstract)
more
"Like all evaluations, advocacy evaluations can draw on a familiar list of traditional data collection methods, such as surveys, interviews, focus groups, or polling. But because the advocacy process can be complex, fast-paced, and dynamic, which makes data collection challenging, and because advoca
...
cy efforts often aim for outcomes that are hard to operationalize and measure (e.g., public will or political will), new and innovative methods are being developed specifically for assessing advocacy and policy change efforts. This brief describes four new methods - Bellwether Methodology, Policymaker Ratings, Intense Period Debriefs, System Mapping - that were developed to respond to advocacy’s unique measurement challenges." (Page 1)
more
"This brief offers a simple one-page tool for thinking about the theories of change that underlie public policy advocacy strategies. It first presents the tool and then offers six questions that advocates, and funders working with advocates, can work through to better articulate their theories of ch
...
ange. The tool—labeled the advocacy strategy framework—has several advantages over more familiar linear box-and-arrow theory-of-change tools: As advocacy is not predictable or linear, the tool does not force linear thinking; it offers a place to start, rather than a blank page; it helps advocates to think more specifically about audiences—who is expected to change and how, and what it will take to get them there; while theories of change often consider advocacy strategies in isolation of other efforts, this tool helps to think about how other advocates (like-minded or in opposition) are positioned; it prompts thinking about useful tactics and meaningful interim outcomes." (Introduction, page 1)
more
"Policy influence and advocacy are increasingly regarded as a means of creating sustainable policy change in international development. It is often also seen as a difficult area to monitor and evaluate. Yet there is an increasingly rich strand of innovation in options to monitor, evaluate and learn
...
from both the successes and failures of policy influence and advocacy interventions. This paper explores current trends in monitoring and evaluating policy influence and advocacy; discusses different theories of how policy influence happens; and presents a number of options to monitor and evaluate different aspects of advocacy interventions. Case studies describe how some organisations have used these options in practice to understand their impact and improve their advocacy strategies." (Abstract)
more
"This document outlines basic steps in planning monitoring and evaluation for advocacy and covers: distinctive features of monitoring and evaluation for advocacy; five questions for planning advocacy monitoring and evaluation; special focuses on equity, humanitarian advocacy monitoring and evaluatio
...
n, and knowledge management; aeventeen data collection tools for measuring advocacy outputs, outcomes and impacts; dour case studies from Kenya, Nigeria, Tanzania, Mexico, Tajikistan and Iceland; following up with next steps. This is the full version of Chapter 4 in UNICEF’s Advocacy Toolkit." (Page 1)
more
"The Advocacy Toolkit provides a broadly accepted definition of advocacy and underscores UNICEF’s unique position and experience in advocacy. The heart of the Toolkit provides detailed steps, guidance and tools for developing and implementing an advocacy strategy. The Toolkit also outlines eight f
...
oundational areas that can help strengthen an office’s capacity for advocacy, and covers several crosscutting aspects of advocacy including monitoring and evaluating advocacy, managing knowledge in advocacy, managing risks in advocacy, building relationships and securing partnerships for advocacy, and working with children and young people in advocacy. Special focuses examine a variety of specific topics, including human rights and equity approaches to advocacy, theories of change, and conducting advocacy in humanitarian situations." (About the Advocacy Toolkit, page iv)
more
"Although the argument for evaluating advocacy is convincing, advocacy has long been considered “too hard to measure,” and so far relatively few advocates, funders, or evaluators have taken on the challenge. But this is now changing. Interest in advocacy evaluation is surging and cutting-edge ad
...
vocates are embracing evaluation as a critical part of their work. The main barrier preventing more organizations from using evaluation is a lack of familiarity with how to think about and design evaluations of advocacy efforts that are useful, manageable, and resource-efficient. Even knowing where to start can be a challenge. This tool was developed to help address that gap in knowledge. It guides users through four basic steps of advocacy evaluation planning." (Page 3)
more
"This paper introduces the scope of, and rational for, engaging in advocacy work as part of development interventions. It then focuses on the issue of monitoring and evaluating these efforts – offering reasons why and when these processes should be planned and implemented, what’s involved, and w
...
ho should be engaged in the process. It concludes by looking at some of the particular challenges and opportunities that the monitoring and evaluation of advocacy work presents. It also offers some helpful considerations to those who are designing or implementing these processes. The paper relies heavily on previous publications produced by INTRAC, based on work in advocacy with organisations such as DFID, Plan, Transparency International, PSI and also on a Save the Children publication, Advocacy Matters." (Introduction, page 1)
more
"This brief offers an overview of current practice in the new and now rapidly growing field of advocacy evaluation. It highlights the kinds of approaches being used, offers specific examples of how they are being used and who is using them, and identifies the advantages and disadvantages of each app
...
roach. The brief is organized around the summary matrix on page 2, which identifies four key evaluation design questions and then offers common advocacy evaluation responses to those questions.1 Questions include: 1) Who will do the evaluation? 2) What will the evaluation measure? 3) When will the evaluation take place? 4) What methodology will the evaluation use? For each question, three options or possible responses are given. Options are based on the experiences of advocates, evaluators, and funders who already have responded to these questions and are learning about the benefits and drawbacks of their choices." (Page 1)
more
"Advocacy and policy change evaluations focus on policy as the unit of analysis rather than the more traditional program or project. There is growing interest in this form of evaluation as evidenced by a new American Evaluation Association Topical Interest Group with this focus. Julia Coffman (2007)
...
began her important article “What’s Different About Evaluating Advocacy and Policy Change?” by noting what’s not different. I want to reaffirm her perspective. Advocacy evaluation, like all evaluation, is guided by the profession’s Principles and Standards. Advocacy evaluation can be, and I believe should be, utilization-focused. That means focusing the evaluation on intended use by intended users, and evaluating the evaluation by that standard." (Page 1)
more
"The overall purpose of this guide is twofold. To help grantmakers think about and talk about measurement of advocacy and policy, this guide puts forth a framework for naming outcomes associated with advocacy and policy work as well as directions for evaluation design. The framework is intended to p
...
rovide a common way to identify and talk about outcomes, providing philanthropic and non-profit audiences an opportunity to react to, refine and adopt the outcome categories presented. In addition, grantmakers can consider some key directions for evaluation design that include a broad range of methodologies, intensities, audiences, timeframes and purposes." (Introduction, page 1)
more