"This report presents the findings from the evaluation of the ASTUTE media campaign. Three cross-sectional surveys were conducted before, during, and at the end of the campaign, with questions targeted to female caregivers and male heads of households when they were available. The impact of the camp
...
aign is estimated by looking at change in key indicators over time (an uncontrolled before-and-after analysis) as well as the association of indicators with reported campaign exposure (cross-sectional analysis at endline). Primary and secondary outcomes were pre-specified for each of the message themes prior to analysis. Over 21,000 men and women with children under two were surveyed across the three timepoints, and the majority of participants reported exposure to the campaign. At endline, 60% of female caregivers and 70% of men had heard or seen spots on the radio or TV. While exposure was even higher among regular radio listeners, the campaign still reached over 40% of individuals who did not report listening regularly. This high level of campaign exposure translated to large impacts across several of the programme themes. The strongest, most consistent evidence for impact was observed for engagement in early childhood development-related activities (for both men and women), for outcomes relating to maternal health and nutrition (including antenatal care attendance) and for WASH indicators. Exclusive breastfeeding indicators also improved over the time period of the campaign, and by the end of the campaign the prevalence of positive outcomes for the majority of breastfeeding indicators had reached a high level (>90%) such that the campaign may have had limited ability to achieve further improvements. Complementary feeding indicators also improved over the time span of the campaign. However the proportions of children receiving the target minimum meal frequency and minimum acceptable diet by the end of the campaign remained low with limited evidence that exposure to the campaign had had a substantial impact on these indicators." (Executive summary)
more
"There is a growing body of evidence from rigorous evaluations demonstrating the effectiveness of education entertainment – ‘edutainment’– interventions in achieving development outcomes. Building on this research, this study presents the results of a pioneering quasi-experimental evaluation
...
of Navrangi Re, a 26-episode television drama aired in India in 2019. The show was the first ever edutainment broadcast on commercial television in India. It aimed to influence sanitation behaviours through changing knowledge and attitudes, increasing risk perception, stimulating conversations, building collective efficacy, and creating social disapproval against poor faecal sludge management practices. The evaluation compared changes in outcomes of those exposed to the TV show with those unexposed, applying differences-in-differences estimation to a panel of 2,959 respondents. Baseline balance tests show high comparability between exposed and unexposed respondents. It found exposure to the drama led to significant changes in most outcomes with 37% of those who watched at least one episode showing behavioural intent to act, rising to 78% of those who had watched at least seven episodes. The show reached 59.6 million unique viewers, confirming drama as an effective, low cost and scalable tool to engage people around faecal sludge management – a critical and hard to address issue." (Abstract)
more
"No News Is Bad News (NNIBN) was an excellent programme in terms of effort, on the part of Free Press Unlimited (FPU), the European Journalism Centre (EJC) and all their partners. The programme has supported courageous, committed, energetic and initiative-taking partners who have promoted rights to
...
access to information, investigative journalism, gender equality, media literacy, content and reach in and for media dark areas, and more. The international programme has achieved some significant milestones around safety of journalists. Most of the partners report increased capacity, stronger organisations and progress on key quality, advocacy and gender goals. Many are more sustainable than before the programme started. FPU has become a learning organisation with strong skills in research, advocacy and M&E. Thanks toNNIBN, FPU has also become a leading advocacy organisation for media freedom that plays a big role in shaping international networks and initiatives.
However, as evaluators we face a conundrum – on the one hand almost all the activities went well, the partners are satisfied, the funders are happy and colleagues in other agencies are, generally, admiring of FPU and EJC. Yet on the other hand, we see few clear decisive impacts at the national level – i.e. ‘a diverse and professional media landscape’ – which is the ‘impact statement’ in the Theory of Change. So why is it difficult to say what it amounted to? Firstly, the programme was probably too thinly spread across too many countries (some with only one or two partners) to really show an impact at the media landscape level as a whole. Which means that the ultimate objective of the Theory of Change is still out of reach in most regions and countries in the programme. Secondly, advocacy and human rights work in general is a slow process, requires working in coalitions (which FPU/EJC do), often invisible, too, and that when there is a breakthrough it is hardly ever acknowledged." (Executive summary)
more
"In April 2020, as cases of the novel COVID-19 spread across the globe, MTV Staying Alive Foundation created the educational entertainment miniseries MTV Shuga: Alone Together. In 70 short episodes released daily on YouTube, Alone Together aimed to disseminate timely and accurate information to incr
...
ease young people’s knowledge, motivation, and actions to prevent COVID-19. We sought to identify Alone Together viewer’s perspectives on the global COVID-19 pandemic and national lockdowns by examining the words, conversations, experiences, and emotions expressed on social media in response to the Alone Together episodes. We also assessed how viewers used the series and its online community as a source of support during the global pandemic. Method: A total of 3982 comments and 70 live chat conversations were extracted from YouTube between April and October 2020 and analyzed through a data-led inductive thematic approach. Aggregated demographic and geographical data were collected using YouTube Analytics." (Abstract)
more
"A clustered randomized trial in two states of India examines alternate strategies to reduce child marriage, increase girls' education and change gender attitudes. GPs were randomized into four treatment and one control group in a 1:1:1:2:2 ratio. The GP level intervention from November 2012 deliver
...
ed to treatment groups i) A Full Package, comprising mass-media, training and community mobilization programs; ii) a Training Package comprising a combination of mass media and a training program at the block level; iii) a Community Mobilization Package comprising a combination of mass-media and a community mobilization program; iv) and an only Mass Media program. Data from 2,542 households seven years after the start of the program shows that the intervention made significant improvements in impact indicators on girls' education and incidence and age at marriage. As compared to the control GPs, the Full Package intervention increased the age at marriage for girls aged 13-25 on an average by 6.5 months and their education by 9 months. The Full Package also increased the percentage of girls enrolled in school by an additional 9% over the level that control group achieved (92% in Full Package vs 83% in Control Group). The program impacts were similar after controlling for stratification and other household and GP level characteristics.
We found that the intervention made an outcome indicator - gender attitudes - more progressive; GP with Full Package had a 0.407 standard deviation (equivalent to approximately 16%) higher gender attitude index than control GPs (p < 0.01). The measure of attitudes is an index of aggregated indices on gender equality, education, marriage, mobility and knowledge. Attitude change was larger for education and mobility indices. For example, the households in the Full Package had 0.26 and 0.33 standard deviations more positive attitude towards girls’ education and mobility than control GPs. We did not find any significant effect on attitudes pertaining to marriage and girl’s work and responsibility. There is little change in norms related to gender roles within the home and education has mostly been perceived as a vehicle for better management of the home, rather than for empowering women to work and be independent." (Summary)
more
"In recent years, Internews’ Earth Journalism Network (EJN) has invested time and resources to better understand the results of the support it provides to journalists and media outlets. EJN already uses a suite of methods to gauge the impact of its efforts to improve environmental media, such as e
...
valuating individual training and capacity development activities. In 2020, it commissioned a study that aimed to go a step further and support deeper learning on whether, how and why EJN’s work contributes to changes in the policies and practices of different actors. This report summarizes the study’s main findings, insights and recommendations [...] Key findings within EJN's control: EJN’s financial and mentoring support enables journalists with varied levels of experience to undertake reporting that they would otherwise find difficult. As a result, they can produce newsworthy stories on environmental issues. Research challenges mentioned earlier resulted in incomplete data, but it appears that stories supported by EJN are likely to achieve higher levels of engagement when published in local languages. Many stories published with EJN’s support are republished and shared through other news outlets and social media platforms such as Facebook and Twitter. Levels of engagement appear to vary considerably across countries and distribution channels, however data explored by the study indicates they may be higher for local language media, particularly video." (Executive summary, page 9-10)
more
"The text presents the results of the external evaluation of the project: ‘Community Radio for Peace and Coexistence’ (CRPC), implemented between 2015 and 2017 by the community media network RESANDER, with the support of the Colombian Government and the European Union. Considering that community
...
radios are key actors for the construction of public dialogue and collective action about peace and coexistence in the Colombian territories, the purpose of the CRPC project was to strengthen community radios through a process that involved: training in radio production skills and communication for peace, production of new radio content and circulation of the content produced throughout the project. The evaluation aimed to generate evidence about the role of these community radios, especially after the signature of the peace agreements between the Colombian State and the FARC guerilla. The results show the contributions of the Colombian community radios for the construction of a culture of peace and coexistence from the local territories, as well as the transformations, tensions and challenges they faced on the production of new radio content and the construction of spaces for dialogue, participation and collective action." (Abstract)
more
"The common service for community engagement and accountability, through its consortium members BBC Media Action and Translators Without Borders (TWB), aims to help agencies and sectors working to support Rohingya refugees and local host communities living in Cox’s Bazar, Bangladesh to achieve thi
...
s by providing a range of specialist, technical support services. Movement restrictions implemented since the beginning of the pandemic meant that humanitarian programmes were reduced to essential services only in the Rohingya camps, community engagement efforts were restricted, and many initiatives planned for the host community were cancelled. The common service adapted its activities in a number of ways: carrying out online training sessions with practitioners; conducting telephone research with communities; shifting the focus of communication products to meet communities’ information needs around Covid-19; and working with partners to use communication channels which would ensure information was still able to reach communities [...] This evaluation has shown that practitioners feel the common service played a critical role in getting important information to Rohingya communities during this time. It did this through:
- Helping partner agencies understand Rohingya communitiesf perspectives and concerns, and how to communicate with them, based on up to date research and a cultural understanding developed over time (using What Matters? as a vehicle for sharing this information, as well as providing practitioner training and bespoke advice).
- Creating and disseminating audio and visual content which is easy for Rohingya people to understand, and helps volunteers and field staff communicate effectively with them.
- Working in partnership with humanitarian agencies to adapt communication strategies and make sure information was reaching people despite Covid-19 restrictions, for example adapting audio programmes to be played through mosque loudspeaker systems; training Imams to communicate on Covid-19; ensuring content is played at food distribution points and other essential services; and equipping community health workers with simple and effective communication materials about Covid-19 to use in door to door visits." (Executive summary)
more
"Colombia’s history of violence and armed conflict has eroded people’s trust in each other and trust, which ranks low among the components that Colombians consider important for reconciliation, is difficult to restore during the course of a project. Still, the Program of Alliances for Reconcilia
...
tion (PAR) has achieved changes in perception that represent an important component in changing mindset that could eventually extend through the communities where participants live and work. Though implementers recognized positive effects on perceptions of reconciliation among participants, these further effects in communities may only be visible among non-participant indirect beneficiaries. Similarly, though PAR participants overwhelmingly reported positive effects of PAR participation on their behavior and some drivers of reconciliation, examples were limited to their personal thoughts and actions and not those of their communities. Without a baseline for comparison, the degree of shifts in attitudes and behaviors cannot be determined, but indications of changes in attitude paired with reporting of ongoing participation and continued use of skills indicate participants remain changed, at least in the short term, by their participation in PAR activities. DecidoSer workshops and the Ambassadors for Reconciliation initiatives are successful tools that provide the opportunity to identify lessons learned and good practices in the management of community changes beyond individual changes. DecidoSer is valuable not just for participants, but implementers (who are themselves members of communities whose contributions to reconciliation are worthwhile and necessary) as well, and the changes in trust and dialogue that contributed to improved business practices are a positive (and perhaps unexpected) result. Respondents generally saw the conceptual benefits of both DecidoSer and Ambassadors for Reconciliation for sustainability of reconciliation, though without specific evidence for believing this, or enough time to demonstrate it. Contrary to the notion that multi-sectoral relationships within a community might contribute to sustainability, the evaluation team did not find evidence of a correlation between PAR’s activities in a community and long-term engagement with other community actors." (Conclusions, page 28)
more
"Since August 2017, USAID/Tanzania’s five-year Data-Driven Advocacy (DDA) Activity has sought to improve and sustain Tanzanian civil society’s ability to advocate for and influence policy on key national human rights issues using higher-quality data and information to drive results. Implemented
...
by Freedom House in collaboration with Pact, DDA is funded at a total estimated cost of $8,700,000. The Activity provides mentorship and technical assistance to rights-focused civil society organizations (CSOs) in data collection, evidence-based advocacy, and digital and physical protection. DDA helps CSOs build or strengthen platforms for coordination with stakeholders and strengthen institutional capacity. It also provides CSOs with resources and mentorship to engage the GoT and encourage a more inclusive, data-driven, human rights–focused policymaking process [...] USAID asked the Data for Development Project to conduct this evaluation to assess how DDA has performed in relation to its specific objectives and performance targets. The evaluation gathered data to assess progress, challenges, and constraints to implementation and performance. It identified potential needs for adaptation or course correction in the remaining activity period. This utilization-focused evaluation also leveraged lessons learned to inform future USAID projects in Tanzania. Specifically, USAID and its Democracy, Human Rights, and Governance (DRG) Office plan to apply evaluation findings, conclusions, and recommendations to thinking and planning for future design." (Executive Summary, page vii)
more
"The purpose of this ex-post evaluation is for Internews to understand the impact that Safe Sisters has had on women in East Africa by documenting and publicly sharing their stories and evaluating the training approach itself. As women and girls come online for the first time, they disproportionatel
...
y face violence, which creates risks to their physical safety and stifles their ability to capture the Internet’s transformative economic and social potential. Internews, together with partner DefendDefenders — a Uganda-based non-governmental organization (NGO) — developed the Safe Sisters program, a year-long fellowship that provides women human rights defenders and journalists with the techniques and tools they need to navigate online spaces safely, assume informed risks, and take control of their lives in an increasingly digital world." (Publisher description)
more
"Studies have shown that many irregular migrants embark on journeys without accurate or complete information about migration. In response, IOM and other organizations run information campaigns intended to inform potential migrants of the dangers associated with irregular migration and to facilitate
...
informed decisions. The use of online and social media platforms for these campaigns has become increasingly popular in recent years, due to their potential for quickly reaching millions of people at low cost. While the evidence base for offline information campaigns has been gradually improving, it remains unclear how online communication campaigns affect potential migrants, and methodologies to do this remain underdeveloped. This report therefore presents results from a pilot impact assessment of an online campaign based on Facebook posts in Guinea, Nigeria and Senegal during September 2019 and February 2020 [.] Conclusion: Facebook ad campaigns can reach many people quickly. However, large audience sizes and relatively cheap costs come with a trade-off. The impact of an information campaign on potential migrants’ perceptions, attitudes, knowledge and behaviour remains largely unknown because many Facebook users that engage with the campaign do not fit the intended target group. The advantage of offline over online campaigns is that implementers have more control and information about who attends activities or who is exposed to campaign content. The engagement of users with campaign posts varies largely by audience and post characteristics. Campaign and social media teams are well advised to conduct pilot tests before scaling up activities. Pilot tests can inform a customized approach for each online activity on Facebook, and thereby maximize impact. In addition to A/B tests, lift tests may offer a useful opportunity to test the impact of Facebook posts. Compared to impact assessments of offline information campaigns, measuring the impact of Facebook ads on potential migrants presents several unique challenges, including: Many potential migrants are not on Facebook or do not use ad content to seek information about migration. Internet connectivity is also limited in many West African countries. While Facebook allows users to narrow down target groups, it is not clear whether those Facebook users that are exposed to the content fit the profile of potential migrants. Whether the intended target group is reached can be affirmed only through surveys, which have a low response rate; It is not possible to interview the same Facebook users several times to track changes in their perceptions. Engagement metrics like post reactions, link clicks or video plays can be useful indicators. However, these low-engagement metrics do not offer clear evidence of the short- or long-term effects on Facebook users." (Executive summary)
more
"This paper presents comparative learning from the evaluation of six international development initiatives that applied various forms of Process Tracing. While these initiatives span across diverse contexts and pursued different aims, they are connected by a common thread: all six case studies centr
...
e around efforts to influence others - often decision makers and those in power - around aspects such as practices of consultation and inclusion; public policy; and resource allocation. The paper is organized in the following manner. We first explain Process Tracing and review common definitions. Secondly, we consider the potential value added of an explicitly Bayesian approach to Process Tracing. Next, we discuss the six cases where Process Tracing was applied, noting similarities and differences. Then, we explore key practical learning emerging from the cases and insights from the use of different forms of Process Tracing across different programming contexts. These reflections are organized under four meta-themes of participation, Theory of Change, methodological decisions, and mitigating bias. Finally, we present our key recommendations, ending with practical tips, targeted at practitioners and evaluators interested in applying Process Tracing, especially for initiatives falling under the ‘influencing’ umbrella." (Introduction)
more
"The factors that led to the overall unsatisfactory assessment of the project can be summarised as follows:
• The project was not anchored in any overarching reference framework. Accordingly, neither its contributions to the strategic objectives of the German development cooperation nor to any oth
...
er national or international development agenda were specified. While this has nothing to do with the actual quality of the project, as already outlined in the introduction, it leaves the question of whether the OECD/DAC criteria were suitable to evaluate it.
• The organisational setup of the project was inefficient. Having two intermediary organisations with separate overheads, trying unsuccessfully to change a national implementing partner, and finally needing to involve GIZ staff on the ground increased the costs and resulted in an inefficient use of financial resources.
• During the planning of the project, fundamental rules for German development cooperation projects were disregarded: there were no preceding government consultations; implementing partners were chosen without considering alternatives (particularly at political level); neither a risk assessment nor stakeholder mapping took place; and the instruments to be applied were defined beforehand. This procedure appears unusual for GIZ.
• The lack of results at impact level can partly be attributed to the fact that the project did not have a political partner. The professionalisation of an industry/economy requires support from political decision-makers as they have the ability to establish and enforce an appropriate regulatory framework. For instance, the introduction of a tax incentive for film productions could only be achieved through collaboration with the ministries for finance and culture." (Conclusions, page 45)
more
"Justice4Her is a cross-sector, multi-stakeholder project of RNW Media and Qianqian Law Firm that seeks to strengthen the rule of law and reduce gender-based violence (GBV) towards women migrant workers in China. The project launched in 2016 and targets women migrant workers in Beijing, Shanghai and
...
Guangdong Province and was scaled further with co-funding from the European Commission in 2018. Justice4Her provides legal support through a digital interactive platform and disseminates information on GBV to millions through online channels (social media platforms, blogs, email, WeChat groups) and integrates this online work with offline activities. The project has increased mainstream media coverage in China of how GBV affects young women migrants and carried out viral online campaigns to raise public awareness and to challenge tolerance and acceptance of GBV. While advocating for the improvement of anti-domestic violence laws in China, Justice4Her provides pro bono legal aid to victims of GBV in partnership with Beijing Qianqian law firm, a local law firm, both in the form of consultations via mail and support in bringing their cases to court and representing them. The project has also trained pro bono lawyers to better represent victims, media professionals to encourage more gender sensitive reporting on cases of GBV and NGOs on how conduct successful anti-GBV media campaigns and effectively counter negative stereotypes and shift public discourse from a victim blaming narrative to a survivor narrative." (Introduction)
more
"The Theories of Change for both Citizens’ Voice [a programme focusing on social cohesion and inclusive governance] and the SRHR programme (which includes Love Matters [delivering information and media on love, sex and relationships to 18-30-year-olds]) contain a key assumption that through provid
...
ing reliable information on our digital platforms, young people gain new knowledge and understanding on topics that matter to them. For Citizens’ Voice, the assumption is that by accessing pluralistic information and engaging in dialogue with people with other backgrounds and perspectives, young people are aware of different viewpoints, accept and respect diversity, challenge restrictive norms and participate in society. For Love Matters, it relates to providing evidence-based and pleasure-positive SRHR information that helps people to make informed decisions in their personal lives. To investigate whether this assumption holds true for both programmes, a dedicated survey question was included in the surveys of platform users conducted as part of the Next Generation Mid-Term Review (MTR) evaluation conducted by external researchers in 2019. This open question asked users whether they could give an example of a change they had experienced through engaging with the platform. The answers and examples given by users have been analysed by the PMEL team to answer the main research question: To what extent do returning users experience a change in knowledge, attitude or behaviour as a result of engaging with the platform?" (Introduction)
more