Toolbox Evaluation & Impact Assessment

"Artificial Intelligence and Evaluation: Emerging Technologies and Their Implications for Evaluation is a groundbreaking exploration of how the landscape of program evaluation will be redefined by artificial intelligence and other emerging digital technologies. In an era where digital technologies a ... more

"This report examines whether and how AI might serve African monitoring, evaluation, research and learning (MERL) systems—or whether these technologies should be adopted at all. We argue that Africa's relationship with AI must be shaped through locally-defined priorities, community participation, ... and epistemic sovereignty rather than externally imposed technological agendas. While AI promises enhanced data analysis and real-time insights, our research reveals fundamental misalignments between current AI tools and African contexts. The systematic absence of African languages from AI training data creates erasure, rendering entire communities invisible in evidence gathering. AI tools consistently fail to surface African research, instead reproducing Western evaluation frameworks inappropriate for African contexts. These are not merely technical limitations but political choices about whose knowledge matters. Using a qualitative design, we interviewed 23 AI experts, technologists, evaluators, civil-society leaders, and policymakers from across Africa and the global MERL ecosystem, complemented by literature review and practitioner consultations with our AI in AfricaMERL Working Group. Our findings: Technical and structural barriers: African MERL practitioners face interconnected challenges including limited digital infrastructure, fragmented data systems, and educational institutions unprepared to integrate AI with evaluation practice. Current regulatory frameworks focus on protecting against AI risks while risking failing to enable African-led innovation; Reframing capacity deficits: The fundamental gap is not African practitioners' ability to use Global North AI systems, but rather these systems' lack of indigenous data, nuance, and contextual sophistication to capture African realities. Dominant evaluation frameworks themselves require capacity development to understand African epistemologies; Made in Africa as resistance and creation: Practitioners define African AI approaches through community ownership, participatory design, African language inclusion, locally-controlled infrastructure, and Ubuntu values prioritizing collective benefit. This represents both resistance to epistemic hegemony and creation of alternative systems; Environmental and human rights concerns: AI's development material and global supply costs, from conflict mineral extraction in the DRC to massive energy consumption, reveal how African resources fuel technological development controlled by and benefiting actors outside the continent." (Executive summary, pages 4-5) more

" Guided by the CGIAR-wide Evaluation Framework, integrating AI tools requires a governance approach to balance innovation with ethical responsibility, ensuring transparency, fairness, accountability, and inclusivity. This Technical Note encourages and guides CGIAR evaluators to ethically explore, n ... egotiate, and experiment with AI tools: Explore: Evaluators are invited to discover how AI, especially GenAI, can enhance evaluation efficiency, from scoping and data analysis to reporting. The Note provides practical guidance on AI applications and examples to support creative yet responsible exploration. Negotiate: Integrating AI should be openly discussed with commissioners, stakeholders, and teams. The Note prioritizes jointly defining boundaries, expectations, and ethical parameters (transparency, accountability, and data sensitivity) at each phase of evaluation. Use AI Responsibly: While AI tools are evolving, evaluators are encouraged to pilot and iterate their use. The document supports experimentation through practical tips, prompt examples, and tool selection criteria, all while emphasizing documentation and learning from each use case. Effective AI governance is grounded in core principles: Transparency requires clear documentation of AI tool usage, data sources, model limitations, and decision-making processes. Accountability involves assigning responsibility for AI decisions and outputs and establishing oversight and redress mechanisms. Fairness and inclusion must proactively mitigate bias and discrimination, with particular attention to underrepresented groups and data gaps. Data privacy and security must align with applicable data protection regulations and ensure secure handling practices. Human oversight ensures that evaluators retain control over processes and can intervene as needed. In operationalizing ethical AI governance in CGIAR evaluations, due diligence is required in assessing AI tools for ethical alignment before deployment: reviewing the transparency of vendors, the documentation of models, and their intended use cases. Where relevant, components involving AI—especially those engaging human subjects or sensitive data—should undergo ethics review. AI applications must be adapted to the local and cultural contexts in which evaluations are conducted, as what is suitable in one setting may be inappropriate in another. Additionally, participants should be informed about the use of AI systems and the implications of data collection or processing to ensure informed consent. Ethical AI governance should be embedded in the entire evaluation lifecycle. During the design phase, evaluators should define AI tools to use, why they are selected, and assess risks. In data collection, AI tools should be used in ways that uphold data privacy and protection standards and avoid reinforcing harmful stereotypes or excluding groups. During the analysis phase, the role of AI in supporting interpretation should be documented, with an acknowledgment of limitations or biases. In dissemination, documentation, and reporting, AI’s contribution, limitations, and human validation should be disclosed. By rapid adaptation of content across formats, languages, and complexity levels, AI opens possibilities for broader, more inclusive communication of findings. Finally, the follow-up phases should include a reflection on the ethical implications observed and how these lessons can improve future evaluations." (Executive summary) more

Artificial Intelligence and Evaluation: Emerging Technologies and Their Implications for Evaluation

Made in Africa: Artificial Intelligence for Monitoring, Evaluation, Research and Learning (MERL). A Practitioner Perspective and Landscape Study

Considerations and Practical Applications for Using Artificial Intelligence (AI) in Evaluations

Machine Learning and Meta-Ethnography: Seven Steps to Synthesising 578 Evaluations Into Four Themes