"We developed the Evidence Quality Rubric, a tool that balances conventional research standards with inclusion factors, such as representation, power analysis, and bias interrogation, which can be applied to assess the quality of existing evidence. The Evidence Quality Rubric, applied to 20 case studies, reveals that solely using conventional standards of triangulation, methodology, and validation provides an incomplete assessment of evidence quality in participatory practices. These conventional standards exhibit a wide range of quality in the case studies, from limited to excellent. However, relying only on conventional standards neglects elements captured by the inclusion standards, such as representation, power dynamics, and bias. These are essential for evaluating participatory approaches involving excluded communities. For instance, a blog post by a refugee leader (case study #6) scored low on conventional standards due to a lack of formal methodology, but excelled on inclusion by directly amplifying refugee voices and examining power dynamics.
Conversely, even research considered strong by conventional standards, such as a monitoring report on a migration project (case study #13), had limitations in representing stakeholder perspectives and analysing power when assessed through an inclusion lens. Notably, some studies demonstrated high quality across both sets of standards. A meta-synthesis of evaluations of refugee-led organisations (case study #10) and an evaluation of a youth participatory grantmaking pilot (case study #19) employed rigorous methodologies while also centring marginalised voices and critically examining power and bias. Both examples showcase that the inclusion standards contribute to a more comprehensive, nuanced, and context-sensitive evaluation of the evidence alongside conventional standards. This expanded assessment framework is essential for recognising the unique value and challenges of participatory approaches and ensuring that the experiences and perspectives of excluded communities are adequately considered in the evaluation of evidence quality.
While the Evidence Quality Rubric has highlighted the strengths and weaknesses of various case studies in terms of conventional and inclusion standards, it is clear that a more structured approach is necessary to generate inclusive evidence that also meets the basic conventional standards. In response, we propose the Inclusive Evidence Guidelines to provide practical guidance for participatory evidence gathering that reflects the priorities of affected communities. These guidelines emphasise principles such as centring participant perspectives, co-designing locally relevant methods, conducting systematic analysis, applying intersectional lenses, and ensuring accountability to communities. Overall, we emphasise the importance of documenting participation not just to meet donor demands but to facilitate learning and promote a more holistic understanding of the effectiveness of participation in contributing to our missions. The aim is to enhance transparency, fairness, and effectiveness in philanthropic practices through documented evidence." (Executive summary, page 8)