Measuring transformative agency in PBL and e-PBL: development and validation of an assessment rubric

Abstract This study aimed to design and develop a coding scheme rubric for evaluating students’ transformative agency using a modified qualitative assessment approach. Data collection was conducted in two phases: rubric development and rubric application. In the first phase, a comprehensive literature review and qualitative analysis of recorded conversations using conversation analysis (CA) in project-based learning (PBL) within Change Laboratory (CL) contexts led to the identification of 77 codes, with the assistance of four coders. To ensure validity, a panel of five experts reviewed the assessment tool and provided feedback on its appropriateness for measuring learners’ transformative agency. Based on their recommendations, revisions were made, resulting in a coding scheme rubric comprising six key categories: resisting change, criticizing current practices, explicating new possibilities, envisioning new possibilities and models, committing to specific actions, and taking consequential actions. The rubric included three levels of assessment: Needs Improvement (1), Progressing (2), and Mastery (3). Reliability was established through inter-coder agreement (81.70%) and Fleiss' kappa (κ = .812). In the second phase, the rubric was applied in a group project task involving 60 students in both face-to-face (PBL) and online (e-PBL) learning environments. Two independent raters used the rubric to assess students’ transformative agency within groups, evaluating inter-rater reliability and consistency across PBL and e-PBL contexts. The findings highlight the significance of this validated and reliable rubric for assessing transformative agency, with implications for teaching, learning, and future research on student agency in active learning environments.