Qubahan Academic Journal | 2026
Authors: Klyprayong R.; Tangdhanakanond K.; Kanjanawasee S.
DOI: 10.48161/qaj.v6n2a2462
Journal: Qubahan Academic Journal
Year: 2026
Publisher: Qubahan
Document Type: Article
Open Access: All Open Access; Gold Open Access; Green Open Access
Cited by: 0
Reliable assessment of classroom action research skill is essential for supporting valid absolute decisions in teacher education, particularly when performance assessments involve multidimensional constructs and rater-mediated judgment. This study investigated how alternative measurement designs influence the index of dependability and variance structure of classroom action research skill assessment scores within a multivariate generalizability theory (MGT) framework. Specifically, the study compared fully crossed and nested measurement designs and examined the number of raters required to achieve acceptable dependability for absolute decision-making. The participants consisted of 58 fourth-year student teachers majoring in primary education whose classroom action research reports were evaluated by four raters using a multidimensional assessment form aligned with the Plan–Act–Observe–Reflect (PAOR) framework and supported by double-layer scoring rubrics. Data analysis was conducted sequentially using the many-facet Rasch model (MFRM) to examine rater effects, followed by MGT-based generalization and decision studies. The findings showed that the fully crossed design produced a higher composite index of dependability than the nested design (Φ = .8468 vs .7823) and generated different composite universe score variance structures. Under the fully crossed design, three raters were sufficient to achieve acceptable dependability for individual-level absolute decisions, whereas the nested design required four raters to reach a comparable level. The study contributes to educational measurement literature by demonstrating that measurement design influences not only the magnitude of dependability but also the variance structure underlying multidimensional performance assessment scores. The findings further highlight the importance of aligning measurement design with the intended interpretation and use of assessment scores in teacher education contexts. © 2026, Qubahan. All rights reserved.
Classroom action research; Index of dependability; Measurement design; Multivariate generalizability theory; Performance assessment