April 30, 2016
On the Role of Multilevel Item Response Models in Multisite Evaluation Studies for Serious Games
Authors:
Li Cai, Kilchan Choi, and Megan Kuhfeld
In this book chapter, we propose a generalizable solution to measurement error and multilevel modeling issues in multisite randomized controlled trials (RCTs). We argue that traditional use of summed scores or conventional IRT-based scaled scores are not optimal as outcomes. We propose a multilevel two-tier (MTT) item factor model for scaling item responses in multisite RCTs with pre- and post-tests. The MTT model is demonstrated with data from a multisite RCT assessing the instructional effects of video games on pre-algebra math topics. The new outcome measure derived from MTT analysis shows more sensitivity to intervention than standard measurement approaches.
Cai, L., Choi, K., & Kuhfeld, M. (2016). On the role of multilevel item response models in multisite evaluation studies for serious games. In H. F. O’Neil, E. L. Baker, & R. S. Perez (Eds.), Using games and simulations for teaching and assessment: Key issues (pp. 280-301). New York, NY: Routledge.|Cai, L., Choi, K., & Kuhfeld, M. (2016). On the role of multilevel item response models in multisite evaluation studies for serious games. In H. F. O’Neil, E. L. Baker, & R. S. Perez (Eds.), Using games and simulations for teaching and assessment: Key issues (pp. 280-301). New York, NY: Routledge.