Thursday, April 27, 2017 Half Day Morning Sessions

8:00 a.m.-12:00 p.m.

Evidenced-Centered Design and Computational Psychometrics Solution for Game/ Simulation- Based Assessments

Jiangang Hao, Alina von Davier, Kristen DiCerbo, and Robert Mislevy

Game/simulation based assessment has a number of advantages over traditional assessment, and is widely considered as an important future direction for assessments (Cope & Kalantzis, 2015). Evidence Centered Design (ECD, Mislevy, Steinberg, & Almond, 2003) provides a theoretical framework for designing game/simulation-based assessments around a validity argument that connects a test taker’s activities to performance on some predefined construct of interest. However, the implementation of ECD principles during the actual development of a game/simulation-based assessment is not necessarily simple. It generally involves rounds of iterations among specialists with different areas of expertise, such as learning

scientists, software developers, data scientists, and psychometricians. In particular, assessing a test taker’s performance depends on having a well-structured record of the player’s activities and situational variables, and having techniques and tools to effectively analyze those records to determine the degree to which the player’s activities reveal the targeted constructs.

The process data that record a test taker’s activities and the situational/environmental variables are very complex. The skills needed to parse, aggregate and model this type of data are generally not well covered in most educational measurement programs. The goal of this training session is to promote consensus among researchers about how to efficiently implement ECD in practice, summarize the computational psychometrics modeling techniques of the process data, and provide a data solution for efficient and cost-effective analytics for game/simulation-based assessment via a set of hands-on exercises.

The whole session is divided into three parts: a) ECD and its implementation; b) data model, analytics and python programming language; and c) computational psychometric modeling of process data. Part a) is intended to give a practical guide about how the ECD principles are implemented in real game/simulation-based assessments with emphasis on those practical considerations that can make the process efficient. Part b) introduces a comprehensive data solution that involves a set of recommended operational procedures to ensure evidence capturing, a data model for log file structure (Hao, Mislevy, von Davier, & Smith, 2014), and a Python library/package, glassPy (Hao, Smith, Mislevy, von Davier, & Bauer, 2016), for evidence identification, aggregation and analytics. The part c) focuses on summarizing the methodologies used to model the process data generated from game/simulation-based assessments, and some exemplar use cases will be introduced.

Keywords: Evidence Centered Design, game assessments, performance, simulation-based assessments