Beyond academics, a broad range of skills (e.g., creativity, communication, collaboration, problem solving) are required to thrive in a 21st century society rich with both information and opportunity. These same skills empower students to fulfill their potential as effective and creative knowledge seekers and collaborative problem solvers.

Measurement and training of such varied skills though present challenges for assessment and education professionals, but also opportunities. This project focuses on Collaborative Problem Solving (CPS), which is part of ACT’s Holistic Framework—a comprehensive description of the knowledge and skills individuals need to know and be able to do to succeed at school and at work (Camara et al., 2015).

Deeper learning environments, such as those found in simulations and games, place students in immersive settings that can be used to gather evidence of the cognitive intrapersonal and interpersonal skills and behavior (Dede, 2014). “Stealth assessment” enables the measurement of a specific skill, such as the CPS skill, while the subject performs tasks in a complex context, such as a game setting, without an explicit separation of the testing activity from the performance activities. (Shute, 2015)

To that end, ACT and Digital Artefacts collaborated to design, develop, and deploy a video game for middle school students. In the collaborative game, Circuit Runner, a student navigates a circuit board maze and interacts with a student “bot” to solve a number of challenges presented at locked gates (Figure 1). Varying levels of collaborative behaviors, communication, and problem solving skills are required to successfully navigate the challenges and unlock the gates.

Over 350 students between the ages of 11 and 14 have played the game. The game was also accompanied by a set of game usability and research survey questions. We will review game features designed to elicit evidence of CPS skills, insights gleaned from the design and development processes, as well as discuss initial learnings.

We conducted multiple clustering analyses of the evidence provided over a series of game plays  that is based on approaches defined by the AI/educational data mining communities (Bauckhage et. al., 2015; Kerr and Chung, 2012). In Figure 2, we show a view of the clusters derived from a K-Means (K=8) analysis over normalized skill score dimensions. Each dot within the inner column represents a game instance skill score for three of the skill categories: Engagement (EN), Finding Information (FI) and Monitoring Understanding (MU). The outermost column/ dot color encoding, represents a unique clustering of game scores. A black line denotes the mean value. This analysis provides an overall labelling of game performance and can also be used to perform queries/comparisons between games/players, e.g. select K nearest neighbor (K-NN) (Arya et. al, 1998) associations of games/players. In addition to this we also performed a Mixture Model clustering of Gaussians (McLachlan and Basford, 1988) that makes soft assignments of game evidence to clusters. This yields a probability result for each game to a particular category/cluster group. The clusters seem to be ordered, which leads to a hypothesis of subskill unidimensionality and to future research using item response theory (IRT) models.


Dede, Chris. “The Role of Digital Technologies in Deeper Learning.” Students at the Center: Deeper Learning Research Series. Boston, MA: Jobs for the Future (2014).

Shute, Val. “Creating stealth assessments.” Learning Assessments (2015): 3.

Kerr, Deirdre, and Gregory KWK Chung. “Identifying Key Features of Student Performance in Educational Video Games and Simulations Through Cluster Analysis.” JEDM-Journal of Educational Data Mining 4.1 (2012): 144-182.

Bauckhage, Christian, Anders Drachen, and Rafet Sifa. “Clustering Game Behavior Data.” IEEE Transactions on Computational Intelligence and AI in Games 7.3 (2015): 266-278.

Arya, Sunil, et al. “An Optimal Algorithm for Approximate Nearest Neighbor Searching Fixed Dimensions.” Journal of the ACM (JACM) 45.6 (1998): 891-923.

McLachlan, Geoffrey J., and Kaye E. Basford. “Mixture Models. Inference and Applications to Clustering.” Statistics: Textbooks and Monographs, New York: Dekker, 1988 1 (1988).

Camara, Wayne, et al. “Beyond Academics: A Holistic Framework for Enhancing Education and Workplace Success. ACT Research Report Series. 2015 (4).” ACT, Inc. (2015).