Two collaborative problem solving projects from ACTNext are finalists in their categories, Best Research and Innovation, for the 2020 International e-Assessment Awards. The shortlists were announced March 12 by e-Assessment Association, a non-profit representing the technology-enabled assessment sector.

Best Research

The Best Research finalist is a paper by Kristin Stoeffler, Yigal Rosen, Maria Bolsinova, and Alina von Davier about the Circuit Runner game, “Gamified performance assessment of collaborative problem solving skills,” published in Computers in Human Behavior, 104, 106036.

Gamified performance assessment of collaborative problem solving skills

Summary: Assessing collaborative problem solving (CPS) is a high priority in education and workforce but designing reliable and valid assessments presents unique challenges. We developed and tested a computerized game-based CPS assessment, Circuit Runner, in which participants interact with a virtual agent to solve a series of challenges in a first-person maze. Circuit Runner provides controlled interdependence between a learner and agent that facilitates assessment of the broad range of cognitive and social skills required for effective CPS. Tasks are designed to incorporate telemetry-based (e.g., log file, clickstream, interaction-based) and item response data to provide a more comprehensive measure of CPS skills.

Our study included 397 participants on Amazon Mechanical Turk, who completed Circuit Runner, pre- and post-game surveys, and a CPS situational judgment test. These elements allowed for exploration of CPS skills with different modalities and types of instruments. The findings support and extend efforts to provide a stronger theoretical and empirical foundation for insights regarding CPS as a skillset, as well as the design of scalable game-based CPS assessments. This research will be published as Stoeffler, Rosen, Bolsinova, & von Davier. Gamified performance assessment of collaborative problem solving skills. Computers in Human Behavior, 106036.

Innovation

The Innovation finalist is based on the collaborative problem solving game, CPSx, Crisis in Space.

Innovative Tool for the Measurement of a 21st Century Competence – Collaborative Learning and Problem Solving (CPS)

Summary: Collaborative problem solving (CPS) is prominently important 21st century skills and is widely considered to be a core competency in the workforce and is a key component of several standards for education. As pointed out by researchers specializing in educational assessment and 21st century skills, the field currently suffers from a scarcity of viable and robust tools for assessing these competencies, especially in K-12 learning contexts.

In fact, the challenge of building valid assessments for CPS, cognitive and social-emotional learning (SEL) skills is two-fold. First, there is a lack of consensus on what the CPS competency model looks like and exactly which behaviors constitute evidence for CPS skills. Second, these skills, which are the primary evidence about collaboration, can be difficult to identify in any context, much less in ecologically valid collaborative experiences. To overcome these challenges, we developed an innovative assessment tool “CPSx – Crisis in Space”, for measuring students’ collaborative, cognitive, and SEL skills. We implement a new area, Computational Psychometrics, and utilize Artificial Intelligence and Machine Learning algorithms of data from rich, immersive interactions in a multitude of sensory modalities (multimodal data), can provide a new generation of assessments of 21st century competencies such as CPS competence.

Find more ACTNext CPS researchprojects and prototypes!

CPSx Crisis in Space Demo