Since its inception, ACTNext has targeted the assessment of collaborative problem solving (CPS) as a cross-cutting capability.
Here are some highlights of ACTNext’s CPS work:
The goal of the ACTNext CPS X project is to understand what kinds of skills and communication patterns are representative of high performance in collaborative tasks and develop automated analysis techniques that will surface these patterns. In CPS X, players work together to achieve group goals in pre-programmed challenges. By measuring players’ real-time behavior, researchers can determine qualitative data about effective collaboration.
In 2018, ACTNext enlisted the help of the ACT Summer Intern cohort to participate in a pilot study of collaboration in cooperative games. Participants paired up to play a game that targets collaboration and teamwork. Their gameplay data and communication patterns were analyzed with AI and Machine Learning models to infer collaboration skills. One method was recording a transcript of the player interactions and analyzing them using Natural Language Processing.
In Spring 2019, the CPS X team conducted a field study with over 100 middle and high school students.
This story about the CPS X game study aired on KCRG-TV on May 14, 2019.
Circuit Runner game
With partners at Digital Artefacts, ACTNext developed a novel, simulation-based approach to elicit skill evidence in a task-oriented, scripted agent environment. The assessment instrument, Circuit Runner, is an immersive puzzle game where a student collaborates with a computer agent to solve a number of challenges presented at locked gates in a maze, and then solve the final challenge of repairing a virtual circuit.
The team designed the agent/bot dialog trees and the scoring rules, and collected data via crowdsourced game play. Working with Dr. Yigal Rosen from Harvard University, they analyzed the rich process data that comes from the log files using data mining and psychometric methods, then refined the game’s alignment with ACT’s Holistic Framework. In the second part of the study, the game will be modified to allow two students to collaborate directly on solving the same set of puzzles.
The “Teaching Teamwork” project, funded by the National Science Foundation, simulated electronic tasks and collected data from vocational schools and community colleges across the country. ACTNext team members supported the principal investigator, Paul Horwitz from the Concord Consortium. Please see the following links for examples of the activities for both the public and for the participating students. Dr. Horowitz outlines the goals of the “Teaching Teamwork” project in this 2016 NSF Video Showcase.
“Multimodal Analytics for Teamwork Assessment under Multiple Casualties” is a US Army-funded project in its second year. ACTNext’s Alina von Davier started as the principal investigator, now succeeded by ACTNext Director of AI/ML Saad Khan. The goal of the project is to use sensors, cameras, microphones, and human raters to collect and analyze data from medical military teams under multiple casualty scenarios.