By: David Edwards, Lead Data Scientist, ACTNext
Games like Fortnite, League of Legends, DOTA2, and World of Warcraft require individuals to collaborate in groups to achieve success, and both players and viewers enjoy the complex interactions and team-based dynamics present in these games. Players work together to achieve group goals, competing against other players, or pre-programmed challenges created by content/game publishers. These collaborating teams require high levels of leadership, coordination, information sharing, and at times rapid responses to novel situations. Yet, when it comes to assessment, this vast reservoir of leadership skills, teamwork, collaboration, and social skills remains untapped. Just as the military benefits from improved soldier hand-eye coordination developed through gaming, society could utilize this channel of leadership development and teamwork already taking place at massive scale online to benefit learners and improve socioemotional skills. Specifically, we can learn from players’ real-time behavior about what constitutes effective collaboration, and help them leverage the 21st century skills they’re already using to better prepare them for education and career success.
This is the goal of the ACTNext CPS X project. We aim to understand what kinds of skills and communication patterns are representative of high performance in collaborative tasks and develop automated analysis techniques that will surface these patterns. In 2018, ACTNext enlisted the help of the ACT Summer Intern cohort to participate in our pilot study of collaboration in cooperative games. Participants paired up to play a game that targets collaboration and teamwork in a novel way, where their gameplay data and communication patterns could be analyzed with AI and Machine Learning models to infer collaboration skills. One way we did this was by recording a transcript of the player interactions and then analyzed them using Natural Language Processing (model to target) and extract specific details from the transcripts, including asking and answering clarifying questions, and were thus able to observe information sharing between participants.
Our game is not a fast-paced team combat game like those mentioned above, but rather it divides play into two distinct roles; the Operator and the Engineer. Both roles have unique information and interfaces, and success requires users to identify key information, and share it with their partner in order to make progress. This structured task allows us to target specific skills and behaviors and identify key features of collaboration. There is plenty of information to share, and only some of it is useful. As the users sift through both the useful and useless information, the signals in the data expose both good collaboration, as well as skill gaps. These signals are mapped to different dimensions of the Collaborative Problem Solving construct in the ACT Holistic Framework, and accumulated over time to develop measures on each dimension. This framework from ACT breaks collaboration into small, measurable components, including individual and team performance, as well as both team- and task-related features.
ACTNext will develop measurement models that can be evaluated through passive data collection, and analyzed at scale to produce profiles of users without them explicitly answering any survey questions. By collecting and evaluating data while users are actually participating in collaborative problem solving, the measurement has a chance of being more representative of user’s actual soft skills. This integrates the measurement into activities the users are already interested in doing, and avoids common pitfalls of surveys which simply ask users to rate themselves on their collaboration capabilities, rather than actually measuring their performance of these skills. This data is collected through routine collaborative game play and observing users’ performance of collaboration behaviors and skills, or the absence of these skills.
Increasing patterns of remote work rely heavily on clear communication and teamwork, online schools and courses require students to exercise their collaboration skills too, and communicate effectively with instructors. Additionally, innovation today and in the future will come from increasingly interdisciplinary teams that must work together and communicate among disparate fields to bring new technology to market. Collaborative problem solving is one of a critical set of 21st century skills that are needed in school and the workplace, and developing a scalable system that can support measurement and learning opportunities could make a significant difference. Stay tuned for more developments from ACTNext’s project – CPS X.