Collaborative problem solving (CPS) is a skill critical for the 21st century workforce, particularly in STEM fields. Assessment of CPS skill has thus received increasing attention. This paper describes a program of research in which we seek to design a suite of CPS simulation-based electronics tasks suitable for assessment use. We focus on the pilot study of one task, describing the task design and subsequent modifications implemented to better capture evidence of students’ skills.