Researchers have demonstrated a new suite of software tools that analyzes student behavior in an educational game in real time and uses that data to assess how well students are developing and making use of collaborative problem-solving (CPS) skills. These real-time assessments can be used to modify the game in response to student behavior in order to improve learning.
“CPS skills are essential for helping students solve complex problems effectively,” says Wookhee Min, co-author of a paper on the work and a senior research scientist at North Carolina State University. “And while there are established techniques for helping students use and develop those CPS skills, it can be difficult to measure student performance without disrupting the learning process to have students take tests.”
“For this work we wanted to develop non-intrusive techniques to measure the cognitive aspects of CPS, as opposed to measuring social aspects of CPS,” says Halim Acosta, first author of the paper and a Ph.D. student at NC State. “And we wanted to do that in real time, so that in the future we can program the game to respond to student behavior in a way that helps students build those critical CPS skills.”
For this project, the researchers worked with a science education video game called EcoJourneys, which is aimed at middle school students.
The analytic framework the researchers developed hinges on logging every action that a student takes when playing the game. The researchers had 61 students, in grades 6-8, play EcoJourneys. The researchers then used statistical modeling techniques to analyze student actions to see whether they could identify patterns of behavior associated with specific learning outcomes.
“We found there are patterns of CPS behavior associated with significant growth in learning, as well as patterns associated with limited improvement in learning,” Acosta says. “In addition, the analysis offers important insights we will be able to use to determine what sorts of interventions could help us improve learning and at which points in the game. So, it’s not just which patterns of behavior seem beneficial, but when and where those patterns seem to make the biggest difference.
“For example, if students make a specific series of choices at one stage in the game, that may suggest that they are not grasping some key CPS concepts,” Acosta says. “We could modify the game so that if students make that series of choices, the game changes in a way that emphasizes or reinforces those concepts.”
“Future steps for this work involve modifying the game to incorporate changes based on student behavior in the game,” says Min. “One of the attractive things about this framework is that it can evolve as more students play the game and the software has more data to draw on. In theory, that should allow us to fine-tune in-game interventions in order to improve learning outcomes even more.”
“We’re also excited about this work from a technical standpoint because we think this is the first time anyone has used these constraint-based pattern mining algorithms to predict learning outcomes, or even in educational contexts more broadly,” says Acosta.
The paper, “Collaborative Game-Based Learning Analytics: Predicting Learning Outcomes from Collaborative Problem-Solving Behaviors,” will be presented at the Fifteenth International Learning Analytics & Knowledge Conference (LAK25), which is being held March 3-7 in Dublin, Ireland. The paper was co-authored by Seung Lee, a research scientist at NC State; Bradford Mott, a senior research scientist at NC State; James Lester, the Goodnight Distinguished University Professor in Artificial Intelligence and Machine Learning at NC State and director of the university’s Center for Educational Informatics; and Daeun Hong and Cindy Hmelo-Silver of Indiana University.
The work was done with support from the National Science Foundation under grants 2112635, 1561486 and 1561655.