Using non-verbal cues to (automatically) assess children’s performance difficulties with arithmetic problems

Intelligent tutoring systems often make use of students’ answers to adapt instruction or feedback on a task. In this paper, we explore the alternative possibility of adapting a system based on the perceived affective and cognitive state of a student. A system can potentially better adapt to the needs of each individual student by using non-verbal behavior. We used a new experimental paradigm inspired by ‘brain training’ software to collect primary school children’s answers to easy and difficult arithmetic problems and made audiovisual recordings of their answers. Adult observers rated these films on perceived difficulty level. Results showed that adults were able to correctly interpret children’s perceived level of difficulty, especially if they saw their face (compared to hearing their voice). They paid attention to features such as ‘looking away’, and ‘frowning’. Then we checked whether we could also automatically predict if the posed problem was either easy or difficult based on the first second of their response. This ‘thin-slice analysis’ could correctly predict the difficulty level in 71% of all cases. When trained on sufficiently many recordings, Adaptive Tutoring Systems should be able to detect children’s state and adapt the difficulty level of their learning materials accordingly.

• Children show performance difficulties in voice and face when learning.
• Adults can correctly interpret this non-verbal behavior.
• Automatic detection of facial expressions is 71% correct within 1 s.
• Tutoring systems can be developed that take learner’s state into account.