How children with autism spectrum disorder behave and explore the 4-dimensional (spatial 3D + time) environment during a joint attention induction task with a robot

•Using ICT, we built a system that employs a Nao robot to elicit joint attention (JA) with children and to capture social engagement cues.•Interaction with Nao depends on the partner: better in typically developing (TD) children than children with ASD.•Multimodal JA induction is more efficient in both TD and ASD.•The 3D spatial world gaze exploration shows less accuracy and the trunk position shows less stability in ASD compared to TD.

We aimed to compare, during a joint attention (JA) elicitation task, how children with autism spectrum disorder (ASD) and children with typical development (TD) behave and explore their 4 dimensional (meaning spatial 3D + time) when interacting with a human or with a robotic agent.We built a system that employed a Nao robot and a perception system based on a RGB-D sensor (Kinect) to capture social engagement cues. A JA induction experiment was performed in which children with ASD (N = 16) and matched TD children (N = 16) had a 3-min interaction with the robot or with a therapist. Nao induced JA by gazing; by gazing and pointing; and by gazing, pointing and vocalizing at pictures. Both groups of children performed well with the therapist. However, with Nao, both groups had lower JA scores, and the children with ASD had a significantly lower score than the TD children. We found that (i) multimodal JA induction was more efficient in both groups; (ii) the 3D spatial world gaze exploration showed less accuracy; and (iii) the trunk position in ASD showed less stability in the 4 dimensions compared to TD controls.We conclude that, in ASD, JA skill depends on the interaction partner, and implies a higher motor and cognitive cost.