Computational modeling of user activity in full-body interaction environments for ASC children: multimodal analysis of social interaction behaviors through psychophysiology, system activity, video coding, questionnaires, and body cues

Abstract

Full-body Interaction experiences based on Mixed Reality (MR) systems are already playing an important role in encouraging socialization behaviors in children with Autism Spectrum Condition (ASC), as seen in the state of the art of this thesis. However, the data from these systems is multimodal in nature and complex to analyze. Fusion and analysis of this data is crucial to achieve a complete understanding of how these resources interact with each other. In this PhD Thesis, given the characteristics of full-body interaction, we developed new multimodal data gathering and evaluation techniques to better understand the effectiveness of the experience developed in our Full-body Interaction Lab (FuBIntLab) called Lands of Fog. This is a large-scale MR, full-body interaction environment, which allows two children to play face-to-face and explore the physical and virtual worlds simultaneously. Specifically, we developed an experimental setup for comparing Lands of Fog with a control condition based on LEGO construction toys, which includes: recording psychophysiological measures synchronized with other data sources such as observed overt behaviors and system logs of game events. In order to capture accurate psychophysiological data, we developed a wearable that is child-friendly and robust to movement artifacts in the context of ambulatory full-body interaction. In order to integrate observed overt behaviors with other data sources, we designed and developed a novel video coding protocol and an adapted coding grid conceived for Social Interaction Behaviors (SIBs) in ASC children. Using a repeated measure design, we collected data from seventy-two children (36 ASC/non-ASC dyads) from the city of Barcelona, with ages between 8-12 years old (N = 12 female, N = 60 male). Data from these trials has been organized into a public database and processed based on a semi-automatic software pipeline developed within this project. Based on this data we developed three different computational models for modelling SIBs in children with ASC during Lands of Fog sessions, compared to LEGO sessions. The results of this research support the idea that full-body interaction MR environments are capable of fostering SIBs in children with ASC with similar success as the LEGO setting, with an added advantage of being more flexible. Findings reported here shed new light on developing a tool that is mediating, guiding, and supporting the progress of the children in terms of practicing SIBs and providing structure and assistance to therapists.