The sound of smile: Auditory biofeedback of facial EMG activity

- We proposed a real-time system for smile sonification using surface EMG.
- The strength of a smile is associated with the pitch with the timbre of bubble sound.
- The sonification system and well reflects a user's facial expressions.
- The sonification system was an enjoyable experience for users.

In this paper, a real-time interactive system for smile detection and sonification using surface Electromyography (sEMG) signals is proposed. When a user smiles, a sound is played. The surface EMG signal is mapped to pitch using a conventional scale. The timbre of the sound is a synthetic sound that mimics bubbles.In a user testing of smiling tasks, 14 participants underwent the system and are required to produce smiles under three conditions, i.e., auditory feedback with sonification, visual feedback with mirror, and no feedback. The impression of the system is evaluated through questionnaires and interviews with the participants. In addition, we analyzed the total amount of muscular activity and temporal envelope patterns of the sEMG during smiling.The questionnaire and interview showed that users felt that (1) the sonification system well reflects their facial expressions, and (2) the sonification system was enjoyable. The users also expressed that the auditory feedback condition is easier to smile with, as compared to the visual feedback or no feedback conditions. However, the analysis of sEMG did not provide a quantitative difference among the three conditions, which is most likely due to the experiment design, which lacks socially engaging settings.