Semi-supervised methods for exploring the acoustics of simple productive feedback

This paper proposes methods for exploring acoustic correlates to feedback functions. A sub-language of Swedish, simple productive feedback, is introduced to facilitate investigations of the functional contributions of base tokens, phonological operations and prosody. The function of feedback is to convey the listeners’ attention, understanding and affective states. In order to handle the large number of possible affective states, the current study starts by performing a listening experiment where humans annotated the functional similarity of feedback tokens with different prosodic realizations. By selecting a set of stimuli that had different prosodic distances from a reference token, it was possible to compute a generalised functional distance measure. The resulting generalised functional distance measure showed to be correlated to prosodic distance but the correlations varied as a function of base tokens and phonological operations. In a subsequent listening test, a small representative sample of feedback tokens were rated for understanding, agreement, interest, surprise and certainty. These ratings were found to explain a significant proportion of the generalised functional distance. By combining the acoustic analysis with an explorative visualisation of the prosody, we have established a map between human perception of similarity between feedback tokens, their measured distance in acoustic space, and the link to the perception of the function of feedback tokens with varying realisations.

► Semi-automatic annotation using a dual-channel decoder gave a recall rate of 95%.
► The generalised functional distance correlates with prosodic distance: r=0.46r=0.46.
► The specified functions correlates to the generalised functional distance: r=0.54r=0.54.
► Functions can be predicted from prosodic cues using regression 0.35► ANOVA yielded an effect of “aha/mhm” for surprise and interest (p<0.05p<0.05).