Designing and evaluating virtual musical instruments: facilitating conversational user interaction

This paper is concerned with the design of interactive virtual musical instruments. An interaction design strategy which uses on-screen objects that respond to user actions in physically realistic ways is described. This approach allows musicians to ‘play’ the virtual instruments using the sound of their familiar acoustic instruments. An investigation of user experience identified three modes of interaction that characterise the musicians' approach to the virtual instruments: instrumental, ornamental and conversational. When using the virtual instruments in instrumental mode, musicians prioritise detailed control; in ornamental mode, they surrender detailed control to the software and allow it to transform their sound; in conversational mode, the musicians allow the virtual instrument to ‘talk back’, helping to shape the musical direction of performance much as a human playing partner might. Finding a balance between controllability and complexity emerged as a key issue in facilitating ‘conversational’ interaction.