Can avatars pass the Turing test? Intelligent agent perception in a 3D virtual environment

• The first natural language Turing Test was conducted in a 3D virtual environment.
• Seventy-eight percent of participants incorrectly judged a chatbot to be human.
• Agency judgments were due to the quality of the AI engine and psychological factors.
• Passage of the Turing Test involves a complex process of human–computer interaction.

The current study involved the first natural language, modified Turing Test in a 3D virtual environment. One hundred participants were given an avatar-guided tour of a virtual clothing store housed in the 3D world of Second Life. In half of the cases, a human research assistant controlled the avatar-guide; in the other half, the avatar-guide was a visually indistinguishable virtual agent or “bot” that employed a chat engine called Discourse, a more robust variant of Artificial Intelligence Markup Language (AIML). Both participants and the human research assistant were blind to variations in the controlling agency of the guide. The results indicated that 78% of participants in the artificial intelligence condition incorrectly judged the bot to be human, significantly exceeding the 50% rate that one would expect by chance alone that is used as the criterion for passage of a modified Turing Test. An analysis of participants׳ decision-making criteria revealed that agency judgments were impacted by both the quality of the AI engine and a number of psychological and contextual factors, including the naivety of participants regarding the possible presence of an intelligent agent, the duration of the trial period, the specificity and structure of the test situation, and the anthropomorphic form and movements of the agent. Thus, passage of the Turing Test is best viewed not as the sole product of advances in artificial intelligence or the operation of psychological and contextual variables, but as a complex process of human–computer interaction.