Evaluation methodology and metrics employed to assess the TRANSTAC two-way, speech-to-speech translation systems

One of the most difficult challenges that military personnel face when operating in foreign countries is clear and successful communication with the local population. To address this issue, the Defense Advanced Research Projects Agency (DARPA) is funding academic institutions and industrial organizations through the Spoken Language Communication and Translation System for Tactical Use (TRANSTAC) program to develop practical machine translation systems. The goal of the TRANSTAC program is to demonstrate capabilities to rapidly develop and field free-form, two-way, speech-to-speech translation systems that enable speakers of different languages to communicate with one another in real-world tactical situations without an interpreter. Evaluations of these technologies are a significant part of the program and DARPA has asked the National Institute of Standards and Technology (NIST) to lead this effort. This article presents the experimental design of the TRANSTAC evaluations and the metrics, both quantitative and qualitative, that were used to comprehensively assess the systems’ performance.

► The DARPA TRANSTAC program developed bidirectional speech-to-speech MT systems.
► Intended users are US Soldiers or Marines in tactical situations, including on foot.
► Communicate with local civilian populations.
► We present the evaluation methodology and metrics employed for usability and utility.
► Performance of automatic speech recognition, machine translation, and text-to-speech.