Joint reranking of parsing and word recognition with automatic segmentation

Evaluation and optimization of automatic speech recognition (ASR) and parsing systems are often done separately. In the context of spoken language processing, however, these problems may be explored jointly via a reranking architecture. In this work, the effects of reranking for word error rate (WER) or reranking for the Sparseval parse-quality measure are examined in conversational speech recognition, while considering the impact of automatic segmentation. Under a WER criterion, the results indicate that the parse language model alone provides little benefit over a large n-gram model, but adding non-local syntactic features leads to improved performance. Under a Sparseval criterion, it is shown that including alternative word-sequence hypotheses has a much greater impact on parse accuracy than including alternate parse hypotheses. In both cases, the biggest performance improvements are obtained with high quality sentence segmentations. Qualitative analyses show that parse features help recover pronouns and improve recognition of main verbs.

► A new framework for joint reranking of parsing and word recognition incorporating automatic sentence segmentation.
► Confirms prior findings: segmentation quality has impact on parsing (Sparseval) quality; parsing language model has word error rate (WER) impact.
► Non-local syntactic features in reranking yield WER improvements vs. parsing LM alone; benefits larger when optimizing for WER (vs. parsing).
► More alternate word-hypotheses in reranking have a larger impact on parse accuracy than more parse hypotheses, and.
► Significant gains in parse accuracy are obtained through joint optimization.