An iterative multimodal framework for the transcription of handwritten historical documents

The transcription of historical documents is one of the most interesting tasks in which Handwritten Text Recognition can be applied, due to its interest in humanities research. One alternative for transcribing the ancient manuscripts is the use of speech dictation by using Automatic Speech Recognition techniques. In the two alternatives similar models (Hidden Markov Models and n-grams) and decoding processes (Viterbi decoding) are employed, which allows a possible combination of the two modalities with little difficulties. In this work, we explore the possibility of using recognition results of one modality to restrict the decoding process of the other modality, and apply this process iteratively. Results of these multimodal iterative alternatives are significantly better than the baseline uni-modal systems and better than the non-iterative alternatives.

► We tackle the task of transcription of handwritten historical document.
► We combine handwritten text and speech recognition for obtaining transcription.
► We use an iterative process that combines the two modalities.
► The multimodal iterative process obtains a high reduction in the transcription error.