Bridging the gap between handwriting recognition and knowledge management

In this paper we introduce a new layer for the task of handwriting recognition (HWR), i.e., the use of semantic information in form of Resource Description Framework (RDF) knowledge bases. In particular, two novel processing stages are proposed for the first time in literature. The first stage is the inclusion of RDF knowledge bases into the HWR process, where we make use of a person’s mental model. This process can be extended to use other ontological resource. The second stage is the transition from pure handwriting recognition to understanding the handwritten notes, i.e., the system extracts knowledge employing RDF knowledge-bases. This is also called ontology-based information extraction (OBIE). The task of our recognizer therefore is not only to recognize the ASCII transcription of the handwritten document, but also to identify the semantic concepts which appear in the text. For both novel approaches we performed a set of experiments on various data. First, the recognition rate of the HWR system is increased on several documents. Second, the performance of information extraction is also remarkable. By using the k-best word recognition alternatives in form of a lattice as an input for the OBIE system, the performance reaches a level which is very close to OBIE applied on pure ASCII text.

► Ontologies are used to increase the recognition performance of handwriting recognition. ► A novel level of information extraction, i.e., OBIE for handwritten texts is proposed. ► Our system can extract instances from an ontology appearing in the text. ► Experiments show results close to those of OBIE for ASCII text.