Article ID | Journal | Published Year | Pages | File Type |
---|---|---|---|---|
457921 | Digital Investigation | 2012 | 10 Pages |
Abstract
This paper presents a trainable open-source utility to extract text from arbitrary data files and disk images which uses language models to automatically detect character encodings prior to extracting strings and for automatic language identification and filtering of non-textual strings after extraction. With a test set containing 923 languages, consisting of strings of at most 65 characters, an overall language identification error rate of less than 0.4% is achieved. False-alarm rates on random data are 0.34% when filtering thresholds are set for high recall and 0.012% when set for high precision, with corresponding miss rates of 0.002% and 0.009% in running text.
Related Topics
Physical Sciences and Engineering
Computer Science
Computer Networks and Communications
Authors
Ralf D. Brown,