A semi-automatic text-based semantic video annotation system for Turkish facilitating multilingual retrieval

It is commonly acknowledged that ever-increasing video archives should be conveniently indexed with the conveyed semantic information to facilitate later video retrieval. Domain-independent semantic video indexing is usually carried out through manual means which is too time-consuming and labor-intensive to be employed in practical settings. On the other hand, fully automated approaches are usually proposed for very specialized domains such as team sports videos. In this paper, we propose a generic text-based semi-automatic system for off-line semantic indexing and retrieval of news videos, since video texts such as speech transcripts stand as a plausible source of semantic information. The proposed system has a pipelined flow of execution where the sole manual intervention takes place during text extraction, yet it could execute in fully automated mode in case the associated video text is already available or a convenient text extractor is available to be incorporated into the system. At the core of the system is an information extraction component – a named entity recognizer – which extracts representative semantic information from the video texts. Based on the proposed generic system, a novel semantic annotation and retrieval system for Turkish is designed, implemented, and evaluated on two distinct news video data sets. By equipping it with the necessary components, the ultimate system is also turned into a multilingual video retrieval system and executed on a video data set in English, thereby facilitating multilingual semantic video retrieval.

► A semi-automatic approach for semantic news video indexing and retrieval is proposed.
► The system can also be executed in fully automated mode under certain settings.
► The system is also turned into an automatic multilingual video retrieval system.