Automatic classification of personal video recordings based on audiovisual features

•A system for automatic classification of personal videos is presented.•Personal video recordings are classified into 24 categories frame by frame.•Features derived from both audio and image data are used to classify.•The system learns the most appropriate parameters and classifiers for the task.

The aim of the present work is to design a system for automatic classification of personal video recordings based on simple audiovisual features that can be easily implemented in different devices. Specifically, the main objective is to classify frame by frame personal video recordings into 24 semantically meaningful categories. Such categories include information about the environment like indoor or outdoor, the presence or absence of people and their activity, ranging from sports to partying. In order to achieve a robust classification, features derived from both audio and image data will be used and combined with state of the art classifiers such as Gaussian Mixture Models or Support Vector Machines. In the process, several combination schemes of features and classifiers are defined and evaluated over a real data set of personal video recordings. The system learns which parameters and classifiers are most appropriate for this task.The experiments show that the approach using specific classifiers for audio features (Mel-Frequency Cepstral Coefficients (MFCCs)) and image features (color, edge histograms), and using a meta-classification combination schema attains significant performance. The best performance obtained over the different approaches evaluated gave a promising f-measuref-measure larger than 57% in average for all the categories and larger than 73% over diverse categories.