Review of objective video quality metrics and performance comparison using different databases

The use of video-based applications has increased in recent years owing to the development of video technology as well as the widespread use of the Internet. Thus the evaluation of perceptual video quality has become very important and numerous video quality assessment (VQA) metrics have been developed over the past years. In this paper, we give a classification and a short review of objective VQA metrics, with a focus on the full reference metrics. With the aim of conducting a reliable test of the VQA metrics performances, we made two databases, each of them including 90 distorted video sequences. We carried out a subjective quality evaluation on these databases and the data were made available to the research community. Furthermore, we compared the performance of nine different, freely available, objective VQA metrics by using three different databases in different resolutions: LIVE Video Quality Database (768×432 resolution) and our two newly created databases for progressively scanned videos, i.e. ETFOS CIF Video Quality (ECVQ) database and ETFOS VGA Video Quality (EVVQ) database. Five different distortion types were used and the total number of 330 video sequences was evaluated. A comparison of metrics was done with respect to accuracy, monotonicity, stability, as well as complexity vs. accuracy criteria. The results show that the resolution, the content of the sequence and the distortion type have a significant influence on the performances of VQA metrics. Metrics that generally achieve a high correlation with subjective results for all databases and all distortion types are MOtion-based Video Integrity Evaluation (MOVIE) and Foveated Mean Squared Error (FMSE), but MOVIE has significantly higher complexity than FMSE. An exception is the distortion caused by IP transmission for which none of the analyzed metrics has shown satisfying accuracy and stability.

► We made two new free available video databases with subjective quality scores.
► We performed the comparison for nine objective quality metrics on three different databases.
► We compared the metrics’ complexity calculation by using a new quantitative measure.
► The metrics that achieve a high correlation on all databases are MOVIE and FMSE.