Memorable and rich video summarization

•A shot segmentation method using perceptual hashing-based mutual-information calculation is proposed; this method is more efficient and accurate than color feature-based shot segmentation method.•Memorability score predicted by the deep network is introduced and combined with entropy value for key frame extraction. The produced summaries are semantically interesting and preserve the diversity of the videos.•Pairwise f-measure is introduced to estimate the performance of generated summaries. This measure is effective in the evaluation of a multi-participant-labeled video summary.

Video summarization can facilitate rapid browsing and efficient video indexing in many applications. A good summary should maintain the semantic interestingness and diversity of the original video. While many previous methods extracted key frames based on low-level features, this study proposes Memorability-Entropy-based video summarization. The proposed method focuses on creating semantically interesting summaries based on image memorability. Further, image entropy is introduced to maintain the diversity of the summary. In the proposed framework, perceptual hashing-based mutual information (MI) is used for shot segmentation. Then, we use a large annotated image memorability dataset to fine-tune Hybrid-AlexNet. We predict the memorability score by using the fine-tuned deep network and calculate the entropy value of the images. The frame with the maximum memorability score and entropy value in each shot is selected to constitute the video summary. Finally, our method is evaluated on a benchmark dataset, which comes with five human-created summaries. When evaluating our method, we find it generates high-quality results, comparable to human-created summaries and conventional methods.