Eyes closeness detection from still images with multi-scale histograms of principal oriented gradients

• An appearance-based architecture for unconstrained eye closeness detection.
• Two novel extensions of HOG descriptor for eye closeness recognition.
• Fusing complementary descriptors for enriched description of eyes with changes.
• Two new publicly available datasets for eye state evaluation.

In this paper, we present a novel approach to deal with the problem of detecting whether the eyes in a given still face image are closed, which has wide potential applications in human–computer interface design, facial expression recognition, driver fatigue detection, and so on. The approach combines the strength of multiple feature sets to characterize the rich information of eye patches (concerning both local/global shapes and local textures) and to construct the eye state model. To further improve the model׳s robustness against image noise and scale changes, we propose a new feature descriptor named Multi-scale Histograms of Principal Oriented Gradients (MultiHPOG). The resulting eye closeness detector handles a much wider range of eye appearance caused by expression, lighting, individual identity, and image noise than prior ones. We test our method on real-world eye datasets including the ZJU dataset and a new Closed Eyes in the Wild (CEW) dataset with promising results. In addition, several crucial design considerations that may have significant influence on the performance of a practical eye closeness detection system, including geometric normalization, feature extraction, and classification strategies, are also studied experimentally in this work.