Robust lip detection based on histogram of oriented gradient features and convolutional neural network under effects of light and background

Detection of the lip area is essential pre-processing for several applications, such as lip reading and visual information services. In this paper, we propose a lip detection method that finds the lip area in an image using the histogram of oriented gradient (HOG) features and convolutional neural network (CNN). We find the face area from an input image, divide the face image in half, and apply sliding window detection to the bottom half of the image. We obtain the HOG feature vector from the image that corresponds to the window, and use it as the input to a pre-trained support vector machine (SVM). HOG and SVM are used for coarse detection. If SVM determines that the image is not the lip, we reapply sliding window detection. Otherwise, the image is used as input to CNN, which is employed for fine detection and to determine whether the image is the lip. If CNN determines that the image is the lip, we apply canny edge detection to the image to obtain the mouth contour. We use MATLAB to confirm the effectiveness of our method, and can find the mouth area with over 94% accuracy and over 98% precision.