Resolving hand over face occlusion

The ability to segment or track the hand is an important problem in computer vision. While various solutions have been proposed, many methods do not work against complex or cluttered backgrounds. Solving these cases is essential to solving many problems in the domain of computer vision such as, human–computer interaction (HCI), surveillance, and virtual reality (i.e., augmented desks). This paper presents a method to segment the hand over complex backgrounds, such as the face. The similar colors and texture of the hand and face make the problem particularly challenging. The method is not restricted to only segmenting hands across faces and uses no knowledge of hands. Our method is based on the underlying concept of an image force field. In this representation change is measured through how particles move through the field. Each individual image location consists of a vector value which is a nonlinear combination of the remaining pixels in the image. We introduce and develop a novel physics-based feature that is able to measure regional structure in the image thus avoiding the problem of local pixel-based analysis, which breaks down under our conditions. The regional image structure changes in the occluded region during occlusion, while elsewhere the regional structure remains relatively constant. We model the regional image structure at all image locations over time using a mixture of Gaussians (MoG) to detect the occluded region in the image. We have tested the method on a number of sequences demonstrating the versatility of the proposed approach.