Mixture models with skin and shadow probabilities for fingertip input applications

•Moving skin regions and shadows are detected.•Skin and shadow probabilities are integrated into the mixture models.•Learning rates are adjusted by the skin and shadow probabilities.•Mandarin Phonetic Symbol (MPS) combination recognition algorithm is designed.•Single fingertip virtual keyboard system is implemented.

This paper proposes an accurate moving skin region detection method for video-based human–computer interface using gestures or fingertips. Using Gaussian mixture models as groundwork, the proposed method expresses the features of skins in a probability form and incorporates them into the mixture-based framework. Moreover, to alleviate the influence of shadows, the properties of shadows are also formulated as probabilities and used for shadow detection and elimination. In addition to moving skin region detection, this paper also develops two practical fingertip input applications to demonstrate the accuracy of the proposed detection method. The two applications are Mandarin Phonetic Symbol combination recognition system and single fingertip virtual keyboard implementation. Experimental results have shown the advantages of the proposed detection method and the effectiveness of the two application implementations.