Detecting and tracking people in real time with RGB-D camera

•A PEI representation is proposed to alleviate overlapping in original image domain while preserving information of all pixels.•A human plausible candidates locating technique is proposed to quickly reduce search space of the detector.•Two novel features are proposed to characterize human shape and appearance in 3D space.•A single-pass, progressive refinement framework enables the system to achieve high accuracy at real time.

We propose a novel approach to automatic detection and tracking of people taking different poses in cluttered and dynamic environments using a single RGB-D camera. The original RGB-D pixels are transformed to a novel point ensemble image (PEI), and we demonstrate that human detection and tracking in 3D space can be performed very effectively with this new representation. The detector in the first phase quickly locates human physiquewise plausible candidates, which are then further carefully filtered in a supervised learning and classification second phase. Joint statistics of color and height are computed for data association to generate final 3D motion trajectories of tracked individuals. Qualitative and quantitative experimental results obtained on the publicly available office dataset, mobile camera dataset and the real-world clothing store dataset we created show very promising results.