Combining shape, texture and intensity features for cell nuclei extraction in Pap smear images

In this work, we present an automated method for the detection and boundary determination of cells nuclei in conventional Pap stained cervical smear images. The detection of the candidate nuclei areas is based on a morphological image reconstruction process and the segmentation of the nuclei boundaries is accomplished with the application of the watershed transform in the morphological color gradient image, using the nuclei markers extracted in the detection step. For the elimination of false positive findings, salient features characterizing the shape, the texture and the image intensity are extracted from the candidate nuclei regions and a classification step is performed to determine the true nuclei. We have examined the performance of two unsupervised (K-means, spectral clustering) and a supervised (Support Vector Machines, SVM) classification technique, employing discriminative features which were selected with a feature selection scheme based on the minimal-Redundancy–Maximal-Relevance criterion. The proposed method was evaluated on a data set of 90 Pap smear images containing 10,248 recognized cell nuclei. Comparisons with the segmentation results of a gradient vector flow deformable (GVF) model and a region based active contour model (ACM) are performed, which indicate that the proposed method produces more accurate nuclei boundaries that are closer to the ground truth.

Research highlights
► Automated method for the detection/segmentation of cell nuclei in Pap smear images.
► Watershed segmentation is applied for the determination of the nuclei boundary.
► Feature selection and classification is employed to distinguish the true nuclei.