Automatic image segmentation of nuclear stained breast tissue sections using color active contour model and an improved watershed method

•A modified geometric active contour model is used for multiple contour detection of the different nuclear staining in the breast cancer tissue images.•A touching nuclei separation method based on watershed algorithm and concave vertex graph is proposed to quantify the total number of cancer nuclei in each class.•The proposed methods are tested on two datasets of breast cancer cell images containing different level of malignancy.•Results show the superiority of the proposed methods when compared with other existing classification and segmentation methods.•On the complete image database, the segmentation accuracy in term of cancer nuclei number is over than 97%, reaching an improvement of 3–4% over earlier methods.

Automatic image segmentation of immunohistologically stained breast tissue sections helps pathologists to discover the cancer disease earlier. The detection of the real number of cancer nuclei in the image is a very tedious and time consuming task. Segmentation of cancer nuclei, especially touching nuclei, presents many difficulties to separate them by traditional segmentation algorithms. This paper presents a new automatic scheme to perform both classification of breast stained nuclei and segmentation of touching nuclei in order to get the total number of cancer nuclei in each class. Firstly, a modified geometric active contour model is used for multiple contour detection of positive and negative nuclear staining in the microscopic image. Secondly, a touching nuclei method based on watershed algorithm and concave vertex graph is proposed to perform accurate quantification of the different stains. Finally, benign nuclei are identified by their morphological features and they are removed automatically from the segmented image for positive cancer nuclei assessment. The proposed classification and segmentation schemes are tested on two datasets of breast cancer cell images containing different level of malignancy. The experimental results show the superiority of the proposed methods when compared with other existing classification and segmentation methods. On the complete image database, the segmentation accuracy in term of cancer nuclei number is over than 97%, reaching an improvement of 3–4% over earlier methods.