Segmentation of cell nuclei in fluorescence microscopy images: An integrated framework using level set segmentation and touching-cell splitting

•Cell segmentation using level sets based segmentation algorithm and a touching-cell splitting method.•Experimental evaluation using fluorescence microscopy images from different datasets with different cell types.•The proposed approach can cope with touching cell nuclei and images with intensity inhomogeneity.

Accurate segmentation of cells in fluorescence microscopy images plays a key role in high-throughput applications such as quantification of protein expression and the study of cell function. In this paper, an integrated framework consisting of a new level sets based segmentation algorithm and a touching-cell splitting method is proposed. For cell nuclei segmentation, a new region-based active contour model in a variational level set formulation is developed where our new level set energy functional minimizes the Bayesian classification risk. For touching-cell splitting, the touching cells are first distinguished from non-touching cells, and then a strategy based on the splitting area identification is proposed to obtain splitting point-pairs. To form the appropriate splitting line, the image properties from different information channels are used to define the surface manifold of the image patch around the selected splitting point-pairs and geodesic distance is used to measure the length of the shortest path on the manifold connecting the two splitting points. The performance of the proposed framework is evaluated using a large number of fluorescence microscopy images from four datasets with different cell types. A quantitative comparison is also performed with several existing segmentation approaches.