A novel benchmark model for intelligent annotation of spectral-domain optical coherence tomography scans using the example of cyst annotation

•A framework including data selection, task assignment and annotation combination stages for a confidence based benchmark dataset for retinal image processing is proposed.•A novel task assignment is used to remove data and reader biases.•The annotation of readers is combined based on their accuracy and performance.•The framework is used to build a confidence based benchmark dataset for cyst segmentation.•The generated benchmark can be used to reliably evaluate cyst segmentation methods.

Background and objectivesThe lack of benchmark data in computational ophthalmology contributes to the challenging task of applying disease assessment and evaluate performance of machine learning based methods on retinal spectral domain optical coherence tomography (SD-OCT) scans. Presented here is a general framework for constructing a benchmark dataset for retinal image processing tasks such as cyst, vessel, and subretinal fluid segmentation and as a result, a benchmark dataset for cyst segmentation has been developed.MethodFirst, a dataset captured by different SD-OCT vendors with different numbers of scans and pathology qualities are selected. Then a robust and intelligent method is used to evaluate performance of readers, partitioning the dataset into subsets. Subsets are then assigned to complementary readers for annotation with respect to a novel confidence based annotation protocol. Finally, reader annotations are combined based on their performance to generate final annotations.ResultThe generated benchmark dataset for cyst segmentation comprises 26 SD-OCT scans with differing cyst qualities, collected from 4 different SD-OCT vendors to cover a wide variety of data. The dataset is partitioned into three subsets which are annotated by complementary readers based on a confidence based annotation protocol. Experimental results show annotations of complementary readers are combined efficiently with respect to their performance, generating accurate annotations.ConclusionOur results facilitate the process of generating benchmark datasets. Moreover the generated benchmark data set for cyst segmentation can be used reliably to train and test machine learning based methods.