Regression with re-labeling for noisy data

Active learning, which focuses on building an accurate prediction model with a reduced cost by actively querying which instances should be labeled for training, has been successfully employed in several real-world applications involving expensive labeling costs. Although most existing active learning strategies have focused on labeling unlabeled instances, it has been shown that improving the quality of previously annotated labels is also important when the annotator produces noisy labels. In this study, we propose a novel active learning framework for regression, which is effective for the scenarios with noisy annotators, by providing a new sampling strategy named exploration-refinement (ER) sampling. The ER sampling performs two main steps: exploration and refinement. The exploration step involves finding unlabeled instances to be labeled, and the refinement step seeks to improve the accuracy of already labeled instances. The experimental results on several benchmark datasets demonstrate the effectiveness of the ER sampling with statistical significance.