Private classification with limited labeled data

Differentially private Support Vector Machines (SVMs) have been extensively studied in recent years. Most design mechanisms are focused on perturbing the solution to a decent convex optimization problem under the theory of Empirical Risk Minimization (ERM). To preserve the accuracy, a large number of labeled data is needed for training the model. However, in most cases, the labeled data is limited. Constructing private SVMs in such cases often suffers from low accuracy. The situation gets worse if the given privacy budget is small. In this paper, we make use of Transductive Support Vector Machines (TSVMs) to learn from the unlabeled data. Through minimizing the overall loss on both labeled and unlabeled data, we generate a label assignment pool. Each label assignment in the pool is first evaluated as an output candidate, then selected with uncertainty for privacy consideration. The proposed algorithm provides high classification accuracy, when the labeled data is limited and when the privacy budget is small, under differential privacy. Extensive experiments show the effectiveness of the proposed algorithm on both real datasets and synthetic datasets.