Private Semi-supervised Knowledge Transfer for Deep Learning from Noisy Labels

by   Qiuchen Zhang, et al.

Deep learning models trained on large-scale data have achieved encouraging performance in many real-world tasks. Meanwhile, publishing those models trained on sensitive datasets, such as medical records, could pose serious privacy concerns. To counter these issues, one of the current state-of-the-art approaches is the Private Aggregation of Teacher Ensembles, or PATE, which achieved promising results in preserving the utility of the model while providing a strong privacy guarantee. PATE combines an ensemble of "teacher models" trained on sensitive data and transfers the knowledge to a "student" model through the noisy aggregation of teachers' votes for labeling unlabeled public data which the student model will be trained on. However, the knowledge or voted labels learned by the student are noisy due to private aggregation. Learning directly from noisy labels can significantly impact the accuracy of the student model. In this paper, we propose the PATE++ mechanism, which combines the current advanced noisy label training mechanisms with the original PATE framework to enhance its accuracy. A novel structure of Generative Adversarial Nets (GANs) is developed in order to integrate them effectively. In addition, we develop a novel noisy label detection mechanism for semi-supervised model training to further improve student model performance when training with noisy labels. We evaluate our method on Fashion-MNIST and SVHN to show the improvements on the original PATE on all measures.


page 1

page 2

page 3

page 4


Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

Some machine learning applications involve training data that is sensiti...

Scalable Private Learning with PATE

The rapid adoption of machine learning has increased concerns about the ...

Private Knowledge Transfer via Model Distillation with Generative Adversarial Networks

The deployment of deep learning applications has to address the growing ...

Self-training with noisy student model and semi-supervised loss function for dcase 2021 challenge task 4

This report proposes a polyphonic sound event detection (SED) method for...

Semi-Supervised Noisy Student Pre-training on EfficientNet Architectures for Plant Pathology Classification

In recent years, deep learning has vastly improved the identification an...

Semi-supervised music emotion recognition using noisy student training and harmonic pitch class profiles

We present Mirable's submission to the 2021 Emotions and Themes in Music...

Robustness of Accuracy Metric and its Inspirations in Learning with Noisy Labels

For multi-class classification under class-conditional label noise, we p...

Please sign up or login with your details

Forgot password? Click here to reset