Towards Plausible Differentially Private ADMM Based Distributed Machine Learning

by   Jiahao Ding, et al.

The Alternating Direction Method of Multipliers (ADMM) and its distributed version have been widely used in machine learning. In the iterations of ADMM, model updates using local private data and model exchanges among agents impose critical privacy concerns. Despite some pioneering works to relieve such concerns, differentially private ADMM still confronts many research challenges. For example, the guarantee of differential privacy (DP) relies on the premise that the optimality of each local problem can be perfectly attained in each ADMM iteration, which may never happen in practice. The model trained by DP ADMM may have low prediction accuracy. In this paper, we address these concerns by proposing a novel (Improved) Plausible differentially Private ADMM algorithm, called PP-ADMM and IPP-ADMM. In PP-ADMM, each agent approximately solves a perturbed optimization problem that is formulated from its local private data in an iteration, and then perturbs the approximate solution with Gaussian noise to provide the DP guarantee. To further improve the model accuracy and convergence, an improved version IPP-ADMM adopts sparse vector technique (SVT) to determine if an agent should update its neighbors with the current perturbed solution. The agent calculates the difference of the current solution from that in the last iteration, and if the difference is larger than a threshold, it passes the solution to neighbors; or otherwise the solution will be discarded. Moreover, we propose to track the total privacy loss under the zero-concentrated DP (zCDP) and provide a generalization performance analysis. Experiments on real-world datasets demonstrate that under the same privacy guarantee, the proposed algorithms are superior to the state of the art in terms of model accuracy and convergence rate.


Differentially Private ADMM Algorithms for Machine Learning

In this paper, we study efficient differentially private alternating dir...

Optimal Differentially Private ADMM for Distributed Machine Learning

Due to massive amounts of data distributed across multiple locations, di...

DP-ADMM: ADMM-based Distributed Learning with Differential Privacy

Distributed machine learning is making great changes in a wide variety o...

Zero-Concentrated Private Distributed Learning for Nonsmooth Objective Functions

This paper develops a fully distributed differentially-private learning ...

From Noisy Fixed-Point Iterations to Private ADMM for Centralized and Federated Learning

We study differentially private (DP) machine learning algorithms as inst...

Learning Privately over Distributed Features: An ADMM Sharing Approach

Distributed machine learning has been widely studied in order to handle ...

Privacy-preserving Distributed Machine Learning via Local Randomization and ADMM Perturbation

With the proliferation of training data, distributed machine learning (D...

Please sign up or login with your details

Forgot password? Click here to reset