(Amplified) Banded Matrix Factorization: A unified approach to private training
share
Matrix factorization (MF) mechanisms for differential privacy (DP) have substantially improved the state-of-the-art in privacy-utility-computation tradeoffs for ML applications in a variety of scenarios, but in both the centralized and federated settings there remain instances where either MF cannot be easily applied, or other algorithms provide better tradeoffs (typically, as ϵ becomes small). In this work, we show how MF can subsume prior state-of-the-art algorithms in both federated and centralized training settings, across all privacy budgets. The key technique throughout is the construction of MF mechanisms with banded matrices. For cross-device federated learning (FL), this enables multiple-participations with a relaxed device participation schema compatible with practical FL infrastructure (as demonstrated by a production deployment). In the centralized setting, we prove that banded matrices enjoy the same privacy amplification results as for the ubiquitous DP-SGD algorithm, but can provide strictly better performance in most scenarios – this lets us always at least match DP-SGD, and often outperform it even at ϵ≪2. Finally, b̂-banded matrices substantially reduce the memory and time complexity of per-step noise generation from 𝒪(n), n the total number of iterations, to a constant 𝒪(b̂), compared to general MF mechanisms.
READ FULL TEXT