Training machine learning models without centralizing sensitive data is appealing. However, federated learning introduces a new risk: trained models might leak the data they're supposed to protect. Recent insights from NIST and the UK's Responsible Technology Adoption Unit show how differential privacy alters the privacy-utility equation, impacting your deployment choices.
What Changed
Differential privacy is now a practical tool that prevents models from memorizing and exposing training data. Research by Carlini et al. showed that sensitive data, like social security numbers, could be extracted from trained language models. Training with differential privacy effectively blocked this attack. The UK-US PETs Prize Challenges confirmed real-world implementations, with teams deploying differentially private federated learning that maintained accuracy while safeguarding individual records.
This shifts the focus from "should we add privacy protections?" to "how do we implement differential privacy without compromising model performance?"
Key Findings
1. Noise injection timing is crucial
In centralized training, noise is added once on the server. In federated learning with horizontally partitioned data, each participant adds noise to their model update before aggregation. This aggregated noise protects the global model, even if the aggregator is untrustworthy.
For vertically partitioned data, timing is more critical. Noise can't be added before entity alignment, as it disrupts attribute matching. It should be added post-alignment, either by a trusted participant or using homomorphic encryption or multiparty computation.
2. Model architecture affects privacy-utility balance
Linear regression, logistic regression, and decision trees adapt well to differential privacy. The PPMLHuskies and Scarlet Pets teams achieved high accuracy with these models in the PETs Prize Challenges.
Neural networks and deep learning models face challenges. Larger models need more noise for privacy, which can reduce accuracy. Consider this in your risk assessment: if deep learning is essential, your privacy budget will impact performance more.
3. Pre-training on public data shifts the balance
Models pre-trained on public data and fine-tuned with differential privacy can achieve accuracy similar to models without privacy protections, as shown by Li et al. This is significant for domains with public data, like language models and image recognition. While public pre-training data isn't protected, ensure its use respects privacy and intellectual property rights. Differential privacy can then be applied during fine-tuning on sensitive data without losing utility.
4. Real-world deployment is feasible
The Scarlet Pets team used a variant of the FedAvg approach with differential privacy, adding noise to each participant's update before aggregation. This wasn't just a lab test; it was a competition proving real-world viability.
What This Means for Your Team
When evaluating federated learning for regulated data, consider three decisions: where to add noise, how much noise to add, and whether your model architecture can handle that noise without losing utility.
Your threat model guides the first decision. If you trust your aggregator, you have flexibility in noise addition. If not, each participant must add noise before sharing updates. For vertically partitioned data, use a trusted party or cryptographic techniques to add noise post-alignment.
Compliance requirements influence the second decision. Differential privacy offers formal guarantees, but you need a privacy budget that meets regulatory obligations, affecting model accuracy.
Your use case drives the third decision. Linear models or decision trees manage the privacy-utility tradeoff well. For deep learning, consider pre-training on public data and fine-tuning with differential privacy. If not feasible, you'll face a tougher tradeoff between privacy and performance.
Action Items by Priority
Immediate (this quarter):
Map your federated learning architecture to the noise injection points discussed. For horizontally partitioned data, ensure each participant adds noise to their update. For vertically partitioned data, identify who adds noise post-alignment and how they'll do it securely.
Document your model architecture and privacy needs. For neural networks, test if pre-training on public data is viable. If not, determine the acceptable accuracy loss to meet your privacy budget.
Near-term (next two quarters):
Develop a test implementation with differential privacy. Measure the actual privacy-utility tradeoff for your data and model. General guidance won't suffice; your data's characteristics will dictate noise tolerance.
Define your privacy budget with legal and compliance teams. Differential privacy's guarantees are meaningful only if your budget aligns with regulatory needs and risk tolerance.
Ongoing:
Stay updated on differentially private deep learning research. The field evolves rapidly, especially for large language models and generative AI. Techniques impractical last year may be viable today.
Review your public data usage for pre-training models. Intellectual property and privacy considerations for public datasets are changing. What's acceptable now may not be in the future.
