Skip to content
The methodOpen data
Back to the collection

Communication-Efficient Learning of Deep Networks from Decentralized Data

H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, Blaise Agüera y Arcas

2016 · arXiv

Statistics & probabilitySecurity & cryptographyPsychology
Read the paper
HN score
0
Mentions
4
Readers
4
Months
3
What’s in a score?

From the abstract

Modern mobile devices have access to a wealth of data suitable for learning models, which in turn can greatly improve the user experience on the device. For example, language models can improve speech recognition and text entry, and image models can automatically select good photos. However, this rich data is often privacy sensitive, large in quantity, or both, which may preclude logging to the data center and training there using conventional approaches. We advocate an alternative that leaves the training data distributed on the mobile devices, and learns a shared model by aggregating locally-computed updates. We term this decentralized approach Federated Learning. We present a practical method for the federated learning of deep networks based on iterative model averaging, and conduct an extensive empirical evaluation, considering five different model architectures and four datasets. These experiments demonstrate the approach is robust to the unbalanced and non-IID data distributions that are a defining characteristic of this setting. Communication costs are the principal constraint, and we show a reduction in required communication rounds by 10-100x as compared to synchronized stochastic gradient descent.

Metadata: arXiv. Abstracts retain their original authors’ rights.

In the conversation

Original HN comments, with labels from simple text rules. A label describes the whole comment and may miss context or irony. Read the discussion to make up your own mind. About the labels

4 selected excerpts · 4 total mentions

techwizrd
Neutral mention · rule
It's quite misleading, actually. GBoard uses Federated Learning [0] so that models are trained locally on the device and keywords are not sent to the central server. Both Google and iOS use Federated Learning, Federated Evaluation & Tuning, Differential Privacy, and other privacy-preserving AI techniques and/or on-device ML to avoid sending user data to the server. Both have done so publicly since 2017 and have published many papers and talks on the subject. 0: https://arxiv.org/abs/1602.05629
Original comment Thumb-Key: a privacy-conscious smart keyboard, made specifically for your thumbs
est31
Neutral mention · rule
One important corollary of this is that "privacy respecting" federated learning schemes [0] where you train a model locally with your data and only upload the deltas might leak your private data after all. [0]: https://arxiv.org/abs/1602.05629
Original comment Does GPT-2 Know Your Phone Number?
andreyk
Neutral mention · rule
The paper: https://arxiv.org/pdf/1602.05629.pdf The key algorithmic detail: it seems they have each device perform multiple batch updates to the model, and then average all the multi-batch updates. "That is, each client locally takes one step of gradient descent on the current model using its local data, and the server then takes a weighted average of the resulting models. Once the algorithm is written this way, we can add more computation to each client by iterating the local update. " They do some sensible things with model initialization to make sure weight update averaging works, and show in practice this way of doing things requires less communication and gets to the goal faster than a more naive approach. It seems like a fairly straighforward idea from the baseline SGD, so the contribution is mostly in actually doing it.
Original comment Machine learning without centralized training data
whym
Neutral mention · rule
Their papers mentioned in the article: Federated Learning: Strategies for Improving Communication Efficiency (2016) https://arxiv.org/abs/1610.05492 Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016) https://arxiv.org/abs/1610.02527 Communication-Efficient Learning of Deep Networks from Decentralized Data (2017) https://arxiv.org/abs/1602.05629 Practical Secure Aggregation for Privacy Preserving Machine Learning (2017) http://eprint.iacr.org/2017/281
Original comment Machine learning without centralized training data

Keep following the thread.

More in Statistics & probability

More from the same topics.

2018arXiv

Talent vs Luck: the role of randomness in success and failure

A. Pluchino. A. E. Biondo, A. Rapisarda

The largely dominant meritocratic paradigm of highly competitive Western cultures is rooted on the belief that success is due mainly, if not exclusively, to personal qualities such as talent, intelligence, skills, efforts or risk taking.

Psychology
16 mentions14 months
2.5HN score