Hivemind: decentralized deep learning in PyTorch
Hivemind is a PyTorch library to train large neural networks across the Internet. Its intended usage is training a single Transformer model on hundreds of computers from different universities, companies, and volunteers.
Key Features
- Train neural networks of arbitrary size: parts of their layers are distributed across the participants.
- Distributed training without a master node: Distributed Hash Table allows connecting computers in a decentralized network.
- Fault-tolerant backpropagation: forward and backward passes succeed even if some nodes are unresponsive or take too long to respond.
- Decentralized parameter averaging: iteratively aggregate updates from multiple workers without the need to synchronize across the entire network.
To learn more about the ideas behind this library, see https://learning-at-home.github.io or read the NeurIPS 2020 paper.
Installation
Before installing hivemind, make sure that your environment has Python 3.7+ and PyTorch with a version at least as new as 1.6.0.
To start using this library, you can either use the pip package manager or build it from source. Since currently the release cycle is not established yet, we recommend installing hivemind from source to keep up with the latest bugfixes and improvements.
With pip
If your versions of Python and PyTorch match the requirements, you can install hivemind from pip:
pip install hivemind
From source
To install hivemind from source, simply clone the repository and install
git clone https://github.com/learning-at-home/hivemind.git
cd hivemind
pip install .
If you would like to verify that your installation is working properly, you can install with pip install -e .[dev] instead. Then, you can run the tests with pytest tests/.
Documentation
- Quickstart: install hivemind, set up a server and train experts
- Documentation & guides are available at learning-at-home.readthedocs.io
Contributing
Hivemind is currently at the active development stage, and we welcome all contributions. Everything, from bug fixes and documentation improvements to entirely new features, is equally appreciated.
If you want to contribute to hivemind but don't know where to start, take a look at the unresolved issues. Open a new issue or join our chat room in case you want to discuss new functionality or report a possible bug. Bug fixes are always welcome, but new features should be preferably discussed with maintainers beforehand.
If you want to start contributing to the source code of hivemind, please see the contributing guidelines first. To learn more about other ways to contribute, read our guide.
Citation
If you found hivemind useful for your experiments, you can cite the paper that inspired it:
@inproceedings{ryabinin2020crowdsourced,
author = {Ryabinin, Max and Gusev, Anton},
booktitle = {Advances in Neural Information Processing Systems},
editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
pages = {3659--3672},
publisher = {Curran Associates, Inc.},
title = {Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts},
url = {https://proceedings.neurips.cc/paper/2020/file/25ddc0f8c9d3e22e03d3076f98d83cb2-Paper.pdf},
volume = {33},
year = {2020}
}
The initial implementation of hivemind used for the paper is available at mryab/learning-at-home.
In the documentation, we list several related projects and acknowledgements.
![[BUG] Dead lock when 'Downloading parameters' cost Took too much time](https://avatars.githubusercontent.com/u/14099092?v=4)
![[BUG] Enable to train a bloat16-compressed model](https://avatars.githubusercontent.com/u/79724149?v=4)
6 Jun 18, 2022
121 Dec 28, 2022
1.4k Jan 5, 2023
73 Oct 17, 2022
1.6k Dec 31, 2022
11.6k Jan 2, 2023
2 Aug 23, 2022
223 Jan 3, 2023
1.4k Jan 6, 2023
1 Oct 28, 2021
2 Jan 18, 2022
2 Jan 22, 2022
1 Dec 3, 2021
92 Dec 14, 2022
366 Jan 3, 2023
152 Jan 7, 2023
5.7k Dec 30, 2022
152 Jan 2, 2023
15 Aug 12, 2022