Deepo is a series of Docker images that
- allows you to quickly set up your deep learning research environment
- supports almost all commonly used deep learning frameworks
- supports GPU acceleration (CUDA and cuDNN included), also works in CPU-only mode
- works on Linux (CPU version/GPU version), Windows (CPU version) and OS X (CPU version)
and their Dockerfile generator that
- allows you to customize your own environment with Lego-like modules
- define your environment in a single command line,
- then deepo will generate Dockerfiles with best practices
- and do all the configuration for you
- automatically resolves the dependencies for you
- deepo knows which combos (CUDA/cuDNN/Python/PyTorch/Tensorflow, ..., tons of dependancies) are compatible
- and will pick the right versions for you
Table of contents
Quick Start
GPU Version
Installation
Step 1. Install Docker and nvidia-docker.
Docker Hub
Step 2. Obtain the all-in-one image fromdocker pull ufoym/deepo
For users in China who may suffer from slow speeds when pulling the image from the public Docker registry, you can pull deepo
images from the China registry mirror by specifying the full path, including the registry, in your docker pull command, for example:
docker pull registry.docker-cn.com/ufoym/deepo
Usage
Now you can try this command:
docker run --gpus all --rm ufoym/deepo nvidia-smi
This should work and enables Deepo to use the GPU from inside a docker container. If this does not work, search the issues section on the nvidia-docker GitHub -- many solutions are already documented. To get an interactive shell to a container that will not be automatically deleted after you exit do
docker run --gpus all -it ufoym/deepo bash
If you want to share your data and configurations between the host (your machine or VM) and the container in which you are using Deepo, use the -v option, e.g.
docker run --gpus all -it -v /host/data:/data -v /host/config:/config ufoym/deepo bash
This will make /host/data
from the host visible as /data
in the container, and /host/config
as /config
. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host
or --shm-size
command line options to docker run
.
docker run --gpus all -it --ipc=host ufoym/deepo bash
CPU Version
Installation
Step 1. Install Docker.
Docker Hub
Step 2. Obtain the all-in-one image fromdocker pull ufoym/deepo:cpu
Usage
Now you can try this command:
docker run -it ufoym/deepo:cpu bash
If you want to share your data and configurations between the host (your machine or VM) and the container in which you are using Deepo, use the -v option, e.g.
docker run -it -v /host/data:/data -v /host/config:/config ufoym/deepo:cpu bash
This will make /host/data
from the host visible as /data
in the container, and /host/config
as /config
. Such isolation reduces the chances of your containerized experiments overwriting or using wrong data.
Please note that some frameworks (e.g. PyTorch) use shared memory to share data between processes, so if multiprocessing is used the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host
or --shm-size
command line options to docker run
.
docker run -it --ipc=host ufoym/deepo:cpu bash
You are now ready to begin your journey.
$ python
>>> import tensorflow
>>> import sonnet
>>> import torch
>>> import keras
>>> import mxnet
>>> import cntk
>>> import chainer
>>> import theano
>>> import lasagne
>>> import caffe
>>> import paddle
$ caffe --version
caffe version 1.0.0
$ darknet
usage: darknet <function>
Customization
Note that docker pull ufoym/deepo
mentioned in Quick Start will give you a standard image containing all available deep learning frameworks. You can customize your own environment as well.
Unhappy with all-in-one solution?
If you prefer a specific framework rather than an all-in-one image, just append a tag with the name of the framework. Take tensorflow for example:
docker pull ufoym/deepo:tensorflow
Jupyter support
Step 1. pull the all-in-one image
docker pull ufoym/deepo
Step 2. run the image
docker run --gpus all -it -p 8888:8888 -v /home/u:/root --ipc=host ufoym/deepo jupyter lab --no-browser --ip=0.0.0.0 --allow-root --LabApp.allow_origin='*' --LabApp.root_dir='/root'
Build your own customized image with Lego-like modules
Step 1. prepare generator
git clone https://github.com/ufoym/deepo.git
cd deepo/generator
Step 2. generate your customized Dockerfile
For example, if you like pytorch
and lasagne
, then
python generate.py Dockerfile pytorch lasagne
or with CUDA 11.1 and CUDNN 8
python generate.py Dockerfile pytorch lasagne --cuda-ver 11.1 --cudnn-ver 8
This should generate a Dockerfile that contains everything for building pytorch
and lasagne
. Note that the generator can handle automatic dependency processing and topologically sort the lists. So you don't need to worry about missing dependencies and the list order.
You can also specify the version of Python:
python generate.py Dockerfile pytorch lasagne python==3.6
Step 3. build your Dockerfile
docker build -t my/deepo .
This may take several minutes as it compiles a few libraries from scratch.
Comparison to alternatives
. | modern-deep-learning | dl-docker | jupyter-deeplearning | Deepo |
---|---|---|---|---|
ubuntu | 16.04 | 14.04 | 14.04 | 18.04 |
cuda | X | 8.0 | 6.5-8.0 | 8.0-10.2/None |
cudnn | X | v5 | v2-5 | v7 |
onnx | X | X | X | O |
theano | X | O | O | O |
tensorflow | O | O | O | O |
sonnet | X | X | X | O |
pytorch | X | X | X | O |
keras | O | O | O | O |
lasagne | X | O | O | O |
mxnet | X | X | X | O |
cntk | X | X | X | O |
chainer | X | X | X | O |
caffe | O | O | O | O |
caffe2 | X | X | X | O |
torch | X | O | O | O |
darknet | X | X | X | O |
paddlepaddle | X | X | X | O |
Tags
Available Tags
. | CUDA 11.3 / Python 3.6 | CPU-only / Python 3.6 |
---|---|---|
all-in-one | latest all all-py36 py36-cu113 all-py36-cu113 |
all-py36-cpu all-cpu py36-cpu cpu |
Theano | theano-py36-cu113 theano-py36 theano |
theano-py36-cpu theano-cpu |
TensorFlow | tensorflow-py36-cu113 tensorflow-py36 tensorflow |
tensorflow-py36-cpu tensorflow-cpu |
Sonnet | sonnet-py36-cu113 sonnet-py36 sonnet |
sonnet-py36-cpu sonnet-cpu |
PyTorch / Caffe2 | pytorch-py36-cu113 pytorch-py36 pytorch |
pytorch-py36-cpu pytorch-cpu |
Keras | keras-py36-cu113 keras-py36 keras |
keras-py36-cpu keras-cpu |
Lasagne | lasagne-py36-cu113 lasagne-py36 lasagne |
lasagne-py36-cpu lasagne-cpu |
MXNet | mxnet-py36-cu113 mxnet-py36 mxnet |
mxnet-py36-cpu mxnet-cpu |
CNTK | cntk-py36-cu113 cntk-py36 cntk |
cntk-py36-cpu cntk-cpu |
Chainer | chainer-py36-cu113 chainer-py36 chainer |
chainer-py36-cpu chainer-cpu |
Caffe | caffe-py36-cu113 caffe-py36 caffe |
caffe-py36-cpu caffe-cpu |
Torch | torch-cu113 torch |
torch-cpu |
Darknet | darknet-cu113 darknet |
darknet-cpu |
paddlepaddle | paddle-cu113 paddle |
paddle-cpu |
Deprecated Tags
. | CUDA 11.1 / Python 3.6 | CUDA 10.1 / Python 3.6 | CUDA 10.0 / Python 3.6 | CUDA 9.0 / Python 3.6 | CUDA 9.0 / Python 2.7 | CPU-only / Python 3.6 | CPU-only / Python 2.7 |
---|---|---|---|---|---|---|---|
all-in-one | py36-cu111 all-py36-cu111 |
py36-cu101 all-py36-cu101 |
py36-cu100 all-py36-cu100 |
py36-cu90 all-py36-cu90 |
all-py27-cu90 all-py27 py27-cu90 |
all-py27-cpu py27-cpu |
|
all-in-one with jupyter | all-jupyter-py36-cu90 |
all-py27-jupyter py27-jupyter |
all-py27-jupyter-cpu py27-jupyter-cpu |
||||
Theano | theano-py36-cu111 |
theano-py36-cu101 |
theano-py36-cu100 |
theano-py36-cu90 |
theano-py27-cu90 theano-py27 |
theano-py27-cpu |
|
TensorFlow | tensorflow-py36-cu111 |
tensorflow-py36-cu101 |
tensorflow-py36-cu100 |
tensorflow-py36-cu90 |
tensorflow-py27-cu90 tensorflow-py27 |
tensorflow-py27-cpu |
|
Sonnet | sonnet-py36-cu111 |
sonnet-py36-cu101 |
sonnet-py36-cu100 |
sonnet-py36-cu90 |
sonnet-py27-cu90 sonnet-py27 |
sonnet-py27-cpu |
|
PyTorch | pytorch-py36-cu111 |
pytorch-py36-cu101 |
pytorch-py36-cu100 |
pytorch-py36-cu90 |
pytorch-py27-cu90 pytorch-py27 |
pytorch-py27-cpu |
|
Keras | keras-py36-cu111 |
keras-py36-cu101 |
keras-py36-cu100 |
keras-py36-cu90 |
keras-py27-cu90 keras-py27 |
keras-py27-cpu |
|
Lasagne | lasagne-py36-cu111 |
lasagne-py36-cu101 |
lasagne-py36-cu100 |
lasagne-py36-cu90 |
lasagne-py27-cu90 lasagne-py27 |
lasagne-py27-cpu |
|
MXNet | mxnet-py36-cu111 |
mxnet-py36-cu101 |
mxnet-py36-cu100 |
mxnet-py36-cu90 |
mxnet-py27-cu90 mxnet-py27 |
mxnet-py27-cpu |
|
CNTK | cntk-py36-cu111 |
cntk-py36-cu101 |
cntk-py36-cu100 |
cntk-py36-cu90 |
cntk-py27-cu90 cntk-py27 |
cntk-py27-cpu |
|
Chainer | chainer-py36-cu111 |
chainer-py36-cu101 |
chainer-py36-cu100 |
chainer-py36-cu90 |
chainer-py27-cu90 chainer-py27 |
chainer-py27-cpu |
|
Caffe | caffe-py36-cu111 |
caffe-py36-cu101 |
caffe-py36-cu100 |
caffe-py36-cu90 |
caffe-py27-cu90 caffe-py27 |
caffe-py27-cpu |
|
Caffe2 | caffe2-py36-cu90 caffe2-py36 caffe2 |
caffe2-py27-cu90 caffe2-py27 |
caffe2-py36-cpu caffe2-cpu |
caffe2-py27-cpu |
|||
Torch | torch-cu111 |
torch-cu101 |
torch-cu100 |
torch-cu90 |
torch-cu90 torch |
torch-cpu |
|
Darknet | darknet-cu111 |
darknet-cu101 |
darknet-cu100 |
darknet-cu90 |
darknet-cu90 darknet |
darknet-cpu |
Citation
@misc{ming2017deepo,
author = {Ming Yang},
title = {Deepo: set up deep learning environment in a single command line.},
year = {2017},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/ufoym/deepo}}
}
Contributing
We appreciate all contributions. If you are planning to contribute back bug-fixes, please do so without any further discussion. If you plan to contribute new features, utility functions or extensions, please first open an issue and discuss the feature with us.