Training DALL-E with volunteers from all over the Internet using hivemind and dalle-pytorch (NeurIPS 2021 demo)

learning@home

Last update: Dec 13, 2022

Related tags

Deep Learning dalle-hivemind

Overview

Training DALL-E with volunteers from all over the Internet

This repository is a part of the NeurIPS 2021 demonstration "Training Transformers Together".

In this demo, we train a model similar to OpenAI DALL-E — a Transformer "language model" that generates images from text descriptions. Training happens collaboratively — volunteers from all over the Internet contribute to the training using hardware available to them. We use LAION-400M, the world's largest openly available image-text-pair dataset with 400 million samples. Our model is based on the dalle‑pytorch implementation by Phil Wang with a few tweaks to make it communication-efficient.

See details about how to join and how it works on our website.

Comments

Make model uploading use access token from authorizer
This PR makes model uploading use the persistent access token from the main authorizer, thus:

You don't need to run transformers-cli login anymore

The model uploading token does not expire anymore

If --upload_interval is specified, the checkpoint is always uploaded at the start of an aux peer (so you're sure that the upload works fine)
opened by borzunov 0
add `self.repo.git_pull` before `self.repo.push_to_hub` in the aux peer

As we are close to the deadline, feel free to ignore this PR if it is a change that is too risky -vs- advantageous in view of the deadline tomorrow.

This PR adds a pull before a push to hub. The idea here is to be able to modify the model card during the training.

This change have been tested with huggingface-hub==0.2.1

opened by SaulLu 0
Use user access token instead of username/password

The use of login with username + password is now deprecated on the huggingface site. Instead, it is recommended to use user access token allowing applications to perform specific actions specified by the scope of permissions (read, write, or admin) granted.

This PR therefore proposes changes to use this kind of access tokens.

Corresponding issue: training-transformers-together/training-transformers-together.github.io#7

opened by SaulLu 0
[WIP] Change authorization client side

Will close: https://github.com/training-transformers-together/training-transformers-together.github.io/issues/7

Todo: still need to change password with an HF token

opened by SaulLu 0
The processed data is not the same as the data shown in the huggingface

In order to transform transform image byte to numpy array, we use np.frombuffer(encoded, np.int16).astype(np.int64), however, I find out that the processsed data is different compared to the original data shown in huggingface. On the otherhand, if we try np.frombuffer(encoded, np.uint8).astype(np.int64), then the processed data is the same. Would that influence the model performance?

opened by elricwan 0
Get the country of each session

As previously discussed, this PR comes shortly before the deadline: I leave you the choice to integrate it or not.

For your information, I tested to modify a trainer in a similar way by modifying the demo notebook of the HF trainer to check these changes. You can check it here.

opened by SaulLu 0

Owner

learning@home

GitHub

Open-AI's DALL-E for large scale training in mesh-tensorflow.

DALL-E in Mesh-Tensorflow [WIP] Open-AI's DALL-E in Mesh-Tensorflow. If this is similarly efficient to GPT-Neo, this repo should be able to train mode

432 Dec 16, 2022

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

DALL-E in Pytorch Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch. It will also contain CLIP for ranking the ge

5k Jan 4, 2023

PyTorch package for the discrete VAE used for DALL·E.

Overview [Blog] [Paper] [Model Card] [Usage] This is the official PyTorch package for the discrete VAE used for DALL·E. Installation Before running th

9.5k Jan 5, 2023

RuDOLPH: One Hyper-Modal Transformer can be creative as DALL-E and smart as CLIP

[Paper] [Хабр] [Model Card] [Colab] [Kaggle] RuDOLPH ?? ?? ☃️ One Hyper-Modal Tr

230 Dec 31, 2022

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS.

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS. It currently supports four examples for you to quickly experience the power of ONNX Runtime Web.

58 Dec 18, 2022

We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.

An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering Papers with code | Paper Nikola Zubić Pietro Lio University

213 Dec 27, 2022

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data This repository provides the implementation details for

124 Dec 27, 2022

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Neural Attention Distillation This is an implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep

84 Jan 4, 2023

Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Geometric Vector Perceptron Implementation of Geometric Vector Perceptron, a simple circuit with 3d rotation equivariance for learning over large biom

59 Nov 24, 2022

BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work

BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work. For this project, I used the sigmoid function as an activation function along with stochastic gradient descent to adjust the weights and biases.

1 Jan 22, 2022

Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

71 Dec 22, 2022

[NeurIPS 2021] Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training Code for NeurIPS 2021 paper "Better Safe Than Sorry: Preventing Delu

29 Sep 20, 2022

[NeurIPS 2021] Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data (NeurIPS 2021) This repository provides the official PyTorch implementation

155 Nov 30, 2021

Orthogonal Over-Parameterized Training

The inductive bias of a neural network is largely determined by the architecture and the training algorithm. To achieve good generalization, how to effectively train a neural network is of great importance. We propose a novel orthogonal over-parameterized training (OPT) framework that can provably minimize the hyperspherical energy which characterizes the diversity of neurons on a hypersphere. See our previous work -- MHE for an in-depth introduction.

11 Apr 18, 2022

Training DALL-E with volunteers from all over the Internet using hivemind and dalle-pytorch (NeurIPS 2021 demo)

Related tags

Overview

Training DALL-E with volunteers from all over the Internet

Comments

Make model uploading use access token from authorizer

add `self.repo.git_pull` before `self.repo.push_to_hub` in the aux peer

Use user access token instead of username/password

[WIP] Change authorization client side

The processed data is not the same as the data shown in the huggingface

Get the country of each session

Owner

learning@home

Open-AI's DALL-E for large scale training in mesh-tensorflow.

Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in Pytorch

PyTorch package for the discrete VAE used for DALL·E.

RuDOLPH: One Hyper-Modal Transformer can be creative as DALL-E and smart as CLIP

ONNX Runtime Web demo is an interactive demo portal showing real use cases running ONNX Runtime Web in VueJS.

We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

An implementation demo of the ICLR 2021 paper Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks in PyTorch.

Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

BasicNeuralNetwork - This project looks over the basic structure of a neural network and how machine learning training algorithms work

Official implementation of NeurIPS 2021 paper "One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective"

[NeurIPS 2021] Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Training

[NeurIPS 2021] Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

Orthogonal Over-Parameterized Training

Demo code for paper "Learning optical flow from still images", CVPR 2021.

Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Pytorch GUI(demo) for iVOS(interactive VOS) and GIS (Guided iVOS)

dataset for ECCV 2020 "Motion Capture from Internet Videos"

Magic tool for managing internet connection in local network by @zalexdev