Train longer, generalize better - Big batch training
This is a code repository used to generate the results appearing in "Train longer, generalize better: closing the generalization gap in large batch training of neural networks" By Elad Hoffer, Itay Hubara and Daniel Soudry.
It is based off convNet.pytorch with some helpful options such as:
- Training on several datasets
- Complete logging of trained experiment
- Graph visualization of the training/validation loss and accuracy
- Definition of preprocessing and optimization regime for each model
Dependencies
- pytorch
- torchvision to load the datasets, perform image transforms
- pandas for logging to csv
- bokeh for training visualization
Data
- Configure your dataset path at data.py.
- To get the ILSVRC data, you should register on their site for access: http://www.image-net.org/
Experiment examples
python main_normal.py --dataset cifar10 --model resnet --save cifar10_resnet44_bs2048_lr_fix --epochs 100 --b 2048 --lr_bb_fix;
python main_normal.py --dataset cifar10 --model resnet --save cifar10_resnet44_bs2048_regime_adaptation --epochs 100 --b 2048 --lr_bb_fix --regime_bb_fix;
python main_gbn.py --dataset cifar10 --model resnet --save cifar10_resnet44_bs2048_ghost_bn256 --epochs 100 --b 2048 --lr_bb_fix --mini-batch-size 256;
python main_normal.py --dataset cifar100 --model resnet --save cifar100_wresnet16_4_bs1024_regime_adaptation --epochs 100 --b 1024 --lr_bb_fix --regime_bb_fix;
python main_gbn.py --model mnist_f1 --dataset mnist --save mnist_baseline_bs4096_gbn --epochs 50 --b 4096 --lr_bb_fix --no-regime_bb_fix --mini-batch-size 128;
- See run_experiments.sh for more examples
Model configuration
Network model is defined by writing a .py file in models folder, and selecting it using the model flag. Model function must be registered in models/__init__.py The model function must return a trainable network. It can also specify additional training options such optimization regime (either a dictionary or a function), and input transform modifications.
e.g for a model definition:
class Model(nn.Module):
def __init__(self, num_classes=1000):
super(Model, self).__init__()
self.model = nn.Sequential(...)
self.regime = {
0: {'optimizer': 'SGD', 'lr': 1e-2,
'weight_decay': 5e-4, 'momentum': 0.9},
15: {'lr': 1e-3, 'weight_decay': 0}
}
self.input_transform = {
'train': transforms.Compose([...]),
'eval': transforms.Compose([...])
}
def forward(self, inputs):
return self.model(inputs)
def model(**kwargs):
return Model()
2 Oct 5, 2022
4 Feb 13, 2022
181 Dec 28, 2022
213 Dec 27, 2022
809 Dec 16, 2022
54 Nov 21, 2022
4 Dec 14, 2021
![A repository that shares tuning results of trained models generated by TensorFlow / Keras. Post-training quantization (Weight Quantization, Integer Quantization, Full Integer Quantization, Float16 Quantization), Quantization-aware training. TensorFlow Lite. OpenVINO. CoreML. TensorFlow.js. TF-TRT. MediaPipe. ONNX. [.tflite,.h5,.pb,saved_model,tfjs,tftrt,mlmodel,.xml/.bin, .onnx]](https://user-images.githubusercontent.com/33194443/104581604-2592cb00-56a2-11eb-9610-5eaa0afb6e1f.png)
2.4k Jan 5, 2023
11 Dec 17, 2022
20 Oct 17, 2022
97 Jan 3, 2023
43 Dec 30, 2022
197 Nov 29, 2022
17 Oct 23, 2021
4 Jun 17, 2022
1 Jun 2, 2022
3 Jan 3, 2023
2 Oct 5, 2022