MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks
This is the code for the paper:
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, Li Fei-Fei
Presented at ICML 2018
Please note that this is not an officially supported Google product.
If you find this code useful in your research then please cite
@inproceedings{jiang2018mentornet,
title={MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels},
author={Jiang, Lu and Zhou, Zhengyuan and Leung, Thomas and Li, Li-Jia and Fei-Fei, Li},
booktitle={ICML},
year={2018}
}
Introduction
We are interested in training a deep network using curriculum learning (Bengio et al., 2009), i.e. learning examples with focus. Each curriculum is implemented as a network (called MentorNet).
- During training, MentorNet supervises the training of the base network (called StudentNet).
- At the test time, StudentNet makes prediction alone without MentorNet.
Setups
All code was developed and tested on Nvidia V100/P100 (16GB) the following environment.
- Ubuntu 18.04
- Python 2.7.15
- TensorFlow 1.8.0
- numpy 1.13.3
- imageio 2.3.0
Download Cloud SDK to get data and models. Next we need to download the dataset and pre-trained MentorNet models. Put them into the same directory as the code directory.
gsutil -m cp -r gs://mentornet_project/data .
gsutil -m cp -r gs://mentornet_project/mentornet_models .
Alternatively, you may download the zip files: data and models.
Running MentorNet on CIFAR
export PYTHONPATH="$PYTHONPATH:$PWD/code/"
python code/cifar_train_mentornet.py \
--dataset_name=cifar10 \
--trained_mentornet_dir=mentornet_models/models/mentornet_pd1_g_1/mentornet_pd \
--loss_p_precentile=0.75 \
--nofixed_epoch_after_burn_in \
--burn_in_epoch=0 \
--example_dropout_rates="0.5,17,0.05,83" \
--data_dir=data/cifar10/0.2 \
--train_log_dir=cifar_models/cifar10/resnet/0.2/mentornet_pd1_g_1/train \
--studentnet=resnet101 \
--max_number_of_steps=39000
A full list of commands can be found in this file. The training script has a number of command-line flags that you can use to configure the model architecture, hyperparameters, and input / output settings:
--trained_mentornet_dir: Directory where to find the trained MentorNet model, created bymentornet_learning/train.py.--loss_p_percentile: p-percentile used to compute the loss moving average. Default is0.7.--burn_in_epoch: Number of first epochs to perform burn-in. In the burn-in period, every sample has a fixed 1.0 weight. Default is0.--fixed_epoch_after_burn_in: Whether to use the fixed epoch as the MentorNet input feature after the burn-in period. Set True for MentorNet DD. Default isFalse.--loss_moving_average_decay: Decay factor used in moving average. Default is0.5.--example_dropout_rates: Comma-separated list indicating the example drop-out rate for the total of 100 epochs. The format is [dropout rate, epoch_num]+, the piecewise drop-out rate from boundaries and values. The sum of epoch_num is 100. Drop-out means the probability of setting sample weights to zeros proposed (Liang et al., 2016). Default is0.5, 17, 0.05, 78, 1.0, 5.
To evaluate a model, run the evaluation job in parallel with the training job (on a different GPU).
python cifar/cifar_eval.py \
--dataset_name=cifar10 \
--data_dir=cifar/data/cifar10/val/ \
--checkpoint_dir=cifar_models/cifar10/resnet/0.2/mentornet_pd1_g_1/train \
--eval_dir=cifar_models/cifar10/resnet/0.2/mentornet_pd1_g_1//eval_val \
--studentnet=resnet101 \
--device_id=1
A complete list of commands of running experiments can be found at commands/train_studentnet_resnet.sh and commands/train_studentnet_inception.sh.
MentorNet Framework
MentorNet is a general framework for curriculum learning, where various curriculums can be learned by the same MentorNet structure of different parameters.
It is flexible as we can switch curriculums by attaching different MentorNets without modifying the pipeline.
We train a few MentorNets listed below. We can think of a MentorNet as a hyper-parameter and will be tuned for different problems.
| Curriculum | Visualization | Intuition | Model Name |
|---|---|---|---|
| No curriculum |  |
Assign uniform weight to every sample uniform. | baseline_mentornet |
| Self-paced (Kuma et al. 2010) |
 |
Favor samples of smaller loss. | self_paced_mentornet |
| SPCL linear (Jiang et al. 2015) |
 |
Discount the weight by loss linearly. | spcl_linear_mentornet |
| Hard example mining (Felzenszwalb et al., 2008) |
 |
Favor samples of greater loss. | hard_example_mining_mentornet |
| Focal loss (Lin et al., 2017) |
 |
Increase the weight by loss by the exponential CDF. | focal_loss_mentornet |
| Predefined Mixture |  |
Mixture of SPL and SPCL changing by epoch. | mentornet_pd |
| MentorNet Data-driven |  |
Learned on a small subset of the CIFAR data. | mentornet_dd |
Note there are many more curriculums can be trained by MentorNet, for example, prediction variance (Chang et al., 2017), implicit regularizer (Fan et al. 2017), self-paced with diversity (Jiang et al. 2014), sample re-weighting (Dehghani et al., 2018, Ren et al., 2018), etc.
Performance
The numbers are slightly different from the ones reported in the paper due to the re-implementation on the third party library.
CIFAR-10 ResNet
| noise_fraction | baseline | self_paced | focal_loss | mentornet_pd | mentornet_dd |
|---|---|---|---|---|---|
| 0.2 | 0.796 | 0.822 | 0.797 | 0.910 | 0.914 |
| 0.4 | 0.568 | 0.802 | 0.634 | 0.776 | 0.887 |
| 0.8 | 0.238 | 0.297 | 0.25 | 0.283 | 0.463 |
CIFAR-100 ResNet
| noise_fraction | baseline | self_paced | focal_loss | mentornet_pd | mentornet_dd |
|---|---|---|---|---|---|
| 0.2 | 0.624 | 0.652 | 0.613 | 0.733 | 0.726 |
| 0.4 | 0.448 | 0.509 | 0.467 | 0.567 | 0.675 |
| 0.8 | 0.084 | 0.089 | 0.079 | 0.193 | 0.301 |
CIFAR-10 Inception
| noise_fraction | baseline | self_paced | focal_loss | mentornet_pd | mentornet_dd |
|---|---|---|---|---|---|
| 0.2 | 0.775 | 0.784 | 0.747 | 0.798 | 0.800 |
| 0.4 | 0.72 | 0.733 | 0.695 | 0.731 | 0.763 |
| 0.8 | 0.29 | 0.272 | 0.309 | 0.312 | 0.461 |
CIFAR-100 Inception
| noise_fraction | baseline | self_paced | focal_loss | mentornet_pd | mentornet_dd |
|---|---|---|---|---|---|
| 0.2 | 0.42 | 0.408 | 0.391 | 0.451 | 0.466 |
| 0.4 | 0.346 | 0.32 | 0.313 | 0.386 | 0.411 |
| 0.8 | 0.108 | 0.091 | 0.107 | 0.125 | 0.203 |
Algorithm
We propose an algorithm to optimize the StudentNet model parameter w jointly with a
given MentorNet. Unlike the alternating minimization, it minimizes w (StudentNet parameter) and v (sample weight) stochastically over mini-batches.
The curriculum can change during training, and MentorNet is updated a few times in the algorithm.
To learn new curriculums (Step 6), see this page.
We found specific MentorNet architectures do not matter that much.
References
- Bengio, Yoshua, et al. "Curriculum learning". In ICML, 2009.
- Kumar M. Pawan, Packer Benjamin, and Koller Daphne "Self-paced learning for latent variable models". In NIPS, 2010.
- Jiang, Lu et al. "Self-paced Learning with Diversity", In NIPS 2014
- Jiang, Lu, et al. "Self-Paced Curriculum Learning." In AAAI. 2015.
- Liang, Junwei et al. Learning to Detect Concepts from Webly-Labeled Video Data, In IJCAI 2016.
- Lin, Tsung-Yi, et al. "Focal loss for dense object detection." In ICCV. 2017.
- Fan, Yanbo, et al. "Self-Paced Learning: an Implicit Regularization Perspective." In AAAI 2017.
- Felzenszwalb, Pedro, et al. "A discriminatively trained, multiscale, deformable part model." In CVPR 2008.
- Dehghani, Mostafa, et al. "Fidelity-Weighted Learning." In ICLR 2018.
- Ren, Mengye, et al. "Learning to reweight examples for robust deep learning." In ICML 2018.
- Fan, Yang, et al. "Learning to Teach." In ICLR 2018.
- Chang, Haw-Shiuan, et al. "Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples." In NIPS 2017.
1k Jan 3, 2023
21 Dec 13, 2022
1.2k Dec 26, 2022
79 Dec 19, 2022
14 Sep 30, 2022
14 Oct 14, 2022
6 Sep 27, 2022
1 Nov 12, 2021
2 Nov 14, 2022
922 Dec 27, 2022
161 Dec 20, 2022
13.9k Dec 26, 2022
210 Jan 4, 2023
84 Jan 4, 2023
214 Jan 8, 2023
7 Oct 22, 2021
106 Dec 29, 2022
188 Dec 29, 2022
14 Sep 16, 2022