BossNAS
This repository contains PyTorch evaluation code, retraining code and pretrained models of our paper: BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search. [pdf link]
Illustration of the Siamese supernets training with ensemble bootstrapping.
Illustration of the fabric-like Hybrid CNN-transformer Search Space with flexible down-sampling positions.
Our Results and Trained Models
-
Here is a summary of our searched models:
Model MAdds Steptime Top-1 (%) Top-5 (%) Url BossNet-T0 w/o SE 3.4B 101ms 80.5 95.0 checkpoint BossNet-T0 3.4B 115ms 80.8 95.2 checkpoint BossNet-T0^ 5.7B 147ms 81.6 95.6 same as above BossNet-T1 7.9B 156ms 81.9 95.6 checkpoint BossNet-T1^ 10.5B 165ms 82.2 95.7 same as above -
Here is a summary of architecture rating accuracy of our method:
Search space Dataset Kendall tau Spearman rho Pearson R MBConv ImageNet 0.65 0.78 0.85 NATS-Bench Ss Cifar10 0.53 0.73 0.72 NATS-Bench Ss Cifar100 0.59 0.76 0.79
Usage
1. Requirements
- Install PyTorch 1.7.0+ and torchvision 0.8.1+, for example:
conda install -c pytorch pytorch torchvision
- Install pytorch-image-models 0.3.2, for example:
pip install timm==0.3.2
-
Download the ImageNet dataset from http://image-net.org/, and move validation images to labeled subfolders
- To do this, you can use the following script: https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.shvalprep.sh
2. Retrain or Evaluate our BossNet-T models
-
First, move to retraining code directory to perform Retraining or Evaluation.
cd HyTra_retrainingOur retraining code of BossNet-T is based on DeiT repository.
-
You can evaluate our BossNet-T models with the following command:
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T0-80_8.pth
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8 --eval --resume PATH/TO/BossNet-T1-81_9.pth
Please download our checkpoint files from the result table. Please change the
--nproc_per_nodeoption to suit your GPU numbers, and change the--data-path,--resumeand--input-sizeaccordingly. -
You can retrain our BossNet-T models with the following command:
Please change the
--nproc_per_nodeand--data-pathaccordingly. Note that the learning rate will be automatically scaled according to the GPU numbers and batchsize. We recommend training with 128 batchsize and 8 GPUs.python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T0 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
python -m torch.distributed.launch --nproc_per_node=8 --use_env main.py --model bossnet_T1 --input-size 224 --batch-size 128 --data-path /PATH/TO/ImageNet --num_workers 8
Architecture of our BossNet-T0
3. Evaluate architecture rating accuracy of BossNAS
-
You can get the ranking correlations of BossNAS on MBConv search space with the following commands:
cd MBConv_ranking python get_model_score_mbconv.py
- You can get the ranking correlations of BossNAS on NATS-Bench Ss with the following commands:
cd NATS_SS_ranking python get_model_score_nats.py
Citation
@article{li2021bossnas,
author = {Li, Changlin and
Tang, Tao and
Wang, Guangrun and
Peng, Jiefeng and
Wang, Bing and
Liang, Xiaodan and
Chang, Xiaojun},
title = {BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised Neural Architecture Search},
journal = {arXiv:2103.12424},
year = 2021,
}
TODO
Searching code will be released later.
5 Dec 10, 2022
39 Dec 17, 2022
8 Nov 2, 2022
291 Nov 18, 2022
5 Dec 13, 2021
84 Jan 1, 2023
214 Jan 8, 2023
92 Nov 30, 2022
3.2k Dec 31, 2022
5 Nov 3, 2022
82 Jan 1, 2023
139 Jan 1, 2023
156 Nov 28, 2022
72 Jan 3, 2023
23 Dec 21, 2022
48 Sep 30, 2022
57 Nov 30, 2022
290 Dec 24, 2022
18 May 27, 2022