Hi @changlin31 , thank you for your great work! I really enjoyed your paper. I want to ask you a question regarding the checkpoint of models under constraint.

• After running the ii) Search for the best architecture under constraint, I only get the best model architecture w/o saving this trained single best model under constraint. If I want to test this best model architecture, should I load the student supernet from the checkpoint in step (i) and use the encoding code to perform this subnetwork in the supernet? Since we have already trained these blocks w/ KD in the previous step (i), I think I can get the trained model directly in the step (ii) and get something like a single model checkpoint like DNA_a, DNA_b, etc?

• Can you point out how to setup the constraint to get these DNA_a ~ DNA_d?

Thank you in advance!

训练方法：
--lr=0.05
--n_gpu=4
--batch_size=256
--n_worker=32
--lr_type=cos
--n_epoch=150
--wd=4e-5
--seed=2018
--optim=SGD

    # preprocessing
    input_size = 224
    imagenet_tran_train = [
        transforms.RandomResizedCrop(input_size, scale=(0.2, 1.0)),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        #transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
    ]
    imagenet_tran_test = [
        transforms.Resize(int(input_size / 0.875)),
        #transforms.Resize([256,256]),
        transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
        #transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),         
    ]

结果： acc/train_top1：80 acc/test_top1：74 能看一下是什么原因吗？或者训练代码或者细节能够开源吗？谢谢

Thanks for your nice work and released code. We have tried the retraining part on ImageNet and here are some questions.

1. When retraining the searched model under your default training settings (8gpus in one machine), we get the accuracy below: DNA_a: 76.31000003662109 (epoch 496) DNA_b: 76.63600003417969 (epoch 483) DNA_c: 77.20800003662109 (epoch 474) DNA_d: 77.7220000366211 (epoch 433)

2. We read the issue in #10 , and change the training settings. Specifically, we use 32 gpus, the batchsize=128 and lr=0.256, and optimize the network at each step.
   (--nproc_per_node=8 --nnodes=4) --model ${model_name} --epochs 500 --warmup-epochs 5 --batch-size 128 --lr 0.256 --opt rmsproptf --opt-eps 0.001 \ --sched step --decay-epochs 3 --decay-rate 0.963 --color-jitter 0.06 --drop 0.2 -j 8 --num-classes 1000 --model-ema \ We only retrained the DNA_a to check the accuracy. However, we only get a worse result: DNA_a best metric=76.08965138479905 (epoch 486).

Could you please help me find out why this difference? Great thanks and best wishes.

Firstly, thanks for your nice work! However, the dimension mismatch occurs when the student model blocks' output is changing but the output dimension of teacher model is fixed and I haven't found more details about this issue in your paper. Especially the channel, did you just take the first N channels directly? So could you please tell me how to deal with it?

Hi,

I followed the steps in the README but saw errors during searching using either single-GPU or multi-GPU boxes.

Have you encountered these issues before or have any idea how to fix them? TIA.

• single-GPU: I modified set --nproc_per_node=1. The searching started as expected but couldn't finish stage 0. The error message is as follows:

12/28 07:27:02 AM WORLD_SIZE in os.environ is 1
12/28 07:27:02 AM Namespace(amp=False, batch_size=64, color_jitter=0.4, cooldown_epochs=0, data_config=None, datadir='/home/ubuntu/workspace/datasets/ILSVRC2012/', dataset='imagenet', decay_epochs=1, decay_rate=0.9, distill_last_stage=True, distributed=False, eval_intervals=2, eval_metric='prec1', eval_mode=False, exp_dir='', feature_train=True, guide_input=True, guide_loss_fn='mse', hyperparam_config=None, img_size=224, index='', init_classifier=False, interpolation='', label_train=False, local_rank=0, log_interval=50, loss_weight=[0.5, 0.5], lr=[0.002, 0.005, 0.005, 0.005, 0.005, 0.002], mean=None, min_lr=1e-08, mixup=0.0, mixup_off_epoch=0, model_ema=False, model_ema_decay=0.9998, model_ema_force_cpu=False, model_pool='', momentum=0.9, num_classes=1000, num_gpu=1, opt='adam', opt_eps=1e-08, output='', potential_eval_times=20, prefetcher=True, pretrain=False, print_detail=True, recovery_interval=0, remode='pixel', reprob=0.5, reset_after_stage=False, reset_bn_eval=True, resume='', reverse_train=False, save_images=False, save_last_feature=True, sched='step', seed=42, separate_train=False, smoothing=0.1, stage_num=6, start_epoch=None, start_stage=None, std=None, step_epochs=20, sync_bn=False, test_dispatch='', top_model_num=3, train_mode=False, update_frequency=1, warmup_epochs=0, warmup_lr=0.001, weight_decay=0.0001, workers=4)
12/28 07:27:02 AM Training with a single process on 1 GPUs.
12/28 07:27:04 AM Data processing configuration for current model + dataset:
12/28 07:27:04 AM       input_size: (3, 224, 224)
12/28 07:27:04 AM       interpolation: bicubic
12/28 07:27:04 AM       mean: (0.485, 0.456, 0.406)
12/28 07:27:04 AM       std: (0.229, 0.224, 0.225)
12/28 07:27:04 AM       crop_pct: 0.875
12/28 07:27:06 AM NVIDIA APEX installed. AMP off.
12/28 07:27:32 AM
Train: stage 0, epoch 1, step [   0/20018]  Loss: 109.597771 (109.5978)  Time: 2.011s,   31.82/s  LR: 1.800e-03  Data & Guide Time: 1.644
GuideMean: -0.64644  GuideStd: 10.40032  OutMean: 0.00000 (0.00000)  OutStd: 0.99985 (0.99985)  Dist_Mean: 0.64644 (0.64644)
GRLoss: 1.00459 (1.00459)  CLLoss: 0.79709 (0.79709)  KLCosLoss: 0.57991 (0.57991)
FeatureLoss: 0.00000 (0.00000)  Top1Acc: 0.00000(0.00000)
Relative MSE loss: 1.01323(1.01323)

.....
12/29 06:58:47 AM Random Test: stage 0, epoch 20  Loss: 20.4754  Prec@1: 0.0000  Time: 0.216s,   74.05/s
12/29 06:58:48 AM Current checkpoints:
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-6.pth.tar', 19.889211503295897)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-14.pth.tar', 19.960276111450195)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-4.pth.tar', 19.97588088684082)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-16.pth.tar', 20.030977337646483)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-8.pth.tar', 20.106792897033692)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-10.pth.tar', 20.107453624572752)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-12.pth.tar', 20.242049604492188)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-18.pth.tar', 20.277006747436523)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-2.pth.tar', 20.39269996520996)
 ('./output/test/adam-step-ep20-lr0.002-bs64-20201228-072702/checkpoint-0-20.pth.tar', 20.47537907836914)

Traceback (most recent call last):
  File "train.py", line 273, in <module>
    main()
  File "train.py", line 268, in main
    writer=writer)
  File "/home/ubuntu/workspace/repos/DNA/searching/dna/distill_train.py", line 100, in distill_train
    reset_data=reset_data)
  File "/home/ubuntu/workspace/repos/DNA/searching/dna/distill_train.py", line 695, in _potential
    for layer in supernet.module.modules():
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 576, in __getattr__
    type(self).__name__, name))
AttributeError: 'StudentSuperNet' object has no attribute 'module'
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ubuntu/anaconda3/envs/pytorch_p36/bin/python', '-u', 'train.py', '--local_rank=0']' returned non-zero exit status 1.

• multi-GPU: --nproc_per_node=4 but it resulted in set faults.

12/30 05:01:12 AM WORLD_SIZE in os.environ is 4
12/30 05:01:12 AM Namespace(amp=False, batch_size=64, color_jitter=0.4, cooldown_epochs=0, data_config=None, datadir='/home/ubuntu/workspace/datasets/ILSVRC2012/', dataset='imagenet', decay_epochs=1, decay_rate=0.9, distill_last_stage=True, distributed=False, eval_intervals=2, eval_metric='prec1', eval_mode=False, exp_dir='', feature_train=True, guide_input=True, guide_loss_fn='mse', hyperparam_config=None, img_size=224, index='', init_classifier=False, interpolation='', label_train=False, local_rank=0, log_interval=50, loss_weight=[0.5, 0.5], lr=[0.002, 0.005, 0.005, 0.005, 0.005, 0.002], mean=None, min_lr=1e-08, mixup=0.0, mixup_off_epoch=0, model_ema=False, model_ema_decay=0.9998, model_ema_force_cpu=False, model_pool='', momentum=0.9, num_classes=1000, num_gpu=1, opt='adam', opt_eps=1e-08, output='', potential_eval_times=20, prefetcher=True, pretrain=False, print_detail=True, recovery_interval=0, remode='pixel', reprob=0.5, reset_after_stage=False, reset_bn_eval=True, resume='', reverse_train=False, save_images=False, save_last_feature=True, sched='step', seed=42, separate_train=False, smoothing=0.1, stage_num=6, start_epoch=None, start_stage=None, std=None, step_epochs=20, sync_bn=False, test_dispatch='', top_model_num=3, train_mode=False, update_frequency=1, warmup_epochs=0, warmup_lr=0.001, weight_decay=0.0001, workers=4)
12/30 05:01:12 AM Training in distributed mode with multiple processes, 1 GPU per process. CUDA 2, Process 2, total 4.
12/30 05:01:12 AM Training in distributed mode with multiple processes, 1 GPU per process. CUDA 3, Process 3, total 4.
12/30 05:01:13 AM Training in distributed mode with multiple processes, 1 GPU per process. CUDA 1, Process 1, total 4.
12/30 05:01:13 AM Training in distributed mode with multiple processes, 1 GPU per process. CUDA 0, Process 0, total 4.
12/30 05:01:15 AM Data processing configuration for current model + dataset:
12/30 05:01:15 AM       input_size: (3, 224, 224)
12/30 05:01:15 AM       interpolation: bicubic
12/30 05:01:15 AM       mean: (0.485, 0.456, 0.406)
12/30 05:01:15 AM       std: (0.229, 0.224, 0.225)
12/30 05:01:15 AM       crop_pct: 0.875
12/30 05:01:18 AM NVIDIA APEX installed. AMP off.
ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@ERROR: Unexpected segmentation fault encountered in worker.
^@Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/site-packages/torch/distributed/launch.py", line 256, in main
    process.wait()
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/subprocess.py", line 1477, in wait
    (pid, sts) = self._try_wait(0)
  File "/home/ubuntu/anaconda3/envs/pytorch_p36/lib/python3.6/subprocess.py", line 1424, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)

Hi,

Thanks for sharing the training code. I try to retrain DNA_c with this config: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 ./distributed_train.sh 8 ~/imagenet --model DNA_c \ --epochs 500 --warmup-epochs 5 --batch-size 128 --lr 0.064 --opt rmsproptf --opt-eps 0.001 --sched step --decay-epochs 3 --decay-rate 0.963 --color-jitter 0.06 --drop 0.2 -j 8 --num-classes 1000 --model-ema After 500 epochs training, the best top1 accuracy is 77.2%, which is 0.6% lower than paper. *** Best metric: 77.19799990478515 (epoch 458)

In the paper, I notice there are CIFAR transfer results on EfficientNet/MixNet and also DNA. Could you be kind to share the details? E.g., what kind of network modifications need to be done for CIFAR? Did you upscale to 224*224? What are the hyperparameters (lr, optimizer, epochs, batch size etc.) for transfer training? Any hint is deeply appreciated.

错误显示如下： File "D:/test/DNA-master-new/DNA-master/searching/train.py", line 277, in main() File "D:/test/DNA-master-new/DNA-master/searching/train.py", line 272, in main writer=writer) File "D:\test\DNA-master-new\DNA-master\searching\dna\distill_train.py", line 104, in distill_train reset_data=reset_data) File "D:\test\DNA-master-new\DNA-master\searching\dna\distill_train.py", line 699, in _potential for layer in supernet.module.modules(): File "D:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 594, in getattr type(self).name, name)) AttributeError: 'StudentSuperNet' object has no attribute 'module' 还请您能够帮忙解答！

I mainly focus on DNA-c model, but I found the model structure in the paper, architecture defined in code, and the released trained model are all different. Which model is the best? Could you provide a definite network structure? Thank you!

Dear authors: Thank you for your great work. Recently I tried to reproduce your paper and perform a complete train & search. By modifying the dist_train.sh file and changing nproc_per_node to 4 to suit my machine(4x3090), I succeeded to finish training stage 0-2, but when entering stage 3, the code hangs after printing infos for stage 3 epoch 1 step 0.

After doing some research, I found several interesting & strange details:

1. The code doesn't hang immediatly after entering stage 3, it successfully perform several complete steps of forward, backward, reduce and step, and hangs around step 6-7 (7 in most cases)
2. The (apparent) reason the program hangs is, one of the process is stuck in optimizer.step() after sucessfully calling loss.backward(). This is strange as I can't imagine how can optimizer.step() fail if gradients are propagated backward successfully but that's exactly the case. This process never print any debug log I set after optimizer.step(). The other processes just wait for it and the whole program hangs.
3. Process of any rank (including rank 0) can get stuck. Only one process gets stuck each time. The other three processes run just fine until the next all_reduce.

The above points, although are seemingly strange and random, can be reproduced stably on our machine.... We have tried several different versions of pytorch docker from nvidia(https://ngc.nvidia.com/catalog/containers/nvidia:pytorch) (1.7.0, 1.8.0 and newer ones) and this problem just continues. As both the codes and dockers are vanila, I can't tell which side the bug is coming from.

Update: Just when I type these lines, a classmate of mine told me that, after changing to another docker with pytorch 1.6.1 and cuda 11.0 (tag 20.06, the oldest docker from nvidia that supports cuda 11.0), this problem disappears mysteriously. I'm still posting this issue to tell researchers in the future, don't run this code on pytorch>1.6.1.

Hi,

I just need to verify I understand your workflow correctly which can also make things clearer for futuristic readers.

1. In the first distillation part sh dist_train.sh, the student supernet is defined as having a single cell per block/stage. Because the default block_cfg in the supernet class have a fixed number of layers and the get_all_models function only gets the combinations of a block's fixed number of layers with the number of candidate operations. Thus, in order to do the search for multiple cells/stage, I need to do multiple separate runs on this search script, with each time I modify the number of layers in block_cfg according to the desired width/depth, leading to multiple .yaml files generated per stage.

2. In process_potential.py, The traversal search can work with multiple cells per block by extending the list entries within potential_yaml and layers_cfgs lists to account for other .yaml files generated for different cells in each stage as this example: potential_yaml=[['$1st_cell$/potential-0.yaml', '$2nd_cell$/potential-0.yaml'], ['$1st_cell$/potential-1.yaml'], ..] and layer_cfgs=[[2,4,4,4,4,1],[3,4,4,4,4,1]]

Thanks in advance.

First of all, thank you for your codes! Using these codes, I'm trying to apply this for my code. Meanwhile I don't understand what are the meaning of subtracting numbers as following : 284976, 358960, 73368, 55412, 51540, etc. Can you explain it to me?

        stage3_max_param = stage4_max_param - 284976 * 4 + 358960
        stage2_max_param = stage3_max_param - 73368 * 3 + 55412
        stage1_max_param = stage2_max_param - 51540 * 3 + 18650
        stage0_max_param = stage1_max_param - 13770 * 3 + 6566

These are 108~111 lines of process_potential.py. Thank you.

Hi, Thanks for your excellent work. As the title show, how to set mse_weight? Any rules to follow ? mse_weight = [0.0684, 0.171, 0.3422, 0.2395, 0.5474, 0.3422] [(https://github.com/changlin31/DNA/blob/master/searching/dna/distill_train.py#L329)]