Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift (ICCV 2021)

Related tags

Deep Learning Pi-NAS
Overview

Π-NAS

This repository provides the evaluation code of our submitted paper: Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift.

Our Trained Models

  • Here is a summary of our searched models:

    ImageNet FLOPs Params Acc@1 Acc@5
    Π-NAS-cls 5.38G 27.1M 81.6% 95.7%
    Mask-RCNN on COCO 2017 APbb APmk
    Π-NAS-trans 44.07 39.50
    DeeplabV3 on ADE20K pixAcc mIoU
    Π-NAS-trans 81.27 45.47
    DeeplabV3 on Cityscapes mIoU
    Π-NAS-trans 80.70

Usage

1. Requirements

  • Install third-party requirements with command pip install -e .
  • Prepare ImageNet, COCO 2017, ADE20K and Cityscapes datasets
    • Our data paths are at /data/ImageNet, /data/coco, /data/ADEChallengeData2016 and /data/citys, respectively.
    • You can specify COCO's data path through environment variable DETECTRON2_DATASETS and others in experiments/recognition/verify.py, encoding/datasets/ade20k.py and encoding/datasets/cityscapes.py.
  • Download our checkpoint files

2. Evaluate our models

  • You can evaluate our models with the following command:

    ImageNet FLOPs Params Acc@1 Acc@5
    Π-NAS-cls 5.38G 27.1M 81.6% 95.7%
    python experiments/recognition/verify.py --dataset imagenet --model alone_resnest50 --choice-indices 3 0 1 3 2 3 1 2 0 3 2 1 3 0 3 2 --resume /path/to/PiNAS_cls.pth.tar
    Mask-RCNN on COCO 2017 APbb APmk
    Π-NAS-trans 44.07 39.50
    DETECTRON2_DATASETS=/data python experiments/detection/plain_train_net.py --config-file experiments/detection/configs/mask_rcnn_ResNeSt_50_FPN_syncBN_1x.yaml --num-gpus 8 --eval-only MODEL.WEIGHTS /path/to/PiNAS_trans_COCO.pth MODEL.RESNETS.CHOICE_INDICES [3,3,3,3,1,1,3,3,3,0,0,1,1,0,2,1]
    DeeplabV3 on ADE20K pixAcc mIoU
    Π-NAS-trans 81.27 45.47
    python experiments/segmentation/test.py --dataset ADE20K --model deeplab --backbone alone_resnest50 --choice-indices 3 3 3 3 1 1 3 3 3 0 0 1 1 0 2 1 --aux --se-loss --resume /path/to/PiNAS_trans_ade.pth.tar --eval
    DeeplabV3 on Cityscapes mIoU
    Π-NAS-trans 80.70
    python experiments/segmentation/test.py --dataset citys --base-size 2048 --crop-size 768 --model deeplab --backbone alone_resnest50 --choice-indices 3 3 3 3 1 1 3 3 3 0 0 1 1 0 2 1 --aux --se-loss --resume /path/to/PiNAS_trans_citys.pth.tar --eval

Training and Searching

This reimplementation is based on OpenSelfSup and MoCo. Please acknowledge their contribution.

cd OpenSelfSup && pip install -v -e .

1. Π-NAS Learning

bash tools/dist_train.sh configs/pinas_learning.py 8 --work_dir /path/to/save/logs/and/models

2. Extract supernet backbone weights

python tools/extract_backbone_weights.py /checkpoint/of/1. /extracted/weight/of/1.

3. Linear Training

bash tools/dist_train.sh configs/pinas_linear_training.py 8 --pretrained /extracted/weight/of/1. --work_dir /path/to/save/logs/and/models

4. Linear Evaluation

bash tools/dist_train.sh configs/pinas_linear_evaluation.py 8 --resume_from /checkpoint/of/3. --work_dir /path/to/save/logs/and/models
You might also like...
[ICLR2021oral] Rethinking Architecture Selection in Differentiable NAS

DARTS-PT Code accompanying the paper ICLR'2021: Rethinking Architecture Selection in Differentiable NAS Ruochen Wang, Minhao Cheng, Xiangning Chen, Xi

《Truly shift-invariant convolutional neural networks》(2021)

Truly shift-invariant convolutional neural networks [Paper] Authors: Anadi Chaman and Ivan Dokmanić Convolutional neural networks were always assumed

[CVPR 2022]
[CVPR 2022] "The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy" by Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, Zhangyang Wang

The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy Codes for this paper: [CVPR 2022] The Pr

Measuring and Improving Consistency in Pretrained Language Models

ParaRel 🤘 This repository contains the code and data for the paper: Measuring and Improving Consistency in Pretrained Language Models as well as the

DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks
DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

What is DeepHyper? DeepHyper is a software package that uses learning, optimization, and parallel computing to automate the design and development of

MXNet implementation for:  Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
MXNet implementation for: Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution

Octave Convolution MXNet implementation for: Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution Imag

Model search is a framework that implements AutoML algorithms for model architecture search at scale
Model search is a framework that implements AutoML algorithms for model architecture search at scale

Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers speed up their exploration process for finding the right model architecture for their classification problems (i.e., DNNs with different types of layers).

[ICLR 2021] "Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective" by Wuyang Chen, Xinyu Gong, Zhangyang Wang

Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective [PDF] Wuyang Chen, Xinyu Gong, Zhangyang Wang In ICLR 2

Official PyTorch implementation of
Official PyTorch implementation of "Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets" (ICLR 2021)

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets This is the official PyTorch implementation for the paper Rapid Neural A

Comments
  • Reproduce experiments on NAS-Bench-201

    Reproduce experiments on NAS-Bench-201

    Could you release the code you used for the experiments on NAS-Bench-201 on CIFAR10? If not, could you kindly share the hyperparameters used for NB201? (e.g. lr, epochs, queue_len,,) Thanks.

    opened by yhuangcb 0
Owner
Jiqi Zhang
Jiqi Zhang
"NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search".

NAS-Bench-301 This repository containts code for the paper: "NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search". The

AutoML-Freiburg-Hannover 57 Nov 30, 2022
CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification (ICCV2021)

CM-NAS Official Pytorch code of paper CM-NAS: Cross-Modality Neural Architecture Search for Visible-Infrared Person Re-Identification in ICCV2021. Vis

JDAI-CV 40 Nov 25, 2022
Naszilla is a Python library for neural architecture search (NAS)

A repository to compare many popular NAS algorithms seamlessly across three popular benchmarks (NASBench 101, 201, and 301). You can implement your ow

null 270 Jan 3, 2023
NAS Benchmark in "Prioritized Architecture Sampling with Monto-Carlo Tree Search", CVPR2021

NAS-Bench-Macro This repository includes the benchmark and code for NAS-Bench-Macro in paper "Prioritized Architecture Sampling with Monto-Carlo Tree

null 35 Jan 3, 2023
This repository is an implementation of paper : Improving the Training of Graph Neural Networks with Consistency Regularization

CRGNN Paper : Improving the Training of Graph Neural Networks with Consistency Regularization Environments Implementing environment: GeForce RTX™ 3090

THUDM 1 Dec 9, 2021
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training

ActNN : Activation Compressed Training This is the official project repository for ActNN: Reducing Training Memory Footprint via 2-Bit Activation Comp

UC Berkeley RISE 178 Jan 5, 2023
code for "AttentiveNAS Improving Neural Architecture Search via Attentive Sampling"

code for "AttentiveNAS Improving Neural Architecture Search via Attentive Sampling"

Facebook Research 94 Oct 26, 2022
code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? Code for paper: Does Unsupervised Architecture Representation

null 39 Dec 17, 2022
[CVPR 2021] 'Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator'

[CVPR2021] Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator Overview This is the entire codebase for the paper

null 35 Dec 1, 2022
Densely Connected Search Space for More Flexible Neural Architecture Search (CVPR2020)

DenseNAS The code of the CVPR2020 paper Densely Connected Search Space for More Flexible Neural Architecture Search. Neural architecture search (NAS)

Jamin Fong 291 Nov 18, 2022