Awesome AutoDL

A curated list of automated deep learning related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, and awesome-architecture-search.

Please feel free to pull requests or open an issue to add papers.

Table of Contents

Awesome Blogs
Awesome AutoDL Libraies
Awesome Benchmarks
Deep Learning-based NAS and HPO
Awesome Surveys

Awesome Blogs

Awesome AutoDL Libraies

Awesome Benchmarks

Title	Venue	Code
NAS-Bench-101: Towards Reproducible Neural Architecture Search	ICML 2019	GitHub
NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search	ICLR 2020	Github
NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search	arXiv 2020	GitHub
NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search	ICLR 2020	GitHub
NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size	TPAMI 2021	GitHub
NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition	ICLR 2021	-
HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark	ICLR 2021	GitHub
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing	arXiv 2020	GitHub
NAS-Bench-x11 and the Power of Learning Curves	NeurIPS 2021	GitHub

Deep Learning-based NAS and HPO

Type	G	RL	EA	PD	Other
Explanation	gradient-based	reinforcement learning	evolutionary algorithm	performance prediction	other types

2021 Venues

Title	Venue	Type	Code
CATE: Computation-aware Neural Architecture Encoding with Transformers	ICML	O	GitHub
Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator	CVPR	G	Github
Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition	ICCV	EA	Github
AutoFormer: Searching Transformers for Visual Recognition	ICCV	EA	GitHub
LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search	CVPR	EA	GitHub
One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking	CVPR	EA	GitHub
DARTS-: Robustly Stepping out of Performance Collapse Without Indicators	ICLR	G	GitHub
Zero-Cost Proxies for Lightweight NAS	ICLR	O	GitHub
Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective	ICLR	-	GitHub
DrNAS: Dirichlet Neural Architecture Search	ICLR	G	GitHub
Rethinking Architecture Selection in Differentiable NAS	ICLR	O	GitHub
Evolving Reinforcement Learning Algorithms	ICLR	EA	GitHub
AutoHAS: Differentiable Hyper-parameter and Architecture Search	ICLR-W	G	-
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function	CVPR	PD	github

2020 Venues

Title	Venue	Type	Code
Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search	NeurIPS	-	GitHub
PyGlove: Symbolic Programming for Automated Machine Learning	NeurIPS	library	-
Does Unsupervised Architecture Representation Learning Help Neural Architecture Search	NeurIPS	PD	GitHub
RandAugment: Practical Automated Data Augmentation with a Reduced Search Space	NeurIPS		GitHub
Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians	NeurIPS	G	GitHub
A Study on Encodings for Neural Architecture Search	NeurIPS		GitHub
AutoBSS: An Efficient Algorithm for Block Stacking Style Search	NeurIPS
Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS	NeurIPS	G	GitHub
Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding	NeurIPS
Revisiting Parameter Sharing for Automatic Neural Channel Number Search	NeurIPS
Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search	NeurIPS	MCTS	GitHub
Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search	AAAI	MCTS	GitHub
Representation Sharing for Fast Object Detector Search and Beyond	ECCV	G	GitHub
Are Labels Necessary for Neural Architecture Search?	ECCV	G	-
Single Path One-Shot Neural Architecture Search with Uniform Sampling	ECCV	EA	-
Neural Predictor for Neural Architecture Search	ECCV	O	-
BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models	ECCV	G	-
BATS: Binary ArchitecTure Search	ECCV	-	-
AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification	ECCV	-	-
Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization	ECCV	-	-
Angle-based Search Space Shrinking for Neural Architecture Search	ECCV	-	-
Anti-Bandit Neural Architecture Search for Model Defense	ECCV	-	-
TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search	ECCV	G	GitHub
Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search	ECCV	G	GitHub
Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search	ECCV	RL	-
DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search	ECCV	G	-
Optimizing Millions of Hyperparameters by Implicit Differentiation	AISTATS	G	-
Evolving Machine Learning Algorithms From Scratch	ICML	EA	-
Stabilizing Differentiable Architecture Search via Perturbation-based Regularization	ICML	G	GitHub
NADS: Neural Architecture Distribution Search for Uncertainty Awareness	ICML	-	-
Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data	ICML	-	-
Neural Architecture Search in a Proxy Validation Loss Landscape	ICML	-	-
Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection	CVPR	-	GitHub
Designing Network Design Spaces	CVPR	-	GitHub
UNAS: Differentiable Architecture Search Meets Reinforcement Learning	CVPR	G/RL	GitHub
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation	CVPR	G	GitHub
A Semi-Supervised Assessor of Neural Architectures	CVPR	PD	-
Binarizing MobileNet via Evolution-based Searching	CVPR	EA	-
Rethinking Performance Estimation in Neural Architecture Search	CVPR	-	GitHub
APQ: Joint Search for Network Architecture, Pruning and Quantization Policy	CVPR	G	GitHub
SGAS: Sequential Greedy Architecture Search	CVPR	G	Github
Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS	CVPR	RL	-
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions	CVPR	G	Github
AdversarialNAS: Adversarial Neural Architecture Search for GANs	CVPR	G	Github
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks	CVPR	G	Github
Block-wisely Supervised Neural Architecture Search with Knowledge Distillation	CVPR	G	Github
Overcoming Multi-Model Forgetting in One-Shot NAS with Diversity Maximization	CVPR	G	Github
Densely Connected Search Space for More Flexible Neural Architecture Search	CVPR	G	Github
EfficientDet: Scalable and Efficient Object Detection	CVPR	RL	-
NAS-BENCH-201: Extending the Scope of Reproducible Neural Architecture Search	ICLR	-	Github
Understanding Architectures Learnt by Cell-based Neural Architecture Search	ICLR	G	GitHub
Evaluating The Search Phase of Neural Architecture Search	ICLR	-
AtomNAS: Fine-Grained End-to-End Neural Architecture Search	ICLR		GitHub
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search	ICLR	-	GitHub
Once for All: Train One Network and Specialize it for Efficient Deployment	ICLR	G	GitHub
Efficient Transformer for Mobile Applications	ICLR	-	-
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search	ICLR	G	GitHub
Adversarial AutoAugment	ICLR	-	-
NAS evaluation is frustratingly hard	ICLR	-	GitHub
FasterSeg: Searching for Faster Real-time Semantic Segmentation	ICLR	G	GitHub
Computation Reallocation for Object Detection	ICLR	-	-
Towards Fast Adaptation of Neural Architectures with Meta Learning	ICLR	-	GitHub
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures	ICLR	EA	-
How to Own the NAS in Your Spare Time	ICLR	-	-
Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search	ICPR	G	github

2019 Venues

Title	Venue	Type	Code
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions	ICLR	-	-
DATA: Differentiable ArchiTecture Approximation	NeurIPS	-	-
Random Search and Reproducibility for Neural Architecture Search	UAI	G	GitHub
Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition	EMNLP	G	-
Continual and Multi-Task Architecture Search	ACL	RL	-
Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation	ICCV	G	GitHub
Multinomial Distribution Learning for Effective Neural Architecture Search	ICCV	-	GitHub
Searching for MobileNetV3	ICCV	EA	-
Multinomial Distribution Learning for Effective Neural Architecture Search	ICCV	-	GitHub
Fast and Practical Neural Architecture Search	ICCV
Teacher Guided Architecture Search	ICCV		-
AutoDispNet: Improving Disparity Estimation With AutoML	ICCV	G	-
Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?	ICCV	EA	-
One-Shot Neural Architecture Search via Self-Evaluated Template Network	ICCV	G	Github
Evolving Space-Time Neural Architectures for Videos	ICCV	EA	GitHub
AutoGAN: Neural Architecture Search for Generative Adversarial Networks	ICCV	RL	github
Discovering Neural Wirings	NeurIPS	G	Github
Towards modular and programmable architecture search	NeurIPS	Other	Github
Network Pruning via Transformable Architecture Search	NeurIPS	G	Github
Deep Active Learning with a NeuralArchitecture Search	NeurIPS	-	-
DetNAS: Backbone Search for Object Detection	NeurIPS	EA	GitHub
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers	NeurIPS	-	-
Efficient Forward Architecture Search	NeurIPS	G	Github
Efficient Neural ArchitectureTransformation Search in Channel-Level for Object Detection	NeurIPS	G	-
XNAS: Neural Architecture Search with Expert Advice	NeurIPS	G	GitHub
DARTS: Differentiable Architecture Search	ICLR	G	github
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware	ICLR	RL/G	github
Graph HyperNetworks for Neural Architecture Search	ICLR	G	-
Learnable Embedding Space for Efficient Neural Architecture Compression	ICLR	Other	github
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution	ICLR	EA	-
SNAS: stochastic neural architecture search	ICLR	G	-
NetTailor: Tuning the Architecture, Not Just the Weights	CVPR	G	Github
Searching for A Robust Neural Architecture in Four GPU Hours	CVPR	G	Github
ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation	CVPR	-	-
Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search	CVPR	EA	github
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search	CVPR	G	-
RENAS: Reinforced Evolutionary Neural Architecture Search	CVPR	G	-
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation	CVPR	G	GitHub
MnasNet: Platform-Aware Neural Architecture Search for Mobile	CVPR	RL	Github
MFAS: Multimodal Fusion Architecture Search	CVPR	EA	-
A Neurobiological Evaluation Metric for Neural Network Model Search	CVPR	Other	-
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells	CVPR	RL	-
Customizable Architecture Search for Semantic Segmentation	CVPR	-	-
Regularized Evolution for Image Classifier Architecture Search	AAAI	EA	-
AutoAugment: Learning Augmentation Policies from Data	CVPR	RL	-
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules	ICML	EA	-
The Evolved Transformer	ICML	EA	Github
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks	ICML	RL	-
NAS-Bench-101: Towards Reproducible Neural Architecture Search	ICML	Other	Github
On Network Design Spaces for Visual Recognition	ICCV	G	Github

2018 Venues

Title	Venue	Type	Code
Towards Automatically-Tuned Deep Neural Networks	BOOK	-	GitHub
NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications	ECCV	-	github
Efficient Architecture Search by Network Transformation	AAAI	RL	github
Learning Transferable Architectures for Scalable Image Recognition	CVPR	RL	github
N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning	ICLR	RL	-
A Flexible Approach to Automated RNN Architecture Generation	ICLR	RL/PD	-
Practical Block-wise Neural Network Architecture Generation	CVPR	RL	-
Path-Level Network Transformation for Efficient Architecture Search	ICML	RL	github
Hierarchical Representations for Efficient Architecture Search	ICLR	EA	-
Understanding and Simplifying One-Shot Architecture Search	ICML	G	-
SMASH: One-Shot Model Architecture Search through HyperNetworks	ICLR	G	github
Neural Architecture Optimization	NeurIPS	G	github
Searching for efficient multi-scale architectures for dense image prediction	NeurIPS	Other	-
Progressive Neural Architecture Search	ECCV	PD	github
Neural Architecture Search with Bayesian Optimisation and Optimal Transport	NeurIPS	Other	github
Differentiable Neural Network Architecture Search	ICLR-W	G	-
Accelerating Neural Architecture Search using Performance Prediction	ICLR-W	PD	-

2017 Venues

Title	Venue	Type	Code
Neural Architecture Search with Reinforcement Learning	ICLR	RL	-
Designing Neural Network Architectures using Reinforcement Learning	ICLR	RL	-
Neural Optimizer Search with Reinforcement Learning	ICML	RL	-
Learning Curve Prediction with Bayesian Neural Networks	ICLR	PD	-
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization	ICLR	PD	-
Hyperparameter Optimization: A Spectral Approach	NeurIPS-W	Other	github
Learning to Compose Domain-Specific Transformations for Data Augmentation	NeurIPS	-	-

Previous Venues

2012-2016

Title	Venue	Type	Code
Speeding up Automatic Hyperparameter Optimization of Deep Neural Networksby Extrapolation of Learning Curves	IJCAI	PD	github

arXiv

Title	Date	Type	Code
NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search	2018.10	EA	-
Training Frankenstein’s Creature to Stack: HyperTree Architecture Search	2018.10	G	-
Population Based Training of Neural Networks	2017.11	EA	GitHub

Awesome Surveys

Title	Venue	Year	Code
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions	ACM Computing Surveys	2021	-
Automated Machine Learning on Graphs: A Survey	ICLR-W	2021	GitHub
On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice	Neurocomputing	2020	github
AutonoML: Towards an Integrated Framework for Autonomous Machine Learning	arXiv	2020	-
Automated Machine Learning	Springer Book	2019	-
Neural architecture search: A survey	JMLR	2019	-
AutoML: A Survey of the State-of-the-Art	arXiv	2019	GitHub
A Survey on Neural Architecture Search	arXiv	2019	-
Taking human out of learning applications: A survey on automated machine learning	arXiv	2018	-

Zen-NAS ICCV 2021

Dear D-X-Y,

We have one ICCV 2021 paper about zero-shot Neural Architecture Search:

paper arXiv link: https://arxiv.org/abs/2102.01063

Github: https://github.com/idstcv/ZenNAS

It would so nice of you if you could kindly append this work to the paper list. Thank you very much!

opened by MingLin-home 3
Please add NeurIPS 2019 paper

Towards modular and programmable architecture search Paper: https://arxiv.org/abs/1909.13404 Code: https://github.com/negrinho/deep_architect

Thank you

opened by negrinho 3
ICLR 2020 Paper

Hi Xuanyi, We have a NAS related paper in ICLR 2020, 'Fast Neural Network Adaptation via Parameter Remapping and Architecture Search'. Paper: https://openreview.net/forum?id=rklTmyBKPH Code: https://github.com/JaminFong/FNA We wonder if it is OK to add this paper. Thank you!

opened by JaminFong 2
ICLR 2019 Paper

Hello,

Here is the link to our ICLR 2019 paper on automatically searching for a compressed architecture with Bayesian optimization: https://openreview.net/forum?id=S1xLN3C9YX . Our code is also available here: https://github.com/Friedrich1006/ESNAC .

Our lab also has a ICLR 2018 paper (https://openreview.net/forum?id=B1hcZZ-AW) that uses reinforcement learning to search for compressed architectures.

Would you please add them to the list? Thanks!

opened by Minione 2
Add new papers

Hi,

Would you please add our recent paper on NAS to the list? The title is Neural Architecture Optimization, which is accepted at NIPS this year. Maybe it can be categorized into the gradient based method. Thanks.

Best

opened by ustctf-zz 2
New paper on CVPR 2020

Hi Xuanyi, Thanks for the awesome paper list for NAS!

Here is our CVPR 2020 paper about NAS based on performance prediction. Paper Title: A Semi-Supervised Assessor of Neural Architectures Paper link: https://openaccess.thecvf.com/content_CVPR_2020/papers/Tang_A_Semi-Supervised_Assessor_of_Neural_Architectures_CVPR_2020_paper.pdf

Thanks!

opened by yehuitang 1
Add a Hyperparameter Optimization survey paper

May I add a comprehensive Hyperparameter Optimization survey paper to this awesome list, please? It is a very comprehensive survey paper about Hyperparameter Optimization of machine learning and deep learning algorithms. I've also added its Github code, which provides examples of hyperparameter tuning in deep learning. Thanks!

opened by LiYangHart 1

AutoML survey

There are over 30 citations of this survey.

@article{he2019automl,
  title={AutoML: A Survey of the State-of-the-Art},
  author={He, Xin and Zhao, Kaiyong and Chu, Xiaowen},
  journal={arXiv preprint arXiv:1908.00709},
  year={2019}
}

opened by marsggbo 1

New Paper Adding CVPR 2020

Hi Xuanyi,

Thanks for making this awesome summary list!

Here is our new CVPR 2020 paper "Binarizing MobileNet via Evolution-based Searching" Paper Link: https://arxiv.org/abs/2005.06305 Type is Evolution Algorithm.

Thanks!

opened by liuzechun 1
New Paper Adding

Hi,

Here is our new CVPR 2020 paper "AdversarialNAS: Adversarial Neural Architecture Search for GANs" Paper Link: https://arxiv.org/pdf/1912.02037.pdf Code link: https://github.com/chengaopro/AdversarialNAS Type is gradient-based.

Thx!

opened by chengaopro 1
Request to Add THIN-Bayes

Paper: https://www.researchgate.net/profile/Sandeep-Sandha/publication/356911955_Enabling_Hyperparameter_Tuning_of_Machine_Learning_Classifiers_in_Production/links/62171fb21857671d0d8b04f4/Enabling-Hyperparameter-Tuning-of-Machine-Learning-Classifiers-in-Production.pdf

Poster: https://cms.tinyml.org/wp-content/uploads/talks2022/Saha-S.-Sayan-SW-tools.pdf

Code: https://github.com/ARM-software/mango/tree/master/examples/THIN-Bayes

We developed a new state-of-art Bayesian NAS framework. We would be happy if you add it to your list.

opened by swapnilsayansaha 1

A curated list of automated deep learning (including neural architecture search and hyper-parameter optimization) resources.

Related tags

Overview

Awesome Blogs

Awesome AutoDL Libraies

Awesome Benchmarks

Deep Learning-based NAS and HPO

2021 Venues

2020 Venues

2019 Venues

2018 Venues

2017 Venues

Previous Venues

arXiv

Awesome Surveys

Comments

Owner

D-X-Y

A curated list of awesome resources related to Semantic Search🔎 and Semantic Similarity tasks.

A curated list of neural network pruning resources.

A curated list of neural rendering resources.

A curated (most recent) list of resources for Learning with Noisy Labels

A curated list of programmatic weak supervision papers and resources

A curated list of resources for Image and Video Deblurring

code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"

Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

Densely Connected Search Space for More Flexible Neural Architecture Search (CVPR2020)

Milano is a tool for automating hyper-parameters search for your models on a backend of your choice.

Deep Multimodal Neural Architecture Search

Model search is a framework that implements AutoML algorithms for model architecture search at scale

Official PyTorch implementation of "Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets" (ICLR 2021)

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

Automates Machine Learning Pipeline with Feature Engineering and Hyper-Parameters Tuning :rocket:

"NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search".

PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation

[ICLR 2021] "Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective" by Wuyang Chen, Xinyu Gong, Zhangyang Wang