PyTorch implementation for Convolutional Networks with Adaptive Inference Graphs

Andreas Veit

Last update: Dec 7, 2022

Related tags

Deep Learning convnet-aig

Overview

Convolutional Networks with Adaptive Inference Graphs (ConvNet-AIG)

This repository contains a PyTorch implementation of the paper Convolutional Networks with Adaptive Inference Graphs presented at ECCV 2018.

The code is based on the PyTorch example for training ResNet on Imagenet.

Introduction
Usage
Citing
Requirements
Contact

Introduction

Do convolutional networks really need a fixed feed-forward structure? What if, after identifying the high-level concept of an image, a network could move directly to a layer that can distinguish fine-grained differences? Currently, a network would first need to execute sometimes hundreds of intermediate layers that specialize in unrelated aspects. Ideally, the more a network already knows about an image, the better it should be at deciding which layer to compute next.

Convolutional networks with adaptive inference graphs (ConvNet-AIG) can adaptively define their network topology conditioned on the input image. Following a high-level structure similar to residual networks (ResNets), ConvNet-AIG decides for each input image on the fly which layers are needed. In experiments on ImageNet we show that ConvNet-AIG learns distinct inference graphs for different categories.

Usage

There are two training files. One for CIFAR-10 train.py and one for ImageNet train_img.py.

The network can be simply trained with python train.py or with optional arguments for different hyperparameters:

$ python train.py --expname {your experiment name}

For ImageNet the folder containing the dataset needs to be supplied

$ python train_img.py --expname {your experiment name} [imagenet-folder with train and val folders]

Training progress can be easily tracked with visdom using the --visdom flag. It keeps track of the learning rate, loss, training and validation accuracy as well as the activation rates of the gates for each class.

By default the training code keeps track of the model with the highest performance on the validation set. Thus, after the model has converged, it can be directly evaluated on the test set as follows

$ python train.py --test --resume runs/{your experiment name}/model_best.pth.tar

Requirements

This implementation is developed for

Python 3.6.5
PyTorch 0.3.1
CUDA 9.1

Target Rate schedules

To improve performance and memory efficiency, the target rates of early, last and downsampling layers can be fixed so as to always execute the layers. Specifically, for the results in the paper the following target rate schedules are used for ResNet 50: [1, 1, 0.8, 1, t, t, t, 1, t, t, t, t, t, 1, 0.7, 1] for t in [0.4, 0.5, 0.6, 0.7] For ResNet 101 the following rates can be used: ([1]* 8).extend([t] * 25) for t in [0.3, 0.5]

For compatibility to newer versions, please make a pull request.

Citing

If you find this helps your research, please consider citing:

@conference{Veit2018,
title = {Convolutional Networks with Adaptive Inference Graphs},
author = {Andreas Veit and Serge Belongie},
year = {2018},
journal = {European Conference on Computer Vision (ECCV)},
}

Contact

andreas at cs dot cornell dot edu

Comments

Two undefined names: 'ResNet18' and 'idx'

flake8 testing of https://github.com/andreasveit/convnet-aig on Python 3.7.0

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./convnet_aig.py:337:11: F821 undefined name 'ResNet18'
    net = ResNet18()
          ^
./gumbelmodule.py:64:24: F821 undefined name 'idx'
                return idx
                       ^
2     F821 undefined name 'ResNet18'
2

opened by cclauss 1

Add support for Python 3.7 on PyTourch 0.4.1

async is now a reserved word in Python 3.7 and later. To fix this pytorch/pytorch#4999 changed cuda(async) to cuda(non_blocking) so this PR tracks with that change.

Also added some missing imports.

opened by cclauss 1
Is the Gumbel-Softmax formulation accurate?

Thanks for releasing the code!

I have been reviewing how the Gumbel-Softmax[1] trick was used and both the paper and the code suggest that the "relevance scores are interpreted as log probabilities"[2] but how come the output of a convolutional layer is interpreted as being a strictly negative quantity? (This is unlikely to break training but silently yield suboptimal performance due to inaccurate approximate sampling from the discrete distribution)

Please let me know, maybe there is a subtle intuition or training dynamic at play here that I am missing. Thanks!

[1] https://arxiv.org/pdf/1611.01144.pdf (Equation 1) [2] https://arxiv.org/pdf/1711.11503.pdf (Section 3.3, page 5)

opened by atiorh 0
pre-trained model on ImageNet

Nice work. I want to use your network in my own applications. I wonder if it is possible for you to provide the pre-trained AIG model on ImageNet ?

It will help a lot. Thank you very much!

opened by yangyu12 0
Which part of the computation has been reduced?
As stated in the paper, the proposed convnet-aig can reduce computational cost. But after checking the source codes, I found each layer must be executed. The way you skip a single layer is by multiplying a mask (0 or 1) to the output. If that is the case, which part of the convnet-aig can reduce the computation? The code below is cited from convnet_aig.py, line 152, where out is the output of a layer, w[:,1] is the mask.

out = self.shortcut(x) + out * w[:,1].unsqueeze(1)
opened by Vincent-Hoo 1
ValueError: expected 2D or 3D input (got 4D input)
Hello Andreas Veit:

My PyTorch version is 4.0 , cuda is 9.0 and python is 3.5. I run the code, but I find this problem.

python train.py

problem:

I think the problem is ResNet110_cifar() function in convnet_aig.py file. I don't know how to fix the debug, can you help me? Thanks. @andreasveit
opened by MrLinNing 2

PyTorch implementation for Convolutional Networks with Adaptive Inference Graphs

Related tags

Overview

Convolutional Networks with Adaptive Inference Graphs (ConvNet-AIG)

Table of Contents

Introduction

Usage

Requirements

Target Rate schedules

Citing

Contact

Comments

Two undefined names: 'ResNet18' and 'idx'

Add support for Python 3.7 on PyTourch 0.4.1

Is the Gumbel-Softmax formulation accurate?

pre-trained model on ImageNet

Which part of the computation has been reduced?

ValueError: expected 2D or 3D input (got 4D input)

Owner

Andreas Veit

This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Two-Stream Adaptive Graph Convolutional Networks for Skeleton-Based Action Recognition in CVPR19

Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

Torchserve server using a YoloV5 model running on docker with GPU and static batch inference to perform production ready inference.

Monocular 3D pose estimation. OpenVINO. CPU inference or iGPU (OpenCL) inference.

Data-depth-inference - Data depth inference with python

Some code of the implements of Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network

Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

PyTorch implementation of "ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context" (INTERSPEECH 2020)

PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021.

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

Official PyTorch Implementation of Convolutional Hough Matching Networks, CVPR 2021 (oral)

A PyTorch implementation of "Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks" (KDD 2019).

Another pytorch implementation of FCN (Fully Convolutional Networks)

A PyTorch implementation for V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation

PyTorch Implementation of Fully Convolutional Networks. (Training code to reproduce the original result is available.)