Oriented Response Networks, in CVPR 2017

Overview

Oriented Response Networks

[Home] [Project] [Paper] [Supp] [Poster]

illustration

Torch Implementation

The torch branch contains:

  • the official torch implementation of ORN.
  • the MNIST-Variants demo.

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

  • Linux (tested on ubuntu 14.04LTS)
  • NVIDIA GPU + CUDA CuDNN (CPU mode and CUDA without CuDNN mode are also available but significantly slower)
  • Torch7

Getting started

You can setup everything via a single command wget -O - https://git.io/vHCMI | bash or do it manually in case something goes wrong:

  1. install the dependencies (required by the demo code):

  2. clone the torch branch:

    # git version must be greater than 1.9.10
    git clone https://github.com/ZhouYanzhao/ORN.git -b torch --single-branch ORN.torch
    cd ORN.torch
    export DIR=$(pwd)
  3. install ORN:

    cd $DIR/install
    # install the CPU/GPU/CuDNN version ORN.
    bash install.sh
  4. unzip the MNIST dataset:

    cd $DIR/demo/datasets
    unzip MNIST
  5. run the MNIST-Variants demo:

    cd $DIR/demo
    # you can modify the script to test different hyper-parameters
    bash ./scripts/Train_MNIST.sh

Trouble shooting

If you run into 'cudnn.find' not found, update Torch7 to the latest version via cd <TORCH_DIR> && bash ./update.sh then re-install everything.

More experiments

CIFAR 10/100

You can train the OR-WideResNet model (converted from WideResNet by simply replacing Conv layers with ORConv layers) on CIFAR dataset with WRN.

dataset=cifar10_original.t7 model=or-wrn widen_factor=4 depth=40 ./scripts/train_cifar.sh

With exactly the same settings, ORN-augmented WideResNet achieves state-of-the-art result while using significantly fewer parameters.

CIFAR

Network Params CIFAR-10 (ZCA) CIFAR-10 (mean/std) CIFAR-100 (ZCA) CIFAR-100 (mean/std)
DenseNet-100-12-dropout 7.0M - 4.10 - 20.20
DenseNet-190-40-dropout 25.6M - 3.46 - 17.18
WRN-40-4 8.9M 4.97 4.53 22.89 21.18
WRN-28-10-dropout 36.5M 4.17 3.89 20.50 18.85
WRN-40-10-dropout 55.8M - 3.80 - 18.3
ORN-40-4(1/2) 4.5M 4.13 3.43 21.24 18.82
ORN-28-10(1/2)-dropout 18.2M 3.52 2.98 19.22 16.15

Table.1 Test error (%) on CIFAR10/100 dataset with flip/translation augmentation)

ImageNet

ILSVRC2012

The effectiveness of ORN is further verified on large scale data. The OR-ResNet-18 model upgraded from ResNet-18 yields significant better performance when using similar parameters.

Network Params Top1-Error Top5-Error
ResNet-18 11.7M 30.614 10.98
OR-ResNet-18 11.4M 28.916 9.88

Table.2 Validation error (%) on ILSVRC-2012 dataset.

You can use facebook.resnet.torch to train the OR-ResNet-18 model from scratch or finetune it on your data by using the pre-trained weights.

-- To fill the model with the pre-trained weights:
model = require('or-resnet.lua')({tensorType='torch.CudaTensor', pretrained='or-resnet18_weights.t7'})

A more specific demo notebook of using the pre-trained OR-ResNet to classify images can be found here.

PyTorch Implementation

The pytorch branch contains:

  • the official pytorch implementation of ORN (alpha version supports 1x1/3x3 ARFs with 4/8 orientation channels only).
  • the MNIST-Variants demo.

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

  • Linux (tested on ubuntu 14.04LTS)
  • NVIDIA GPU + CUDA CuDNN (CPU mode and CUDA without CuDNN mode are also available but significantly slower)
  • PyTorch

Getting started

  1. install the dependencies (required by the demo code):

    • tqdm: pip install tqdm
    • pillow: pip install Pillow
  2. clone the pytorch branch:

    # git version must be greater than 1.9.10
    git clone https://github.com/ZhouYanzhao/ORN.git -b pytorch --single-branch ORN.pytorch
    cd ORN.pytorch
    export DIR=$(pwd)
  3. install ORN:

    cd $DIR/install
    bash install.sh
  4. run the MNIST-Variants demo:

    cd $DIR/demo
    # train ORN on MNIST-rot
    python main.py --use-arf
    # train baseline CNN
    python main.py

Caffe Implementation

The caffe branch contains:

  • the official caffe implementation of ORN (alpha version supports 1x1/3x3 ARFs with 4/8 orientation channels only).
  • the MNIST-Variants demo.

Please follow the instruction below to install it and run the experiment demo.

Prerequisites

  • Linux (tested on ubuntu 14.04LTS)
  • NVIDIA GPU + CUDA CuDNN (CPU mode and CUDA without CuDNN mode are also available but significantly slower)
  • Caffe

Getting started

  1. install the dependency (required by the demo code):

  2. clone the caffe branch:

    # git version must be greater than 1.9.10
    git clone https://github.com/ZhouYanzhao/ORN.git -b caffe --single-branch ORN.caffe
    cd ORN.caffe
    export DIR=$(pwd)
  3. install ORN:

    # modify Makefile.config first
    # compile ORN.caffe
    make clean && make -j"$(nproc)" all
  4. run the MNIST-Variants demo:

    cd $DIR/examples/mnist
    bash get_mnist.sh
    # train ORN & CNN on MNIST-rot
    bash train.sh

Note

Due to implementation differences,

  • upgrading Conv layers to ORConv layers can be done by adding an orn_param
  • num_output of ORConv layers should be multipied by nOrientation of ARFs

Example:

layer {
  type: "Convolution"
  name: "ORConv" bottom: "Data" top: "ORConv"
  # add this line to replace regular filters with ARFs
  orn_param {orientations: 8}
  param { lr_mult: 1 decay_mult: 2}
  convolution_param {
    # this means 10 ARF feature maps
    num_output: 80
    kernel_size: 3
    stride: 1
    pad: 0
    weight_filler { type: "msra"}
    bias_filler { type: "constant" value: 0}
  }
}

Check the MNIST demo prototxt (and its visualization) for more details.

Citation

If you use the code in your research, please cite:

@INPROCEEDINGS{Zhou2017ORN,
    author = {Zhou, Yanzhao and Ye, Qixiang and Qiu, Qiang and Jiao, Jianbin},
    title = {Oriented Response Networks},
    booktitle = {CVPR},
    year = {2017}
}
Comments
  • Is there any plan to release tensorflow impl. of ORN?

    Is there any plan to release tensorflow impl. of ORN?

    First, thank you for your great research and contribution.

    I really need tensorflow implementation of this repo. Is there any plan to release tensorflow impl.?

    If so, I will just wait your release. If not, I have the plan to implement it in long-term on my hands.

    opened by mikigom 3
  • Function in Pytorch and caffe support

    Function in Pytorch and caffe support

    This wok is very nice. And thank you very much for release the codes. My issue is that I can not find orpooling in pytorch version and oralign in caffe version. Will you plan to release the corresponding codes in future? @ZhouYanzhao

    opened by billqxg 2
  • ORN in detection framework

    ORN in detection framework

    First, thank you for this great contribution.

    In your CVPR 2017 poster at Application 5 section there is a comparison of Faster R-CNN detection (bounding box only) with an ORN detection (bounding box + orientation). Could your elaborate on how you used ORN in a detection framework (maybe you have a github repo) and what dataset was used.

    opened by zvadaszi 2
  •  when i input bash instal.sh, it shows error : orn/src/liborn_kernel.cu.o: 没有那个文件或目录.

    when i input bash instal.sh, it shows error : orn/src/liborn_kernel.cu.o: 没有那个文件或目录.

    Hello,ZhouYanzhao.
    when i input bash instal.sh, it shows error : orn/src/liborn_kernel.cu.o: 没有那个文件或目录. Can you help me? Thank you!

    The instructions:

    bash instal.sh ... ... ... build/temp.linux-x86_64-3.5/home/yangyu/ORN/ORN.pytorch/install/orn/src/liborn_cuda.o orn/src/liborn_kernel.cu.o -o build/lib.linux-x86_64-3.5/orn/_ext/liborn/_liborn.abi3.so x86_64-linux-gnu-gcc: error: orn/src/liborn_kernel.cu.o: 没有那个文件或目录

    opened by yuy01 1
  • PyTorch Using filters with kernel_size=1 causes exception.

    PyTorch Using filters with kernel_size=1 causes exception.

    Exception: 'int' object does not support indexing, for line kernel = kernel_indices[kW][angle][j]

    Wrapping the kernel_indices for size 1 as a list fixes the issue. New kernel_indices:

        kernel_indices = {
            1: {
                0: [1],
                45: [1],
                90: [1],
                135: [1],
                180: [1],
                225: [1],
                270: [1],
                315: [1]
            },
            3: {
                0: (1,2,3,4,5,6,7,8,9),
                45: (2,3,6,1,5,9,4,7,8),
                90: (3,6,9,2,5,8,1,4,7),
                135: (6,9,8,3,5,7,2,1,4),
                180: (9,8,7,6,5,4,3,2,1),
                225: (8,7,4,9,5,1,6,3,2),
                270: (7,4,1,8,5,2,9,6,3),
                315: (4,1,2,7,5,3,8,9,6)
            }
        }
    
    opened by utkarsh-VRL 1
  • default train_mnist.sh can't get the desired result.

    default train_mnist.sh can't get the desired result.

    note [Rot_ORN_8_Align] | epoch 197 | elapse 17.76 | remain  0: 0:53 | testLoss 0.046485 | testAcc 98.7000 | trainLoss 0.039256 | trainAcc 98.8201
    note [Rot_ORN_8_Align] | epoch 198 | elapse 17.78 | remain  0: 0:35 | testLoss 0.056883 | testAcc 98.5600 | trainLoss 0.038547 | trainAcc 98.8722
    note [Rot_ORN_8_Align] | epoch 199 | elapse 17.81 | remain  0: 0:17 | testLoss 0.068868 | testAcc 98.3600 | trainLoss 0.038445 | trainAcc 98.8421
    note [Rot_ORN_8_Align] | epoch 200 | elapse 17.82 | remain  0: 0: 0 | testLoss 0.046446 | testAcc 98.6600 | trainLoss 0.041818 | trainAcc 98.7800
    | Thu Jul 20 15:49:03 2017 | best model: /data/Repo/ORN.torch/demo/logs/Rot_ORN_8_Align/192-99.04-20170720154638.t7
    {{MSG:| Thu Jul 20 15:49:03 2017 | [Rot_ORN_8_Align] all done. bestValidationAcc: 99.04, finalTestAcc: 99.02}}
    

    Not as good as the paper presented.

    opened by samson-wang 1
  • PyTorch/Caffe Implementation (alpha version)

    PyTorch/Caffe Implementation (alpha version)

    • [x] added PyTorch implementation (including MNIST-Variants demo)
    • [x] added Caffe implementation (including MNIST-Variants demo)
    • [x] added the Poster for CVPR2017
    • [x] added CIFAR performance of DenseNet to Table.1 for comparison
    opened by ZhouYanzhao 0
  • More results of CIFAR and ImageNet

    More results of CIFAR and ImageNet

    1. added more results of mean/std normalized CIFAR10/100.
    2. added the performance of OR-ResNet-18 on ILSVRC2012 dataset.
    3. released the imagenet pre-trained weights of OR-ResNet-18 model.
    4. added a demo notebook of using imagenet pre-trained ORN to classify images.
    opened by ZhouYanzhao 0
  • Torch Implementation

    Torch Implementation

    Here is the official Torch implementation of ORN, which provides:

    • [x] CPU/GPU/CuDNN versions
    • [x] MNIST-Variants demo
    • [x] OR-WideResNet model for CIFAR

    Caffe and pyTorch implementation will come soon.

    opened by ZhouYanzhao 0
  • about Norientation channel and Nrotated versions

    about Norientation channel and Nrotated versions

    Dear author, I have read your paper recently, but I do not quite understand some details in it.It is ,why Norientation channels when had eight rotational versions.Looking forward to your answer.Thank you.

    opened by BooKing-hub 0
  • how to make the feature map visualization?

    how to make the feature map visualization?

    Sincerely Zhou, I appreciate your great work on oriented feature. Now I have a question that how you make the feature map visualization. so would you like to give me some advice?

    opened by cdshdow 0
  • Upgrade the pytorch branch to 1.0, but fails while grad checking

    Upgrade the pytorch branch to 1.0, but fails while grad checking

    Thanks for sharing!

    Recently, I upgrade the pytorch branch to be compatible with pytorch 1.0. When I conduct the grad check for activerotaingfilter, I failed. But the rotationinvariantencoding success. I have checked my codes many times, but still don't solve it. Do you plan to upgrade this branch to python1.0?

    opened by ayumiymk 1
  • Question about using or-resnet for original --> rot experiments

    Question about using or-resnet for original --> rot experiments

    I use the network defined in the demo.py, it is ok to reproduce the similar results for original --> rot experiments on mnist. But when I changed to or-resnet, it got very low results, the accuracy is no more than 50%(even lower than the simple OR-CNN defined in demo.py ). I do not know what's wrong with it. The following is my code. https://paste.ubuntu.com/p/NssKJbxDqZ/ What's more, when I do original --> rot experiments on cifar-10. I use the network defined in the demo.py. And got results of 35.26 and 44.97 for without/with orn respectively. Look forward to your reply.

    opened by dingjiansw101 3
  • Error in liborn for pytorch branch

    Error in liborn for pytorch branch

    At first I ran

    python3 setup.py install 
    

    which created build, dist and orn.egg-info folders. Inside the build folder there was the orn folder containing _ext which has liborn. But init.py file inside the liborn folder gives reference error on the line :

    from ._liborn import lib as _lib, ffi as _ffi
    
    opened by nayeemiiitd 1
Fader Networks: Manipulating Images by Sliding Attributes - NIPS 2017

FaderNetworks PyTorch implementation of Fader Networks (NIPS 2017). Fader Networks can generate different realistic versions of images by modifying at

Facebook Research 753 Dec 23, 2022
PyTorch version of the paper 'Enhanced Deep Residual Networks for Single Image Super-Resolution' (CVPRW 2017)

About PyTorch 1.2.0 Now the master branch supports PyTorch 1.2.0 by default. Due to the serious version problem (especially torch.utils.data.dataloade

Sanghyun Son 2.1k Jan 1, 2023
Improving Convolutional Networks via Attention Transfer (ICLR 2017)

Attention Transfer PyTorch code for "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Tran

Sergey Zagoruyko 1.4k Dec 23, 2022
The PyTorch improved version of TPAMI 2017 paper: Face Alignment in Full Pose Range: A 3D Total Solution.

Face Alignment in Full Pose Range: A 3D Total Solution By Jianzhu Guo. [Updates] 2020.8.30: The pre-trained model and code of ECCV-20 are made public

Jianzhu Guo 3.4k Jan 2, 2023
An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge.

Bottom-Up and Top-Down Attention for Visual Question Answering An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge. The

Hengyuan Hu 731 Jan 3, 2023
PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules

Dynamic Routing Between Capsules - PyTorch implementation PyTorch implementation of NIPS 2017 paper Dynamic Routing Between Capsules from Sara Sabour,

Adam Bielski 475 Dec 24, 2022
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Karan Desai 105 Nov 25, 2022
A PyTorch implementation of the paper "Semantic Image Synthesis via Adversarial Learning" in ICCV 2017

Semantic Image Synthesis via Adversarial Learning This is a PyTorch implementation of the paper Semantic Image Synthesis via Adversarial Learning. Req

Seonghyeon Nam 146 Nov 25, 2022
Implementation supporting the ICCV 2017 paper "GANs for Biological Image Synthesis"

GANs for Biological Image Synthesis This codes implements the ICCV-2017 paper "GANs for Biological Image Synthesis". The paper and its supplementary m

Anton Osokin 95 Nov 25, 2022
Code for the USENIX 2017 paper: kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels

kAFL: Hardware-Assisted Feedback Fuzzing for OS Kernels Blazing fast x86-64 VM kernel fuzzing framework with performant VM reloads for Linux, MacOS an

Chair for Sys­tems Se­cu­ri­ty 541 Nov 27, 2022
Implementation of EMNLP 2017 Paper "Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog" using PyTorch and ParlAI

Language Emergence in Multi Agent Dialog Code for the Paper Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog Satwik Kottur, José M.

Karan Desai 105 Nov 25, 2022
meProp: Sparsified Back Propagation for Accelerated Deep Learning (ICML 2017)

meProp The codes were used for the paper meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (ICML 2017) [pdf]

LancoPKU 107 Nov 18, 2022
A Research-oriented Federated Learning Library and Benchmark Platform for Graph Neural Networks. Accepted to ICLR'2021 - DPML and MLSys'21 - GNNSys workshops.

FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks A Research-oriented Federated Learning Library and Benchmark Platform

FedML-AI 175 Dec 1, 2022
On Size-Oriented Long-Tailed Graph Classification of Graph Neural Networks

On Size-Oriented Long-Tailed Graph Classification of Graph Neural Networks We provide the code (in PyTorch) and datasets for our paper "On Size-Orient

Zemin Liu 4 Jun 18, 2022
Source Code for DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances (https://arxiv.org/pdf/2012.01775.pdf)

DialogBERT This is a PyTorch implementation of the DialogBERT model described in DialogBERT: Neural Response Generation via Hierarchical BERT with Dis

Xiaodong Gu 67 Jan 6, 2023
《Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis》(2021)

Image2Reverb Image2Reverb is an end-to-end neural network that generates plausible audio impulse responses from single images of acoustic environments

Nikhil Singh 48 Nov 27, 2022
Cancer Drug Response Prediction via a Hybrid Graph Convolutional Network

DeepCDR Cancer Drug Response Prediction via a Hybrid Graph Convolutional Network This work has been accepted to ECCB2020 and was also published in the

Qiao Liu 50 Dec 18, 2022
FAST-RIR: FAST NEURAL DIFFUSE ROOM IMPULSE RESPONSE GENERATOR

This is the official implementation of our neural-network-based fast diffuse room impulse response generator (FAST-RIR) for generating room impulse responses (RIRs) for a given acoustic environment.

Anton Jeran Ratnarajah 89 Dec 22, 2022
Optimal Adaptive Allocation using Deep Reinforcement Learning in a Dose-Response Study

Optimal Adaptive Allocation using Deep Reinforcement Learning in a Dose-Response Study Supplementary Materials for Kentaro Matsuura, Junya Honda, Imad

Kentaro Matsuura 4 Nov 1, 2022