A Pytorch Implementation for Compact Bilinear Pooling.

Overview

CompactBilinearPooling-Pytorch

A Pytorch Implementation for Compact Bilinear Pooling. Adapted from tensorflow_compact_bilinear_pooling

Prerequisites

Install pytorch_fft by

pip install pytorch_fft

Usage

from torch import nn
from torch.autograd import Variable
from CompactBilinearPooling import CompactBilinearPooling

bottom1 = Variable(torch.randn(128, 512, 14, 14)).cuda()
bottom2 = Variable(torch.randn(128, 512, 14, 14)).cuda()

layer = CompactBilinearPooling(512, 512, 8000)
layer.cuda()
layer.train()

out = layer(bottom1, bottom2)

Reference

Yang Gao, et al. "Compact Bilinear Pooling." in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (2016).
Akira Fukui, et al. "Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding." arXiv preprint arXiv:1606.01847 (2016).
Comments
  • Complex product should be used in eltwise product

    Complex product should be used in eltwise product

    I was investigating differences in results of this package and of the original Caffe version. Things found so far:

    1. Rfft can be used directly, it is provided by pytorch_fft
    2. Complex product should be used here: https://github.com/DeepInsight-PCALab/CompactBilinearPooling-Pytorch/blob/master/CompactBilinearPooling.py#L91, the tf.multiply does complex product, original caffe version does complex product
    3. Output should be mutliplied by output_dim in order to achieve full equivalence with original caffe version
    opened by vadimkantorov 6
  • No module named _th_fft

    No module named _th_fft

    Hi, thanks for your job! lt looks like there is no module named _th_fft when follow the command as you said. Do you have any idea about it? Thanks a lot~ ` dl@dl:~/wxptest$ sudo python wxp.py

    Traceback (most recent call last): File "wxp.py", line 3, in from CompactBilinearPooling import CompactBilinearPooling File "/home/dl/wxptest/CompactBilinearPooling.py", line 6, in import pytorch_fft.fft.autograd as afft File "/home/dl/wxptest/pytorch_fft/init.py", line 1, in from . import fft File "/home/dl/wxptest/pytorch_fft/fft/init.py", line 1, in from .fft import * File "/home/dl/wxptest/pytorch_fft/fft/fft.py", line 3, in from .._ext import th_fft File "/home/dl/wxptest/pytorch_fft/_ext/th_fft/init.py", line 3, in from ._th_fft import lib as _lib, ffi as _ffi ImportError: No module named _th_fft `

    opened by kopingwu 4
  • pytorch_fft install

    pytorch_fft install

    when I install pytorch_fft,the problem is: error in pytorch_fft setup command: 'C:\Users\14375\AppData\Local\Temp\pip-install-6vh12s8w\pytorch-fft\build.py:ffi' must be of the form 'path/build.py:ffi_va riable' Including CUDA code. ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

    opened by 1437539743 3
  • Possible error in computation

    Possible error in computation

    According to the existing code beginning here https://github.com/DeepInsight-PCALab/CompactBilinearPooling-Pytorch/blob/56e23d01e687dad067dfeb9a07bb4d012430e1df/CompactBilinearPooling.py#L92 the product of two complex numbers(a+ib, x+iy) is {(ax-by) + i(ax+by)}.

       `temp_rr, temp_ii = fft1_real.mul(fft2_real), fft1_imag.mul(fft2_imag)
        fft_product_real = temp_rr - temp_ii
        fft_product_imag = temp_rr + temp_ii
    
        cbp_flat = afft.Ifft()(fft_product_real, fft_product_imag)[0]`
    

    However the correct product is {(ax-by) + i(ay+bx)} in which case the following product is wrong.

    Correct me if I am wrong.

    opened by dormantrepo 1
  • NotImplementedError in fft

    NotImplementedError in fft

    I am getting this particular error while running sample test given in readme.

      File "CompactBilinearPooling/test.py", line 14, in <module>
        out = layer(bottom1, bottom2) 
      File "anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
        result = self.forward(*input, **kwargs)
      File "CompactBilinearPooling/CompactBilinearPooling.py", line 91, in forward
        fft1_real, fft1_imag = afft.Fft()(sketch_1, Variable(torch.zeros(sketch_1.size())).cuda())
      File "anaconda3/lib/python3.6/site-packages/pytorch_fft/fft/autograd.py", line 16, in forward
        return fft(X_re, X_im)
      File "anaconda3/lib/python3.6/site-packages/pytorch_fft/fft/fft.py", line 25, in fft
        raise NotImplementedError
    NotImplementedError
    

    Now this is due to this function fft.py:

        if 'Float' in type(X_re).__name__ :
            f = th_fft.th_Float_fft1
        elif 'Double' in type(X_re).__name__: 
            f = th_fft.th_Double_fft1
        else: 
            raise NotImplementedError
        return _fft(X_re, X_im, f, 1)
    

    Because inputs to fft() are tensors, type(sketch_1) does not include 'Float' or 'Double'. Any help is appreciated.

    opened by gullalc 1
  • No parameters for optmizer

    No parameters for optmizer

    Thanks for your working, when I apply your code, I meet the situation that model.parameters() is empty and fail to creat optimizer. Model is compactbilinearpooling(1536,1536,400). Am I wrong? Thanks for answering

    opened by bupt-zsp 1
  • How's the output

    How's the output

    Thank you for your job! I have some question, if x = (4,512,64,64), y = (4,512,64,64), a=CompactBilinearPooling(x,y) , what the shape of a ? And How I can make the a‘s shape is (batch_size,channels,height,width) which the same as x and y?

    opened by Paranoidv 1
  • In the code seem to be different from the setting of the paper.

    In the code seem to be different from the setting of the paper.

    rand_h_1 = np.random.randint(output_dim, size=self.input_dim1) and rand_s_1 = 2 * np.random.randint(2, size=self.input_dim1) - 1 in the code seem to be different from the setting of the paper. In the paper, hk is {1,2,...,k}, sk is {+1 ,-1}.

    opened by roseif 0
  • multi GPUs error: RuntimeError: arguments are located on different GPUs

    multi GPUs error: RuntimeError: arguments are located on different GPUs

    For multi GPU, it outputs:

    RuntimeError: arguments are located on different GPUs at /pytorch/torch/lib/THC/generic/THCTensorMathBlas.cu:236
    

    How to fix it?

    opened by JingyunLiang 3
Owner
null
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Bilinear attention networks for visual question answering

Bilinear Attention Networks This repository is the implementation of Bilinear Attention Networks for the visual question answering and Flickr30k Entit

Jin-Hwa Kim 506 Nov 29, 2022
This repository is an open-source implementation of the ICRA 2021 paper: Locus: LiDAR-based Place Recognition using Spatiotemporal Higher-Order Pooling.

Locus This repository is an open-source implementation of the ICRA 2021 paper: Locus: LiDAR-based Place Recognition using Spatiotemporal Higher-Order

Robotics and Autonomous Systems Group 96 Dec 15, 2022
Code for "Learning the Best Pooling Strategy for Visual Semantic Embedding", CVPR 2021

Learning the Best Pooling Strategy for Visual Semantic Embedding Official PyTorch implementation of the paper Learning the Best Pooling Strategy for V

Jiacheng Chen 106 Jan 6, 2023
Source code for paper "Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling", AAAI 2021

ATLOP Code for AAAI 2021 paper Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling. If you make use of this co

Wenxuan Zhou 146 Nov 29, 2022
Code for Understanding Pooling in Graph Neural Networks

Select, Reduce, Connect This repository contains the code used for the experiments of: "Understanding Pooling in Graph Neural Networks" Setup Install

Daniele Grattarola 37 Dec 13, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
Contrastive Learning for Compact Single Image Dehazing, CVPR2021

AECR-Net Contrastive Learning for Compact Single Image Dehazing, CVPR2021. Official Pytorch based implementation. Paper arxiv Pytorch Version TODO: mo

glassy 253 Jan 1, 2023
ALBERT-pytorch-implementation - ALBERT pytorch implementation

ALBERT-pytorch-implementation developing... 모델의 개념이해를 돕기 위한 구현물로 현재 변수명을 상세히 적었고

BG Kim 3 Oct 6, 2022
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
RealFormer-Pytorch Implementation of RealFormer using pytorch

RealFormer-Pytorch Implementation of RealFormer using pytorch. Includes comparison with classical Transformer on image classification task (ViT) wrt C

Simo Ryu 90 Dec 8, 2022
A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

Mixup: Beyond Empirical Risk Minimization in PyTorch This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The co

Harry Yang 121 Dec 17, 2022
A pytorch implementation of Pytorch-Sketch-RNN

Pytorch-Sketch-RNN A pytorch implementation of https://arxiv.org/abs/1704.03477 In order to draw other things than cats, you will find more drawing da

Alexis David Jacq 172 Dec 12, 2022
PyTorch implementation of Advantage async actor-critic Algorithms (A3C) in PyTorch

Advantage async actor-critic Algorithms (A3C) in PyTorch @inproceedings{mnih2016asynchronous, title={Asynchronous methods for deep reinforcement lea

LEI TAI 111 Dec 8, 2022
Pytorch-diffusion - A basic PyTorch implementation of 'Denoising Diffusion Probabilistic Models'

PyTorch implementation of 'Denoising Diffusion Probabilistic Models' This reposi

Arthur Juliani 76 Jan 7, 2023
Fang Zhonghao 13 Nov 19, 2022
RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

RETRO - Pytorch (wip) Implementation of RETRO, Deepmind's Retrieval based Attent

Phil Wang 556 Jan 4, 2023
HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

HashNeRF-pytorch Instant-NGP recently introduced a Multi-resolution Hash Encodin

Yash Sanjay Bhalgat 616 Jan 6, 2023
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022