CVPR2020 Counterfactual Samples Synthesizing for Robust VQA

Related tags

Deep Learning vqa debias counterfactual cvpr2020
Overview

CVPR2020 Counterfactual Samples Synthesizing for Robust VQA

This repo contains code for our paper "Counterfactual Samples Synthesizing for Robust Visual Question Answering" This repo contains code modified from here,many thanks!

Prerequisites

Make sure you are on a machine with a NVIDIA GPU and Python 2.7 with about 100 GB disk space.
h5py==2.10.0
pytorch==1.1.0
Click==7.0
numpy==1.16.5
tqdm==4.35.0

Data Setup

You can use

bash tools/download.sh

to download the data
and the rest of the data and trained model can be obtained from BaiduYun(passwd:3jot) or GoogleDrive unzip feature1.zip and feature2.zip and merge them into data/rcnn_feature/
use

bash tools/process.sh

to process the data

Training

Run

CUDA_VISIBLE_DEVICES=0 python main.py --dataset cpv2 --mode q_v_debias --debias learned_mixin --topq 1 --topv -1 --qvp 5 --output [] --seed 0

to train a model

Testing

Run

CUDA_VISIBLE_DEVICES=0 python eval.py --dataset cpv2 --debias learned_mixin --model_state []

to eval a model

Citation

If you find this code useful, please cite the following paper:

@inproceedings{chen2020counterfactual,
title={Counterfactual Samples Synthesizing for Robust Visual Question Answering},
author={Chen, Long and Yan, Xin and Xiao, Jun and Zhang, Hanwang and Pu, Shiliang and Zhuang, Yueting},
booktitle={CVPR},
year={2020}
}
Comments
  • Upload files to a different server

    Upload files to a different server

    Hi,

    I find your work exciting! can you please upload the features and utils folders to a different server? The current server is not accessible outside of China.

    opened by imri 2
  • Hintscore and type, no_type masks

    Hintscore and type, no_type masks

    Hello! Could you explain how did you generate the files corresponding to hintscores (e.g., test_cpv2_hintscore.json) and the type and no_type masks (e.g., cpv2_type_mask.json and cpv2_notype_mask.json)?

    opened by gri1 1
  • Cannot reproduce the best result

    Cannot reproduce the best result

    I followed all the Readme and use the default hyperparameters, which should give me the best results. However, I can only get about 52(yes/no:82, num:50 , other:31) on the test set, which is far lower than the reported results. Can you give me some advice on how to fix the gap? Thx!

    opened by NLOuYang 1
  • I cannot find train/val36_imgid2img.pkl file in /utils.

    I cannot find train/val36_imgid2img.pkl file in /utils.

    https://github.com/yanxinzju/CSS-VQA/blob/9c422f3e34e8bab2ee26af99ec4a0949f81d8548/dataset.py#L189

    maybe you can give me a link, so I can download this file(BaiduYun will be ok).

    thanks!

    opened by exquisitedice 1
  • No such file or directory: 'data/rcnn_feature/25.pth'

    No such file or directory: 'data/rcnn_feature/25.pth'

    Hi, thanks for your code! I followed the instruction in Readme.md and when i run the main.py to train the model, it reports the error : No such file or directory: 'data/rcnn_feature/25.pth'. I downloaded the feature from the google drive and it seems there is an missing image feature file. Just for confirm, i wonder the data in google drive is complete for experiments, right? thanks!

    opened by xuliwalker 0
  • import cPickle 在python3 环境有问题

    import cPickle 在python3 环境有问题

    我改成了import _pickle as cPickle 然后在数据预处理时，报错 vals = list(map(float, vals[1:])) feb077d81bf6621670285c5cb2f554b 但是在运行main 时又出现新的编码问题 utf-8 image 通过 fe=torch.load('data/rcnn_feature/'+str(img_id)+'.pth', encoding="ISO-8859-1")['image_feature'] classifier nn.Dropout(dropout, inplace=False) 又跑通了

    最大 57.98 与文中的最好值相差1个点，是正常现象吗？ image

    opened by alice-cool 0
  • Killed when training

    Killed when training

    When I tired to run CUDA_VISIBLE_DEVICES=0 python main.py --dataset cpv2 --mode q_v_debias --debias learned_mixin --topq 1 --topv -1 --qvp 5 --output [] --seed 0, it went error. @yanxinzju

    image
    opened by CaffreyR 1
  • About the parameter bias

    About the parameter bias

    Thank you very much for your excellent work. I would like to ask you about the parameter entry["bias"] when processing the dataset. What does it refer to and how did you get it? The entry["bias"] I got when I migrated is 0?

    opened by qinqinq9 0
Owner
null
GitHub
Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

PyTorch Implementation of Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers 1 Using Colab Please notic

Hila Chefer 489 Jan 7, 2023
Synthesizing Long-Term 3D Human Motion and Interaction in 3D in CVPR2021

Long-term-Motion-in-3D-Scenes This is an implementation of the CVPR'21 paper "Synthesizing Long-Term 3D Human Motion and Interaction in 3D". Please ch

Jiashun Wang 76 Dec 13, 2022
This is the research repository for Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition.

Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition This is the research repository for Vid2

Future Interfaces Group (CMU) 26 Dec 24, 2022
TAP: Text-Aware Pre-training for Text-VQA and Text-Caption, CVPR 2021 (Oral)

TAP: Text-Aware Pre-training TAP: Text-Aware Pre-training for Text-VQA and Text-Caption by Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Flo

Microsoft 61 Nov 14, 2022
An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge.

Bottom-Up and Top-Down Attention for Visual Question Answering An efficient PyTorch implementation of the winning entry of the 2017 VQA Challenge. The

Hengyuan Hu 731 Jan 3, 2023
Neural Module Network for VQA in Pytorch

Neural Module Network (NMN) for VQA in Pytorch Note: This is NOT an official repository for Neural Module Networks. NMN is a network that is assembled

Harsh Trivedi 111 Nov 24, 2022
Synthesizing and manipulating 2048x1024 images with conditional GANs

pix2pixHD Project | Youtube | Paper Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic image-to-image translatio

NVIDIA Corporation 6k Dec 27, 2022
Pytorch implementation of winner from VQA Chllange Workshop in CVPR'17

2017 VQA Challenge Winner (CVPR'17 Workshop) pytorch implementation of Tips and Tricks for Visual Question Answering: Learnings from the 2017 Challeng

Mark Dong 166 Dec 11, 2022
This is the official pytorch implementation for our ICCV 2021 paper "TRAR: Routing the Attention Spans in Transformers for Visual Question Answering" on VQA Task

?? ERASOR (RA-L'21 with ICRA Option) Official page of "ERASOR: Egocentric Ratio of Pseudo Occupancy-based Dynamic Object Removal for Static 3D Point C

Hyungtae Lim 225 Dec 29, 2022
FACIAL: Synthesizing Dynamic Talking Face With Implicit Attribute Learning. ICCV, 2021.

FACIAL: Synthesizing Dynamic Talking Face with Implicit Attribute Learning PyTorch implementation for the paper: FACIAL: Synthesizing Dynamic Talking

null 226 Jan 8, 2023
[SIGGRAPH Asia 2021] DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning.

DeepVecFont This is the homepage for "DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning". Yizhi Wang and Zhouhui Lian. WI

Yizhi Wang 17 Dec 22, 2022
[SIGGRAPH Asia 2021] DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning.

DeepVecFont This is the homepage for "DeepVecFont: Synthesizing High-quality Vector Fonts via Dual-modality Learning". Yizhi Wang and Zhouhui Lian. WI

Yizhi Wang 5 Oct 22, 2021
Attention-guided gan for synthesizing IR images

SI-AGAN Attention-guided gan for synthesizing IR images This repository contains the Tensorflow code for "Pedestrian Gender Recognition by Style Trans

null 1 Oct 25, 2021
[ICLR'21] Counterfactual Generative Networks

This repository contains the code for the ICLR 2021 paper "Counterfactual Generative Networks" by Axel Sauer and Andreas Geiger. If you want to take the CGN for a spin and generate counterfactual images, you can try out the Colab below.

null 88 Jan 2, 2023
[CVPR 2021] Released code for Counterfactual Zero-Shot and Open-Set Visual Recognition

Counterfactual Zero-Shot and Open-Set Visual Recognition This project provides implementations for our CVPR 2021 paper Counterfactual Zero-S

null 144 Dec 24, 2022
CausaLM: Causal Model Explanation Through Counterfactual Language Models

CausaLM: Causal Model Explanation Through Counterfactual Language Models Authors: Amir Feder, Nadav Oved, Uri Shalit, Roi Reichart Abstract: Understan

Amir Feder 39 Jul 10, 2022
CARLA: A Python Library to Benchmark Algorithmic Recourse and Counterfactual Explanation Algorithms

CARLA - Counterfactual And Recourse Library CARLA is a python library to benchmark counterfactual explanation and recourse models. It comes out-of-the

Carla Recourse 200 Dec 28, 2022
[ICCV 2021] Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

Counterfactual Attention Learning Created by Yongming Rao*, Guangyi Chen*, Jiwen Lu, Jie Zhou This repository contains PyTorch implementation for ICCV

Yongming Rao 90 Dec 31, 2022
The code for MM2021 paper "Multi-Level Counterfactual Contrast for Visual Commonsense Reasoning"

The Code for MM2021 paper "Multi-Level Counterfactual Contrast for Visual Commonsense Reasoning" Setting up and using the repo Get the dataset. Follow

null 4 Apr 20, 2022