Using / reproducing ACD from the paper "Hierarchical interpretations for neural network predictions" 🧠 (ICLR 2019)

Last update: Jan 3, 2023

Overview

Hierarchical neural-net interpretations (ACD) 🧠

Produces hierarchical interpretations for a single prediction made by a pytorch neural network. Official code for Hierarchical interpretations for neural network predictions (ICLR 2019 pdf).

Documentation • Demo notebooks

Note: this repo is actively maintained. For any questions please file an issue.

examples/documentation

installation: pip install acd (or clone and run python setup.py install)
examples: the reproduce_figs folder has notebooks with many demos
src: the acd folder contains the source for the method implementation
allows for different types of interpretations by changing hyperparameters (explained in examples)
all required data/models/code for reproducing are included in the dsets folder

Inspecting NLP sentiment models	Detecting adversarial examples	Analyzing imagenet models

notes on using ACD on your own data

the current CD implementation often works out-of-the box, especially for networks built on common layers, such as alexnet/vgg/resnet. However, if you have custom layers or layers not accessible in net.modules(), you may need to write a custom function to iterate through some layers of your network (for examples see cd.py).
to use baselines such build-up and occlusion, replace the pred_ims function by a function, which gets predictions from your model given a batch of examples.

related work

CDEP (ICML 2020 pdf, github) - penalizes CD / ACD scores during training to make models generalize better
TRIM (ICLR 2020 workshop pdf, github) - using simple reparameterizations, allows for calculating disentangled importances to transformations of the input (e.g. assigning importances to different frequencies)
PDR framework (PNAS 2019 pdf) - an overarching framewwork for guiding and framing interpretable machine learning
DAC (arXiv 2019 pdf, github) - finds disentangled interpretations for random forests
Baseline interpretability methods - the file scores/score_funcs.py also contains simple pytorch implementations of integrated gradients and the simple interpration technique gradient * input

reference

feel free to use/share this code openly
if you find this code useful for your research, please cite the following:

@inproceedings{
   singh2019hierarchical,
   title={Hierarchical interpretations for neural network predictions},
   author={Chandan Singh and W. James Murdoch and Bin Yu},
   booktitle={International Conference on Learning Representations},
   year={2019},
   url={https://openreview.net/forum?id=SkEqro0ctQ},
}

Comments

What is the meaning of "CD(x, candidateGroup;model)-CD(x,selectedGroup; model)"?

Hi, @csinva

Referring to the CD paper, we have p=Softmax(Wβ+Wγ), and Wβ provides a quantitative score for the given phrase. But for different phrases, the Wβ terms may have very different values while the softmax results remain similar, is it reasonable to do subtraction between two Wβ terms?

Thanks!

opened by LMdeLiangMi 5
How unimportant features/words are being handled?

Hi @csinva, in linear regression or logistic regression we do calculate the standard error or do t-test to identify whether that variable is significant or not. Let me mathematically formulate my question here, assume I am doing one linear regression with two variables (variables can be words also), so the predicted equation will be,

y_hat = b0_hat + b1_hat * x1 + b2_hat * x2

and let's say that b1_hat = 100 and b2_hat = 2. We may think that as b1_hat is larger than b2_hat, then it has more positive effect on output, but it may happen that in spite of high value, x1 may come insignificant by doing t-test of b1_hat because of it's high standard error.

What I understood reading your paper that the ACD/CD score is somewhat similar to the b (or beta or coefficients) values of the above linear (or logistic) equation. Now in spite of very higher (or lower) CD/ACD score of a word or phrase, it may happen that the specific word/phrase is not important/insignificant to the model. Could you please tell me how your method is handling this scenario?
question

opened by pidahbus 3
Update reproduceable figs

Hi. Is there any chance this code is going to be updated to newer PyTorch versions? E.g 1.6. Current code throws a couple of exceptions both in reproduce figs and train.py for LSTM.

opened by neeps 1
Explanation of this method and features on Regression Problem

Hi,

Assume I want to apply this method on a NLP regression problem: Prediction of a score given some answer scripts. In this scenario, let's assume that this phrase: "it was very good" has a very positive CD score. Does it mean that the inclusion of the above phrase in the answer increases the predicted score and vice versa?

Can the similar kind analogy be drawn for a phrase having negative CD score i.e. inclusion of that phrase will decrease the score of the answer.
question

opened by pidahbus 1
cd_text 'stop' is mixed inclusive/exclusive?
while going through the code, I noticed on line 90 of scores.py:

if i >= start and i <= stop:

That is, the set of relevant tokens is inclusive of the token at stop. But on line 108, adding the bias activations to the relevant component:

if i >= start and i < stop:

So the bias is treated differently, with stop being exclusive for the bias but inclusive for the propagated inputs. Why is this? I didn't see anything in the paper that indicated that the propagated bias and input are treated differently.
opened by nsaphra 1
Implementation on multi class text classification problem

Hi , Great work Guys !!! Paper was describing more on the sentiment analysis which is a binary classification. I was trying to use the same for multi class classification for a nlp use case as one vs rest of other classes to get the CD scores. But I am not sure if there is any other way to do for the multi class. Would really appreciate any help.

opened by mahiidharv 0
CD computation for Tree LSTMs

I am trying to get words and phrase level attribution values for TreeLSTMs (dependency) trained on SST dataset. But not sure what the correct CD computation equations/code would be.
Would really appreciate any help!
enhancement

opened by sopankhosla 1

Owner

Chandan Singh

Working on interpretable machine learning across domains 🧠⚕️🦠 Let's do good with models.

GitHub https://arxiv.org/abs/1806.05337

L2X - Code for replicating the experiments in the paper Learning to Explain: An Information-Theoretic Perspective on Model Interpretation.

L2X Code for replicating the experiments in the paper Learning to Explain: An Information-Theoretic Perspective on Model Interpretation at ICML 2018,

113 Sep 6, 2022

Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Image Crop Analysis This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post. If you plan to use this

239 Jan 2, 2023

This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Deep Continuous Clustering Introduction This is a Pytorch implementation of the DCC algorithms presented in the following paper (paper): Sohil Atul Sh

197 Nov 29, 2022

Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

StackGAN-v2 StackGAN-v1: Tensorflow implementation StackGAN-v1: Pytorch implementation Inception score evaluation Pytorch implementation for reproduci

809 Dec 16, 2022

PyTorch framework, for reproducing experiments from the paper Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks. Code, based on the PyTorch framework, for reprodu

3 Dec 27, 2022

A PyTorch implementation of "Capsule Graph Neural Network" (ICLR 2019).

CapsGNN ⠀⠀ A PyTorch implementation of Capsule Graph Neural Network (ICLR 2019). Abstract The high-quality node embeddings learned from the Graph Neur

1.2k Jan 2, 2023

A PyTorch implementation of "Predict then Propagate: Graph Neural Networks meet Personalized PageRank" (ICLR 2019).

APPNP ⠀ A PyTorch implementation of Predict then Propagate: Graph Neural Networks meet Personalized PageRank (ICLR 2019). Abstract Neural message pass

329 Dec 30, 2022

A PyTorch implementation of "Graph Wavelet Neural Network" (ICLR 2019)

Graph Wavelet Neural Network ⠀⠀ A PyTorch implementation of Graph Wavelet Neural Network (ICLR 2019). Abstract We present graph wavelet neural network

490 Dec 16, 2022

Deep Anomaly Detection with Outlier Exposure (ICLR 2019)

Outlier Exposure This repository contains the essential code for the paper Deep Anomaly Detection with Outlier Exposure (ICLR 2019). Requires Python 3

464 Dec 27, 2022

Repository for reproducing `Model-Based Robust Deep Learning`

Model-Based Robust Deep Learning (MBRDL) In this repository, we include the code necessary for reproducing the code used in Model-Based Robust Deep Le

16 Sep 19, 2022

PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

PySlowFast PySlowFast is an open source video understanding codebase from FAIR that provides state-of-the-art video classification models with efficie

5.3k Jan 3, 2023

Reproducing code of hair style replacement method from Barbershorp.

Barbershorp Reproducing code of hair style replacement method from Barbershorp. Also reproduces II2S, an improved version of Image2StyleGAN. Requireme

1 Dec 24, 2021

This repository contains unofficial code reproducing Agent57

Agent57 This repository contains unofficial code reproducing Agent57, which outp

19 Dec 29, 2022

This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivariant Continuous Convolution

Trajectory Prediction using Equivariant Continuous Convolution (ECCO) This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivar

45 Jul 22, 2022

This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures

Introduction This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures. @inproceedings{Wa

42 Jan 7, 2023

Code for the CIKM 2019 paper "DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting".

Dual Self-Attention Network for Multivariate Time Series Forecasting 20.10.26 Update: Due to the difficulty of installation and code maintenance cause

223 Dec 16, 2022

Pre-trained model, code, and materials from the paper "Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation" (MICCAI 2019).

Adaptive Segmentation Mask Attack This repository contains the implementation of the Adaptive Segmentation Mask Attack (ASMA), a targeted adversarial

53 Jul 4, 2022