Using / reproducing ACD from the paper "Hierarchical interpretations for neural network predictions" 🧠 (ICLR 2019)

Related tags

Deep Learning Model Explanation python data-science machine-learning statistics deep-neural-networks ai deep-learning neural-network jupyter-notebook ml pytorch artificial-intelligence convolutional-neural-networks acd interpretation iclr interpretability feature-importance explainable-ai explainability
Overview

Hierarchical neural-net interpretations (ACD) 🧠

Produces hierarchical interpretations for a single prediction made by a pytorch neural network. Official code for Hierarchical interpretations for neural network predictions (ICLR 2019 pdf).

Documentation • Demo notebooks

Note: this repo is actively maintained. For any questions please file an issue.

examples/documentation

  • installation: pip install acd (or clone and run python setup.py install)
  • examples: the reproduce_figs folder has notebooks with many demos
  • src: the acd folder contains the source for the method implementation
  • allows for different types of interpretations by changing hyperparameters (explained in examples)
  • all required data/models/code for reproducing are included in the dsets folder
Inspecting NLP sentiment models Detecting adversarial examples Analyzing imagenet models

notes on using ACD on your own data

  • the current CD implementation often works out-of-the box, especially for networks built on common layers, such as alexnet/vgg/resnet. However, if you have custom layers or layers not accessible in net.modules(), you may need to write a custom function to iterate through some layers of your network (for examples see cd.py).
  • to use baselines such build-up and occlusion, replace the pred_ims function by a function, which gets predictions from your model given a batch of examples.

related work

  • CDEP (ICML 2020 pdf, github) - penalizes CD / ACD scores during training to make models generalize better
  • TRIM (ICLR 2020 workshop pdf, github) - using simple reparameterizations, allows for calculating disentangled importances to transformations of the input (e.g. assigning importances to different frequencies)
  • PDR framework (PNAS 2019 pdf) - an overarching framewwork for guiding and framing interpretable machine learning
  • DAC (arXiv 2019 pdf, github) - finds disentangled interpretations for random forests
  • Baseline interpretability methods - the file scores/score_funcs.py also contains simple pytorch implementations of integrated gradients and the simple interpration technique gradient * input

reference

  • feel free to use/share this code openly
  • if you find this code useful for your research, please cite the following:
@inproceedings{
   singh2019hierarchical,
   title={Hierarchical interpretations for neural network predictions},
   author={Chandan Singh and W. James Murdoch and Bin Yu},
   booktitle={International Conference on Learning Representations},
   year={2019},
   url={https://openreview.net/forum?id=SkEqro0ctQ},
}
Comments
  • What is the meaning of

    What is the meaning of "CD(x, candidateGroup;model)-CD(x,selectedGroup; model)"?

    Hi, @csinva

    Referring to the CD paper, we have p=Softmax(Wβ+Wγ), and Wβ provides a quantitative score for the given phrase. But for different phrases, the Wβ terms may have very different values while the softmax results remain similar, is it reasonable to do subtraction between two Wβ terms?

    Thanks!

    opened by LMdeLiangMi 5
  • How unimportant features/words are being handled?

    How unimportant features/words are being handled?

    Hi @csinva, in linear regression or logistic regression we do calculate the standard error or do t-test to identify whether that variable is significant or not. Let me mathematically formulate my question here, assume I am doing one linear regression with two variables (variables can be words also), so the predicted equation will be,

    y_hat = b0_hat + b1_hat * x1 + b2_hat * x2

    and let's say that b1_hat = 100 and b2_hat = 2. We may think that as b1_hat is larger than b2_hat, then it has more positive effect on output, but it may happen that in spite of high value, x1 may come insignificant by doing t-test of b1_hat because of it's high standard error.

    What I understood reading your paper that the ACD/CD score is somewhat similar to the b (or beta or coefficients) values of the above linear (or logistic) equation. Now in spite of very higher (or lower) CD/ACD score of a word or phrase, it may happen that the specific word/phrase is not important/insignificant to the model. Could you please tell me how your method is handling this scenario?

    question 
    opened by pidahbus 3
  • Update reproduceable figs

    Update reproduceable figs

    Hi. Is there any chance this code is going to be updated to newer PyTorch versions? E.g 1.6. Current code throws a couple of exceptions both in reproduce figs and train.py for LSTM.

    opened by neeps 1
  • Explanation of this method and features on Regression Problem

    Explanation of this method and features on Regression Problem

    Hi,

    Assume I want to apply this method on a NLP regression problem: Prediction of a score given some answer scripts. In this scenario, let's assume that this phrase: "it was very good" has a very positive CD score. Does it mean that the inclusion of the above phrase in the answer increases the predicted score and vice versa?

    Can the similar kind analogy be drawn for a phrase having negative CD score i.e. inclusion of that phrase will decrease the score of the answer.

    question 
    opened by pidahbus 1
  • cd_text 'stop' is mixed inclusive/exclusive?

    cd_text 'stop' is mixed inclusive/exclusive?

    while going through the code, I noticed on line 90 of scores.py:

    if i >= start and i <= stop:

    That is, the set of relevant tokens is inclusive of the token at stop. But on line 108, adding the bias activations to the relevant component:

    if i >= start and i < stop:

    So the bias is treated differently, with stop being exclusive for the bias but inclusive for the propagated inputs. Why is this? I didn't see anything in the paper that indicated that the propagated bias and input are treated differently.

    opened by nsaphra 1
  • Implementation on multi class text classification problem

    Implementation on multi class text classification problem

    Hi , Great work Guys !!! Paper was describing more on the sentiment analysis which is a binary classification. I was trying to use the same for multi class classification for a nlp use case as one vs rest of other classes to get the CD scores. But I am not sure if there is any other way to do for the multi class. Would really appreciate any help.

    opened by mahiidharv 0
  • CD computation for Tree LSTMs

    CD computation for Tree LSTMs

    I am trying to get words and phrase level attribution values for TreeLSTMs (dependency) trained on SST dataset. But not sure what the correct CD computation equations/code would be.
    Would really appreciate any help!

    enhancement 
    opened by sopankhosla 1
Owner
Chandan Singh
Working on interpretable machine learning across domains 🧠⚕️🦠 Let's do good with models.
Chandan Singh
GitHub https://arxiv.org/abs/1806.05337
L2X - Code for replicating the experiments in the paper Learning to Explain: An Information-Theoretic Perspective on Model Interpretation.

L2X Code for replicating the experiments in the paper Learning to Explain: An Information-Theoretic Perspective on Model Interpretation at ICML 2018,

Jianbo Chen 113 Sep 6, 2022
Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Image Crop Analysis This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post. If you plan to use this

Twitter Research 239 Jan 2, 2023
This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Deep Continuous Clustering Introduction This is a Pytorch implementation of the DCC algorithms presented in the following paper (paper): Sohil Atul Sh

Sohil Shah 197 Nov 29, 2022
Pytorch implementation for reproducing StackGAN_v2 results in the paper StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks

StackGAN-v2 StackGAN-v1: Tensorflow implementation StackGAN-v1: Pytorch implementation Inception score evaluation Pytorch implementation for reproduci

Han Zhang 809 Dec 16, 2022
PyTorch framework, for reproducing experiments from the paper Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks. Code, based on the PyTorch framework, for reprodu

Asaf 3 Dec 27, 2022
A PyTorch implementation of "Capsule Graph Neural Network" (ICLR 2019).

CapsGNN ⠀⠀ A PyTorch implementation of Capsule Graph Neural Network (ICLR 2019). Abstract The high-quality node embeddings learned from the Graph Neur

Benedek Rozemberczki 1.2k Jan 2, 2023
A PyTorch implementation of "Predict then Propagate: Graph Neural Networks meet Personalized PageRank" (ICLR 2019).

APPNP ⠀ A PyTorch implementation of Predict then Propagate: Graph Neural Networks meet Personalized PageRank (ICLR 2019). Abstract Neural message pass

Benedek Rozemberczki 329 Dec 30, 2022
A PyTorch implementation of "Graph Wavelet Neural Network" (ICLR 2019)

Graph Wavelet Neural Network ⠀⠀ A PyTorch implementation of Graph Wavelet Neural Network (ICLR 2019). Abstract We present graph wavelet neural network

Benedek Rozemberczki 490 Dec 16, 2022
Deep Anomaly Detection with Outlier Exposure (ICLR 2019)

Outlier Exposure This repository contains the essential code for the paper Deep Anomaly Detection with Outlier Exposure (ICLR 2019). Requires Python 3

Dan Hendrycks 464 Dec 27, 2022
Repository for reproducing `Model-Based Robust Deep Learning`

Model-Based Robust Deep Learning (MBRDL) In this repository, we include the code necessary for reproducing the code used in Model-Based Robust Deep Le

Alex Robey 16 Sep 19, 2022
PySlowFast: video understanding codebase from FAIR for reproducing state-of-the-art video models.

PySlowFast PySlowFast is an open source video understanding codebase from FAIR that provides state-of-the-art video classification models with efficie

Meta Research 5.3k Jan 3, 2023
Reproducing code of hair style replacement method from Barbershorp.

Barbershorp Reproducing code of hair style replacement method from Barbershorp. Also reproduces II2S, an improved version of Image2StyleGAN. Requireme

null 1 Dec 24, 2021
This repository contains unofficial code reproducing Agent57

Agent57 This repository contains unofficial code reproducing Agent57, which outp

null 19 Dec 29, 2022
This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivariant Continuous Convolution

Trajectory Prediction using Equivariant Continuous Convolution (ECCO) This is the codebase for the ICLR 2021 paper Trajectory Prediction using Equivar

Spatiotemporal Machine Learning 45 Jul 22, 2022
This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures

Introduction This Repo is the official CUDA implementation of ICCV 2019 Oral paper for CARAFE: Content-Aware ReAssembly of FEatures. @inproceedings{Wa

Jiaqi Wang 42 Jan 7, 2023
Code for the CIKM 2019 paper "DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting".

Dual Self-Attention Network for Multivariate Time Series Forecasting 20.10.26 Update: Due to the difficulty of installation and code maintenance cause

Kyon Huang 223 Dec 16, 2022
Pre-trained model, code, and materials from the paper "Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation" (MICCAI 2019).

Adaptive Segmentation Mask Attack This repository contains the implementation of the Adaptive Segmentation Mask Attack (ASMA), a targeted adversarial

Utku Ozbulak 53 Jul 4, 2022
The source code of CVPR 2019 paper "Deep Exemplar-based Video Colorization".

Deep Exemplar-based Video Colorization (Pytorch Implementation) Paper | Pretrained Model | Youtube video ?? | Colab demo Deep Exemplar-based Video Col

Bo Zhang 253 Dec 27, 2022
A PyTorch implementation of SlowFast based on ICCV 2019 paper "SlowFast Networks for Video Recognition"

SlowFast A PyTorch implementation of SlowFast based on ICCV 2019 paper SlowFast Networks for Video Recognition. Requirements Anaconda PyTorch conda in

Hao Ren 8 Dec 23, 2022
Official implementation of the ICLR 2021 paper

You Only Need Adversarial Supervision for Semantic Image Synthesis Official PyTorch implementation of the ICLR 2021 paper "You Only Need Adversarial S

Bosch Research 272 Dec 28, 2022