Saliency - Framework-agnostic implementation for state-of-the-art saliency methods (XRAI, BlurIG, SmoothGrad, and more).

Overview

Saliency Methods

πŸ”΄    Now framework-agnostic! (Example core notebook)   πŸ”΄

πŸ”—    For further explanation of the methods and more examples of the resulting maps, see our Github Pages website   πŸ”—

If upgrading from an older version, update old imports to import saliency.tf1 as saliency. We provide wrappers to make the framework-agnostic version compatible with TF1 models. (Example TF1 notebook)

Introduction

This repository contains code for the following saliency techniques:

*Developed by PAIR.

This list is by no means comprehensive. We are accepting pull requests to add new methods!

Download

# To install the core subpackage:
pip install saliency

# To install core and tf1 subpackages:
pip install saliency[tf1]

or for the development version:

git clone https://github.com/pair-code/saliency
cd saliency

Usage

The saliency library has two subpackages:

  • core uses a generic call_model_function which can be used with any ML framework.
  • tf1 accepts input/output tensors directly, and sets up the necessary graph operations for each method.

Core

Each saliency mask class extends from the CoreSaliency base class. This class contains the following methods:

  • GetMask(x_value, call_model_function, call_model_args=None): Returns a mask of the shape of non-batched x_value given by the saliency technique.
  • GetSmoothedMask(x_value, call_model_function, call_model_args=None, stdev_spread=.15, nsamples=25, magnitude=True): Returns a mask smoothed of the shape of non-batched x_value with the SmoothGrad technique.

The visualization module contains two methods for saliency visualization:

  • VisualizeImageGrayscale(image_3d, percentile): Marginalizes across the absolute value of each channel to create a 2D single channel image, and clips the image at the given percentile of the distribution. This method returns a 2D tensor normalized between 0 to 1.
  • VisualizeImageDiverging(image_3d, percentile): Marginalizes across the value of each channel to create a 2D single channel image, and clips the image at the given percentile of the distribution. This method returns a 2D tensor normalized between -1 to 1 where zero remains unchanged.

If the sign of the value given by the saliency mask is not important, then use VisualizeImageGrayscale, otherwise use VisualizeImageDiverging. See the SmoothGrad paper for more details on which visualization method to use.

call_model_function

call_model_function is how we pass inputs to a given model and receive the outputs necessary to compute saliency masks. The description of this method and expected output format is in the CoreSaliency description, as well as separately for each method.

Examples

This example iPython notebook showing these techniques is a good starting place.

Here is a condensed example of using IG+SmoothGrad with TensorFlow 2:

import saliency.core as saliency
import tensorflow as tf

...

# call_model_function construction here.
def call_model_function(x_value_batched, call_model_args, expected_keys):
	tape = tf.GradientTape()
	grads = np.array(tape.gradient(output_layer, images))
	return {saliency.INPUT_OUTPUT_GRADIENTS: grads}

...

# Load data.
image = GetImagePNG(...)

# Compute IG+SmoothGrad.
ig_saliency = saliency.IntegratedGradients()
smoothgrad_ig = ig_saliency.GetSmoothedMask(image, 
											call_model_function, 
                                            call_model_args=None)

# Compute a 2D tensor for visualization.
grayscale_visualization = saliency.VisualizeImageGrayscale(
    smoothgrad_ig)

TF1

Each saliency mask class extends from the TF1Saliency base class. This class contains the following methods:

  • __init__(graph, session, y, x): Constructor of the SaliencyMask. This can modify the graph, or sometimes create a new graph. Often this will add nodes to the graph, so this shouldn't be called continuously. y is the output tensor to compute saliency masks with respect to, x is the input tensor with the outer most dimension being batch size.
  • GetMask(x_value, feed_dict): Returns a mask of the shape of non-batched x_value given by the saliency technique.
  • GetSmoothedMask(x_value, feed_dict): Returns a mask smoothed of the shape of non-batched x_value with the SmoothGrad technique.

The visualization module contains two visualization methods:

  • VisualizeImageGrayscale(image_3d, percentile): Marginalizes across the absolute value of each channel to create a 2D single channel image, and clips the image at the given percentile of the distribution. This method returns a 2D tensor normalized between 0 to 1.
  • VisualizeImageDiverging(image_3d, percentile): Marginalizes across the value of each channel to create a 2D single channel image, and clips the image at the given percentile of the distribution. This method returns a 2D tensor normalized between -1 to 1 where zero remains unchanged.

If the sign of the value given by the saliency mask is not important, then use VisualizeImageGrayscale, otherwise use VisualizeImageDiverging. See the SmoothGrad paper for more details on which visualization method to use.

Examples

This example iPython notebook shows these techniques is a good starting place.

Another example of using GuidedBackprop with SmoothGrad from TensorFlow:

from saliency.tf1 import GuidedBackprop
from saliency.tf1 import VisualizeImageGrayscale
import tensorflow.compat.v1 as tf

...
# Tensorflow graph construction here.
y = logits[5]
x = tf.placeholder(...)
...

# Compute guided backprop.
# NOTE: This creates another graph that gets cached, try to avoid creating many
# of these.
guided_backprop_saliency = GuidedBackprop(graph, session, y, x)

...
# Load data.
image = GetImagePNG(...)
...

smoothgrad_guided_backprop =
    guided_backprop_saliency.GetMask(image, feed_dict={...})

# Compute a 2D tensor for visualization.
grayscale_visualization = visualization.VisualizeImageGrayscale(
    smoothgrad_guided_backprop)

Conclusion/Disclaimer

If you have any questions or suggestions for improvements to this library, please contact the owners of the PAIR-code/saliency repository.

This is not an official Google product.

Comments
  • added Grad-CAM implementation

    added Grad-CAM implementation

    Here's an implementation of Grad-CAM that extends SaliencyMask class.

    I've included a few optional arguments in the GetMask function (i.e., should_resize, three_dims) to allow the option to return the original, low-res 2D Grad-CAM heatmap, which I need.

    opened by ruthcfong 7
  • Example code doesn't work

    Example code doesn't work

    Hi. I tried to play with the same code, inception model graph and test image (doberman.png) which you provided in examples section as iPython notebook. But I got an error during making the prediction:

    ValueError: Cannot feed value of shape (1, 252, 261, 4) for Tensor 'Placeholder:0', which has shape '(?, 299, 299, 3)'
    

    Do you have any idea how to force example code to work? Thanks.

    # Boilerplate imports.
    import tensorflow as tf
    import numpy as np
    import PIL.Image
    from matplotlib import pylab as P
    import pickle
    import os
    from subprocess import call
    from tensorflow.contrib.slim.python.slim.nets import inception_v3
    import saliency
    
    slim = tf.contrib.slim
    
    if not os.path.exists('models/research/slim'):
      call(["git", "clone", "https://github.com/tensorflow/models/"])
    old_cwd = os.getcwd()
    os.chdir('models/research/slim')
    os.chdir(old_cwd)
    
    # Boilerplate methods.
    def ShowImage(im, title='', ax=None):
      if ax is None:
        P.figure()
      P.axis('off')
      im = ((im + 1) * 127.5).astype(np.uint8)
      P.imshow(im)
      P.title(title)
    
    def ShowGrayscaleImage(im, title='', ax=None):
      if ax is None:
        P.figure()
      P.axis('off')
    
      P.imshow(im, cmap=P.cm.gray, vmin=0, vmax=1)
      P.title(title)
    
    def ShowDivergingImage(grad, title='', percentile=99, ax=None):
      if ax is None:
        fig, ax = P.subplots()
      else:
        fig = ax.figure
    
      P.axis('off')
      divider = make_axes_locatable(ax)
      cax = divider.append_axes('right', size='5%', pad=0.05)
      im = ax.imshow(grad, cmap=P.cm.coolwarm, vmin=-1, vmax=1)
      fig.colorbar(im, cax=cax, orientation='vertical')
      P.title(title)
    
    def LoadImage(file_path):
      im = PIL.Image.open(file_path)
      im = np.asarray(im)
      return im / 127.5 - 1.0
    
    if not os.path.exists('inception_v3.ckpt'):
      call(["curl", "-O", "http://download.tensorflow.org/models/inception_v3_2016_08_28.tar.gz"])
      call(["tar", "-xvzf", "inception_v3_2016_08_28.tar.gz"])
    
    ckpt_file = './inception_v3.ckpt'
    
    graph = tf.Graph()
    
    with graph.as_default():
      images = tf.placeholder(tf.float32, shape=(None, 299, 299, 3))
    
      with slim.arg_scope(inception_v3.inception_v3_arg_scope()):
        _, end_points = inception_v3.inception_v3(images, is_training=False, num_classes=1001)
    
        # Restore the checkpoint
        sess = tf.Session(graph=graph)
        saver = tf.train.Saver()
        saver.restore(sess, ckpt_file)
    
      # Construct the scalar neuron tensor.
      logits = graph.get_tensor_by_name('InceptionV3/Logits/SpatialSqueeze:0')
      neuron_selector = tf.placeholder(tf.int32)
      y = logits[0][neuron_selector]
    
      # Construct tensor for predictions.
      prediction = tf.argmax(logits, 1)
    
      # Load the image
    im = LoadImage('./doberman.png')
    
    # Show the image
    ShowImage(im)
    
    # Make a prediction.
    prediction_class = sess.run(prediction, feed_dict = {images: [im]})[0]
    
    print("Prediction class: " + str(prediction_class))  # Should be a doberman, class idx = 237
    
    opened by ghost 5
  • Add XRAI saliency algorithm

    Add XRAI saliency algorithm

    This PR adds XRAI saliency algorithm from [1].

    [1] Kapishnikov, Andrei, et al. "Segment Integrated Gradients: Better attributions through regions." arXiv preprint arXiv:1906.02825 (2019).


    This change is Reviewable

    opened by tolga-b 4
  • NoneType in vanilla_mask_3d

    NoneType in vanilla_mask_3d

    Hi, I would like to create the saliency map for a DNN that I use for classification of 4 types of images. I manage to calculate the gradient_saliency but when I try to calculate any of the masks e.g vanilla_mask_3d I get the following error:

    vanilla_mask_3d = gradient_saliency.GetMask(im, feed_dict = {neuron_selector: prediction_class}) Traceback (most recent call last): File "<ipython-input-19-7508dfaae246>", line 1, in <module> vanilla_mask_3d = gradient_saliency.GetMask(im, feed_dict = {neuron_selector: prediction_class})

    File "/usr/local/lib/python2.7/dist-packages/saliency/base.py", line 97, in GetMask return self.session.run(self.gradients_node, feed_dict=feed_dict)[0]

    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run run_metadata_ptr)

    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1105, in _run self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)

    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 414, in __init__ self._fetch_mapper = _FetchMapper.for_fetch(fetches)

    File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 231, in for_fetch (fetch, type(fetch)))

    TypeError: Fetch argument None has invalid type <type NoneType'>

    Could you please tell me where might this error be coming from since I don't seem to have a NoneType argument in my prediction_class.

    Cheers, Pouyan

    opened by pazadeh 4
  • module 'saliency.core' has no attribute 'GuidedIG'

    module 'saliency.core' has no attribute 'GuidedIG'

    Hi, I am running the Example_pytorch.ipynb in the branch pytorch-notebook. When I run the cell of Guided IG, the error is shown, module 'saliency.core' has no attribute 'GuidedIG'

    How to solve this? Thanks for any suggestion.

    opened by zhixiongzh 3
  • saliency for reconstruction task

    saliency for reconstruction task

    Thanks for the great implementation! I am wondering what would be different if I'd like to investigate the saliency map of a reconstruction task. In the classification task you can select one neuron, but for reconstruction task, the output is the whole image, is it possible to visualize it in similar way?

    opened by james20141606 3
  • Revert

    Revert "Allow batching for integrated gradient and multithreading for SmoothGrad"

    Reverts PAIR-code/saliency#20 Integrated gradients is producing incorrect results after this PR. Verified that the Examples.ipynb notebook IG results look different:

    New results: ig_broken

    Old results: old_ig

    Verified that the IG attributions no longer sum to input_confidence - baseline_confidence. The IG sum seems to scale with steps now (e.g. if steps=100 and IG_sum=1.0, when steps is set to 200, IG_sum becomes 0.5.).

    opened by tolga-b 3
  • Saliency for regression task

    Saliency for regression task

    Dear authors, first of all, thanks for publishing this code and for your amazing paper. I would like to use your technique to inspect the behavior of a neural network use for 6DoF Pose regression from a single RGB image. In particular, I would like to have a visual insight on which pixels the network could consider most important in the task of localization. In your paper, as well as in the other cited by you, the focus is on the classification problem. Hence, I was wondering how to adapt this visualization technique in the case of regression.

    Thanks in advance for your help.

    opened by ClaudioCimarelli 3
  • The returned masks were all nan

    The returned masks were all nan

    After predicted 237 correctly. It failed becaused the returned masks were all nan `# Compute the vanilla mask and the smoothed mask.

    vanilla_mask_3d = gradient_saliency.GetMask(im, feed_dict = {neuron_selector: prediction_class}) ` https://github.com/PAIR-code/saliency/blob/e96253955a8c8b9fe5f7e89daccbdc161473e089/saliency/base.py#L97

    values of vanilla_mask_3d: [[[nan nan nan] [nan nan nan] [nan nan nan] ... [nan nan nan] [nan nan nan] [nan nan nan]]

    opened by irwenqiang 3
  • Examples_core.ipynb doesn't work

    Examples_core.ipynb doesn't work

    Hi, I am running the core example code for Vanilla Gradient & SmoothGrad on a Kaggle notebook, and I get the error below:


    ValueError Traceback (most recent call last) in 6 ShowImage(im_orig) 7 ----> 8 _, predictions = model(np.array([im])) 9 prediction_class = np.argmax(predictions[0]) 10 call_model_args = {class_idx_str: prediction_class}

    /opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py in call(self, *args, **kwargs) 996 inputs = self._maybe_cast_inputs(inputs, input_list) 997 --> 998 input_spec.assert_input_compatibility(self.input_spec, inputs, self.name) 999 if eager: 1000 call_fn = self.call

    /opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/input_spec.py in assert_input_compatibility(input_spec, inputs, layer_name) 272 ' is incompatible with layer ' + layer_name + 273 ': expected shape=' + str(spec.shape) + --> 274 ', found shape=' + display_shape(x.shape)) 275 276

    ValueError: Input 0 is incompatible with layer model_1: expected shape=(None, 224, 224, 3), found shape=(1, 224, 224, 4)


    Here is the kaggle notebook with the error.

    The only lines I have modified in the example are:

    # From our repository.
    import saliency.core as saliency
    

    and

    # Load the image
    im_orig = LoadImage('./doberman.png')
    

    as

    # From our repository.
    try:
        import saliency.core as saliency`
    except:
        ! pip install saliency
        import saliency.core as saliency`
    

    and

    # Load the image
    im_orig = LoadImage('../input/saliency-imgs/doberman.png')
    

    Also, I have seen there was an issue (#5) created for the code examples to be adapted to TF2. But since the current title of the examples states "for TF2 and other frameworks" I assume the code was finally adapted, and this issue isn't relevant anymore for the problem I am describing.

    Thank you.

    opened by alfredonuhe 2
  • Remove empty dictionary as a default argument in GetMask

    Remove empty dictionary as a default argument in GetMask

    The GetMask function currently uses an empty dictionary as the default argument for feed_dict. This is potentially dangerous, as the global variable may be edited during a run of GetMask. We should change the default value to None and only pass the feed_dict argument when it is specified by the user.

    opened by bwedin 2
  • Update Examples_core.ipynb grad-cam to calculate wrt single class

    Update Examples_core.ipynb grad-cam to calculate wrt single class

    One-line change: adds output_layer = output_layer[:,target_class_idx] to the else statement in def call_model_function. This means that previously Grad-CAM was calculating wrt the entire output layer, this should fix it to only be wrt a single class.

    opened by bwedin 0
  • Call_model_function

    Call_model_function

    I tried to define the call_model_function in an other file and to import the function in my main file. There resulting a lot of errors. Does anybody tried same and has a solution for the problem?

    opened by TizianBo96 0
  • Gradient Problems

    Gradient Problems

    I'm trying to call:

    gradient_saliency = saliency.GradientSaliency()
    vanilla_mask_3d = gradient_saliency.GetMask(im, call_model_function, call_model_args)
    

    but I'm stuck in a loop where I can't solve the problem.

    If I setup my input like this: im = img.unsqueeze(dim=0).to("cuda").requires_grad_(True) Then I get RuntimeError: Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

    But if I change it to im = img.unsqueeze(dim = 0).to(device) I get RuntimeError: One of the differentiated Tensors does not require grad

    It makes sense that it requires the gradient given its task, but why does it try to call .numpy() without doing a .detach()? Maybe I'm setting up something else wrong, but it seems to be isolated to this specific section.

    opened by Squeemos 0
  • Citing PAIR-code/saliency

    Citing PAIR-code/saliency

    I'm using the XRAI implementation from this repository for a project I am working on, and I would like to cite this library. Is there a particular citation I should use? I didn't find anything in the README or via GitHub search. The repository has been very helpful, thank you!

    opened by TylerADavis 0
  • Use batches for SmoothGrad

    Use batches for SmoothGrad

    To speed up the computation for obtaining smoothed masks, the noisy samples are created first and gradients are obtained as a single batch for all the noisy samples. Afterwards the gradients (or their squares) are summed to give the final smoothed mask.

    opened by lenbrocki 0
Owner
PAIR code
Code repositories for projects from the People+AI Research (PAIR) Initiative
PAIR code
An Agnostic Computer Vision Framework - Pluggable to any Training Library: Fastai, Pytorch-Lightning with more to come

IceVision is the first agnostic computer vision framework to offer a curated collection with hundreds of high-quality pre-trained models from torchvision, MMLabs, and soon Pytorch Image Models. It orchestrates the end-to-end deep learning workflow allowing to train networks with easy-to-use robust high-performance libraries such as Pytorch-Lightning and Fastai

airctic 789 Dec 29, 2022
A PyTorch-based open-source framework that provides methods for improving the weakly annotated data and allows researchers to efficiently develop and compare their own methods.

Knodle (Knowledge-supervised Deep Learning Framework) - a new framework for weak supervision with neural networks. It provides a modularization for se

null 93 Nov 6, 2022
Propose a principled and practically effective framework for unsupervised accuracy estimation and error detection tasks with theoretical analysis and state-of-the-art performance.

Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles This project is for the paper: Detecting Errors and Estimating

Jiefeng Chen 13 Nov 21, 2022
A fast, dataset-agnostic, deep visual search engine for digital art history

imgs.ai imgs.ai is a fast, dataset-agnostic, deep visual search engine for digital art history based on neural network embeddings. It utilizes modern

Fabian Offert 5 Dec 14, 2022
How to Become More Salient? Surfacing Representation Biases of the Saliency Prediction Model

How to Become More Salient? Surfacing Representation Biases of the Saliency Prediction Model

Bogdan Kulynych 49 Nov 5, 2022
FEDn is an open-source, modular and ML-framework agnostic framework for Federated Machine Learning

FEDn is an open-source, modular and ML-framework agnostic framework for Federated Machine Learning (FedML) developed and maintained by Scaleout Systems. FEDn enables highly scalable cross-silo and cross-device use-cases over FEDn networks.

Scaleout 75 Nov 9, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Implementation of the state of the art beat-detection, downbeat-detection and tempo-estimation model

The ISMIR 2020 Beat Detection, Downbeat Detection and Tempo Estimation Model Implementation. This is an implementation in TensorFlow to implement the

Koen van den Brink 1 Nov 12, 2021
πŸ˜‡A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

------ Update September 2018 ------ It's been a year since TorchMoji and DeepMoji were released. We're trying to understand how it's being used such t

Hugging Face 865 Dec 24, 2022
Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch

NÜWA - Pytorch (wip) Implementation of NÜWA, state of the art attention network for text to video synthesis, in Pytorch. This repository will be popul

Phil Wang 463 Dec 28, 2022
Implementation of the state-of-the-art vision transformers with tensorflow

ViT Tensorflow This repository contains the tensorflow implementation of the state-of-the-art vision transformers (a category of computer vision model

Mohammadmahdi NouriBorji 2 Mar 16, 2022
Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

?? Flamingo - Pytorch Implementation of Flamingo, state-of-the-art few-shot visual question answering attention net, in Pytorch. It will include the p

Phil Wang 630 Dec 28, 2022
Implementation of ETSformer, state of the art time-series Transformer, in Pytorch

ETSformer - Pytorch Implementation of ETSformer, state of the art time-series Transformer, in Pytorch Install $ pip install etsformer-pytorch Usage im

Phil Wang 121 Dec 30, 2022
Current state of supervised and unsupervised depth completion methods

Awesome Depth Completion Table of Contents About Sparse-to-Dense Depth Completion Current State of Depth Completion Unsupervised VOID Benchmark Superv

null 224 Dec 28, 2022
aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

Bayesian Methods for Hackers Using Python and PyMC The Bayesian method is the natural approach to inference, yet it is hidden from readers behind chap

Cameron Davidson-Pilon 25.1k Jan 2, 2023
tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

timeseriesAI 2.8k Jan 8, 2023
Deep Text Search is an AI-powered multilingual text search and recommendation engine with state-of-the-art transformer-based multilingual text embedding (50+ languages).

Deep Text Search - AI Based Text Search & Recommendation System Deep Text Search is an AI-powered multilingual text search and recommendation engine w

null 19 Sep 29, 2022