Class activation maps for your PyTorch models (CAM, Grad-CAM, Grad-CAM++, Smooth Grad-CAM++, Score-CAM, SS-CAM, IS-CAM, XGrad-CAM, Layer-CAM)

Overview

TorchCAM: class activation explorer

License: Apache 2.0 Codacy Badge Build Status codecov Docs Pypi

Simple way to leverage the class-specific activation of convolutional layers in PyTorch.

gradcam_sample

Quick Tour

Setting your CAM

TorchCAM leverages PyTorch hooking mechanisms to seamlessly retrieve all required information to produce the class activation without additional efforts from the user. Each CAM object acts as a wrapper around your model.

You can find the exhaustive list of supported CAM methods in the documentation, then use it as follows:

# Define your model
from torchvision.models import resnet18
model = resnet18(pretrained=True).eval()

# Set your CAM extractor
from torchcam.cams import SmoothGradCAMpp
cam_extractor = SmoothGradCAMpp(model)

Please note that by default, the layer at which the CAM is retrieved is set to the last non-reduced convolutional layer. If you wish to investigate a specific layer, use the target_layer argument in the constructor.

Retrieving the class activation map

Once your CAM extractor is set, you only need to use your model to infer on your data as usual. If any additional information is required, the extractor will get it for you automatically.

from torchvision.io.image import read_image
from torchvision.transforms.functional import normalize, resize, to_pil_image
from torchvision.models import resnet18
from torchcam.cams import SmoothGradCAMpp

model = resnet18(pretrained=True).eval()
cam_extractor = SmoothGradCAMpp(model)
# Get your input
img = read_image("path/to/your/image.png")
# Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

# Preprocess your data and feed it to the model
out = model(input_tensor.unsqueeze(0))
# Retrieve the CAM by passing the class index and the model output
activation_map = cam_extractor(out.squeeze(0).argmax().item(), out)

If you want to visualize your heatmap, you only need to cast the CAM to a numpy ndarray:

import matplotlib.pyplot as plt
# Visualize the raw CAM
plt.imshow(activation_map.numpy()); plt.axis('off'); plt.tight_layout(); plt.show()

raw_heatmap

Or if you wish to overlay it on your input image:

import matplotlib.pyplot as plt
from torchcam.utils import overlay_mask

# Resize the CAM and overlay it
result = overlay_mask(to_pil_image(img), to_pil_image(activation_map, mode='F'), alpha=0.5)
# Display it
plt.imshow(result); plt.axis('off'); plt.tight_layout(); plt.show()

overlayed_heatmap

Setup

Python 3.6 (or higher) and pip/conda are required to install TorchCAM.

Stable release

You can install the last stable release of the package using pypi as follows:

pip install torchcam

or using conda:

conda install -c frgfm torchcam

Developer installation

Alternatively, if you wish to use the latest features of the project that haven't made their way to a release yet, you can install the package from source:

git clone https://github.com/frgfm/torch-cam.git
pip install -e torch-cam/.

CAM Zoo

This project is developed and maintained by the repo owner, but the implementation was based on the following research papers:

  • Learning Deep Features for Discriminative Localization: the original CAM paper
  • Grad-CAM: GradCAM paper, generalizing CAM to models without global average pooling.
  • Grad-CAM++: improvement of GradCAM++ for more accurate pixel-level contribution to the activation.
  • Smooth Grad-CAM++: SmoothGrad mechanism coupled with GradCAM.
  • Score-CAM: score-weighting of class activation for better interpretability.
  • SS-CAM: SmoothGrad mechanism coupled with Score-CAM.
  • IS-CAM: integration-based variant of Score-CAM.
  • XGrad-CAM: improved version of Grad-CAM in terms of sensitivity and conservation.
  • Layer-CAM: Grad-CAM alternative leveraging pixel-wise contribution of the gradient to the activation.

What else

Documentation

The full package documentation is available here for detailed specifications.

Demo app

A minimal demo app is provided for you to play with the supported CAM methods!

You will need an extra dependency (Streamlit) for the app to run:

pip install -r demo/requirements.txt

You can then easily run your app in your default browser by running:

streamlit run demo/app.py

torchcam_demo

Example script

An example script is provided for you to benchmark the heatmaps produced by multiple CAM approaches on the same image:

python scripts/cam_example.py --model resnet18 --class-idx 232

gradcam_sample

All script arguments can be checked using python scripts/cam_example.py --help

Citation

If you wish to cite this project, feel free to use this BibTeX reference:

@misc{torcham2020,
    title={TorchCAM: class activation explorer},
    author={François-Guillaume Fernandez},
    year={2020},
    month={March},
    publisher = {GitHub},
    howpublished = {\url{https://github.com/frgfm/torch-cam}}
}

Contributing

Feeling like extending the range of possibilities of CAM? Or perhaps submitting a paper implementation? Any sort of contribution is greatly appreciated!

You can find a short guide in CONTRIBUTING to help grow this project!

License

Distributed under the Apache 2.0 License. See LICENSE for more information.

Comments
  • Autograd Warning

    Autograd Warning

    🐛 Bug

    When I am integrating densenet121 with torchcam I am getting following warning, which I am not getting while using the model diectly.

    usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:1204: UserWarning: Output 0 of BackwardHookFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is deprecated and will be forbidden starting version 1.6. You can remove this warning by cloning the output of the custom Function. (Triggered internally at /pytorch/torch/csrc/autograd/variable.cpp:547.) result = torch.relu_(input)

    Here is the code

    model = densenet121(pretrained=True).eval()
    
    img = read_image("image.png")
    input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    extract = GradCAM(model=model)
    out = model(input_tensor.unsqueeze(0))
    

    Environment

    • I am working on Google colab
    • Installed Torchcam using pip
    • Pytorch version: 1.8.1+cu101
    • Torchcam Version: 0.2.0
    • Python version: 3.7.10
    type: bug module: methods 
    opened by akshay-gupta123 13
  • Getting the CAM for BiT

    Getting the CAM for BiT

    Hi.

    I am trying to get CAM for a particular variant of the BiT (Big Transfer) family of ResNets. I think I am providing the the target_layer name correctly however I am unable to make it work.

    Here's how the last portion of the model looks like:

    (norm): GroupNormAct(
        32, 6144, eps=1e-05, affine=True
        (act): ReLU(inplace=True)
      )
      (head): ClassifierHead(
        (global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=False)
        (fc): Conv2d(6144, 1000, kernel_size=(1, 1), stride=(1, 1))
      )
    )
    

    Here's the error I am getting:

    KeyError                                  Traceback (most recent call last)
    <ipython-input-13-f63345257370> in <module>()
          1 model = timm.create_model('resnetv2_101x3_bitm', pretrained=True).eval()
    ----> 2 cam_extractor = CAM(model, fc_layer='fc')
          3 
          4 # Get your input
          5 img = read_image("welsh-corgi-1581119_960_720.jpg")
    
    /usr/local/lib/python3.7/dist-packages/torchcam/cams/cam.py in __init__(self, model, target_layer, fc_layer, input_shape)
         62                 raise ValueError("unable to resolve `fc_layer` automatically, please specify its value.")
         63         # Softmax weight
    ---> 64         self._fc_weights = self.submodule_dict[fc_layer].weight.data
         65 
         66     def _get_weights(self, class_idx: int, scores: Optional[Tensor] = None) -> Tensor:
    
    KeyError: 'fc'
    

    The main code:

    model = timm.create_model('resnetv2_101x3_bitm', pretrained=True).eval()
    cam_extractor = CAM(model, fc_layer='fc')
    

    Here's the Colab Notebook to reproduce the issue.

    type: enhancement module: methods 
    opened by sayakpaul 11
  • What are the requirements of the model by using these visual code?

    What are the requirements of the model by using these visual code?

    Bug description

    image

    I have got a error, we failed to use these method to obtain a results. I have got 4D matrix, but it needs to be 2D. My model have not a fully connected layer,while consists of many convolution layers. Should I how to deal with this problem? Thank you

    Code snippet to reproduce the bug

    def _backprop(self, scores: Tensor, class_idx: int) -> None:
        """Backpropagate the loss for a specific output class"""
    
        # Backpropagate to get the gradients on the hooked layer
    
    
        loss = scores[:, class_idx].sum()
        self.model.zero_grad()
        loss.backward(retain_graph=True)
    

    Error traceback

    image

    Environment

    Cuda 11.0 torch-gpu==1.7.0

    wontfix module: methods awaiting response 
    opened by long123524 10
  • Some models will calculate 'nan'

    Some models will calculate 'nan'

    I try your code with your example picture on resnet34 and resnet50 ,the the scoreCAM, SSCAM, ISCAM will calculate 'nan' . can you help me to solve this problem?

    I try on torch 1.5

    type: bug help wanted ext: scripts 
    opened by LuckGuySam 9
  • Any interest to suppport LayerCAM

    Any interest to suppport LayerCAM

    Hello, @frgfm, Our paper "LayerCAM: Exploring Hierarchical Class Activation Maps for Localization" is accepted by TIP recently, which can visualize the class activation maps from any cnn layer of an off-the-shelf network. Could you add our method to your popular repository for more people to try this method? Our method is a simple modification of Grad-CAM. It should easy to implement. Here is the paper and code. Hope for your reply.

    type: enhancement module: methods 
    opened by PengtaoJiang 8
  •  self._normalize() got  NAN...

    self._normalize() got NAN...

    Bug description

    Sir, when I print upsampled_a in upsampled_a = self._normalize(self.hook_a, self.hook_a.ndim - 2) I got NAN in some value. Is there any bug?

    I set model to vgg19 target_layer1='features.35' fc_layer1='classifier.6'

    Code snippet to reproduce the bug

    nan

    Error traceback

    nan

    Environment

    nan

    type: bug module: methods 
    opened by noobgrow 7
  • Multi-Inputs CAM support

    Multi-Inputs CAM support

    🚀 Feature

    Hi, thanks for this great toolbox.

    I am wondering how to get CAM for multiple input cases. Currently, I found that the CAM I am getting is from the last input.

    The model which I want to check takes 3 images as a single input and then concatenates or stack the feature maps for the final FC layer to make a multi-label classification.

    Would it be possible to generate 3 CAMs for each image?

    Thanks in advance

    Motivation & pitch

    • Multi-view model or multi-input model take a list of items as a single input
    • Current CAM code will only generate a class activation map based on the last item in the list (depending on how items are passed to the forward function tho).

    Alternatives

    No response

    Additional context

    No response

    question module: methods awaiting response 
    opened by icedpanda 7
  • Make target_layer also a `nn.Module`

    Make target_layer also a `nn.Module`

    🚀 Feature

    Hello there,

    I would like to pass an nn.Module instance to any *Cam constructor but currently, I can only use 'str'.

    Motivation

    Well, if your model has a good design it is much easier to pass a reference than the key.

    Thanks!

    Francesco

    type: enhancement question module: methods 
    opened by FrancescoSaverioZuppichini 7
  • What is class_idx in __call__() ??

    What is class_idx in __call__() ??

    Hello sir, I want to display heat-map for my classification task. So i found your library. What is class_idx of call()?? (I think that it is class index of top-1)

    And, Using my datasets, I met nan. In compute_cam() of core.py My result of torch.nansum has all < 0. Then F.relu return all 0. So, in normalization, it return nan.

    Is this nornal?? If not, what should i change in my data??

    Thanks. Edward cho

    type: bug question awaiting response 
    opened by edwardcho 6
  • element 0 of tensors does not require grad and does not have a grad_fn

    element 0 of tensors does not require grad and does not have a grad_fn

    I get this error when I use the API.

    `model = models.resnet34(pretrained=True).eval() cam = GradCAMpp(model,'layer4') import numpy as np from PIL import Image from torch.autograd import Variable

    from torchvision.transforms.functional import normalize,resize,to_tensor img_path = '' pil_img = Image.open(img_path) #pil_img = np.expand_dims(img_tensor,axis=0)

    img_tensor = normalize(to_tensor(resize(pil_img,(224,224))),[0.485,0.456,0.406],[0.229,0.224,0.225]) _ = _forward(model,torch.stack((img_tensor,img_tensor))) scores = _forward(model, img_tensor.unsqueeze(0)) #img_variable = Variable(img_tensor.unsqueeze(0),requires_grad=True) cam(class_idx=1,scores=scores) `

    topic: documentation question module: methods 
    opened by sosoIrving 6
  • Model evaluation

    Model evaluation

    Hi,

    I have a question about cam_example.py. Shouldn't the model be in eval() mode? I noticed without setting the model.eval() the prediction labels are wrong while after setting to eval mode the ScoreCam and SSCAM heatmaps are zero. Could you please check this?

    type: bug question ext: scripts 
    opened by gulleh 6
  • Update README.md

    Update README.md

    Add Chinese video and jupyter notebook tutorials

    What does this PR do?

    Closes # (issue)

    Before submitting

    • [ ] Was this discussed/approved in a Github issue or a discussion? Please add a link to it if that's the case.
    • [ ] You have read the contribution guidelines and followed them in this PR.
    • [ ] Did you make sure to update the documentation with your changes? Here are the documentation guidelines.
    • [ ] Did you write any new necessary tests?
    topic: documentation type: enhancement awaiting response 
    opened by TommyZihao 1
  • Release tracker - v0.5.0

    Release tracker - v0.5.0

    This issue is to be used to track the roadmap of TorchCAM for release v0.5.0, and collect feedback from users & contributors.

    Methods

    Scripts

    Demo

    opened by frgfm 0
  • Upcoming support for new CAM methods

    Upcoming support for new CAM methods

    TorchCAM is now mature enough to welcome new CAM methods :+1: This issue is meant to track upcoming additions on the next release!

    A few things to consider:

    • TorchCAM is not meant to support all available methods. This should focus on the ones providing SOTA performances (or perf milestones)
    • the rightful contributors have to be credited: if a piece of code is borrowed from someone else's implementation, that should be indicated ("borrowed from", "inspired by", etc.)

    Here is the list of envisioned CAM methods:

    Suggestions are welcome :smile:

    help wanted module: methods type: new feature 
    opened by frgfm 0
  • [demo] Automate deployment to HF Spaces

    [demo] Automate deployment to HF Spaces

    The demo hosted on HF Spaces has been updated manually so far. However it is possible to configure an automated deployment: https://huggingface.co/docs/hub/spaces-github-actions

    ext: demo topic: ci type: new feature 
    opened by frgfm 0
  • Release tracker - v0.4.0

    Release tracker - v0.4.0

    This issue is to be used to track the roadmap of TorchCAM for release v0.4.0, and collect feedback from users & contributors.

    Methods

    • [ ] #133
    • [ ] Investigate whether the new PyTorch FX could be used instead of hooks (cf. https://pytorch.org/blog/FX-feature-extraction-torchvision)
    • [x] #139
    • [x] #174
    • [ ] #178
    • [x] #197

    Scripts

    • [x] #140

    Demo

    • [ ] #177
    opened by frgfm 0
  • Add support of Vision Transformer

    Add support of Vision Transformer

    🚀 Feature

    I am appreciated for your great job! However, I have a question. Can Layer-CAM be used with Vision Transformer Network? If it does work, what aspects should I change?

    Motivation & pitch

    I'm working on the job related to CAM.

    Alternatives

    No response

    Additional context

    No response

    type: enhancement module: methods 
    opened by Yung-zi 4
Releases(v0.3.2)
  • v0.3.2(Aug 2, 2022)

    This patch release fixes the Score-CAM methods and improves the base API for CAM computation.

    Note: TorchCAM 0.3.2 requires PyTorch 1.7.0 or higher.

    Highlights

    :hushed: Batch processing

    CAM computation now supports batch sizes larger than 1 (#143) ! Practically, this means that you can compute CAMs for multiple samples at the same time, which will let you make the most of your GPU as well :zap:

    The following snippet:

    import torch
    from torchcam.methods import LayerCAM
    from torchvision.models import resnet18
    
    # A preprocessed (resized & normalized) tensor
    img_tensor = torch.rand((2, 3, 224, 224))
    model = resnet18(pretrained=True).eval()
    # Hook your model before inference
    cam_extractor = LayerCAM(model)
    out = model(img_tensor)
    # Compute the CAM
    activation_map = cam_extractor(out[0].argmax().item(), out)
    print(activation_map[0].ndim)
    

    will yield 3 as the batch dimension is now also used.

    :paintbrush: Documentation theme

    New year, new documentation theme! For clarity and improved interface, the documentation theme was changed from Read the Docs to Furo (#162)

    image

    This comes with nice features like dark mode and edit button!

    :computer_mouse: Contribution process

    Contributions are important to OSS projects, and for this reason, a few improvements were made to the contribution process:

    • added a Makefile for easier development (#109)
    • added a dedicated README for the documentation (#109)
    • updated CONTRIBUTING (#109, #166)

    Breaking changes

    CAM signature

    CAM extractors now outputs a list of tensors. The size of the list is equal to the number of target layers and ordered the same way. Each of these elements used to be a 2D spatial tensor, and is now a 3D tensor to include the batch dimension:

    # Model was hooked and a tensor of shape (2, 3, 224, 224) was forwarded to it
    amaps = cam_extractor(0, out)
    for elt in amaps: print(elt.shape)
    

    will, from now on, yield

    torch.Size([2, 7, 7])
    

    What's Changed

    Breaking Changes 🛠

    • feat: Adds support for batch processing and fixes ScoreCAMs by @frgfm in https://github.com/frgfm/torch-cam/pull/143

    New Features 🚀

    • ci: Added release note template and a job to check PR labels by @frgfm in https://github.com/frgfm/torch-cam/pull/138
    • docs: Added CITATION file by @frgfm in https://github.com/frgfm/torch-cam/pull/144

    Bug Fixes 🐛

    • fix: Updated headers and added pydocstyle by @frgfm in https://github.com/frgfm/torch-cam/pull/137
    • chore: Updated PyTorch version specifier by @frgfm in https://github.com/frgfm/torch-cam/pull/149
    • docs: Fixed deprecated method call by @frgfm in https://github.com/frgfm/torch-cam/pull/158
    • chore: Fixed jinja2 deps (subdep of sphinx) by @frgfm in https://github.com/frgfm/torch-cam/pull/159
    • docs: Fixed docstring of ISCAM by @frgfm in https://github.com/frgfm/torch-cam/pull/160
    • docs: Fixed multi-version build by @frgfm in https://github.com/frgfm/torch-cam/pull/163
    • docs: Fixed codacy badge by @frgfm in https://github.com/frgfm/torch-cam/pull/164
    • docs: Fixed typo in CONTRIBUTING by @frgfm in https://github.com/frgfm/torch-cam/pull/166
    • docs: Fixed author entry in pyproject by @frgfm in https://github.com/frgfm/torch-cam/pull/168
    • style: Fixed import order by @frgfm in https://github.com/frgfm/torch-cam/pull/175

    Improvements

    • docs: Added PR template and tools for contributing by @frgfm in https://github.com/frgfm/torch-cam/pull/109
    • refactor: Removed unused import by @frgfm in https://github.com/frgfm/torch-cam/pull/110
    • feat: Added text strip for multiple target selection in demo by @frgfm in https://github.com/frgfm/torch-cam/pull/111
    • refactor: Updated environment collection script by @frgfm in https://github.com/frgfm/torch-cam/pull/112
    • style: Updated flake8 config by @frgfm in https://github.com/frgfm/torch-cam/pull/115
    • ci: Updated isort config and related CI job by @frgfm in https://github.com/frgfm/torch-cam/pull/118
    • ci: Speeded up the example script CI check by @frgfm in https://github.com/frgfm/torch-cam/pull/130
    • refactor: Updated the timing function for latency eval by @frgfm in https://github.com/frgfm/torch-cam/pull/129
    • docs: Updated TOC of documentation by @frgfm in https://github.com/frgfm/torch-cam/pull/161
    • refactor: Updated build config and documentation theme by @frgfm in https://github.com/frgfm/torch-cam/pull/162
    • style: Updated mypy and isort configs by @frgfm in https://github.com/frgfm/torch-cam/pull/167
    • chore: Improved version specifiers and fixed conda recipe by @frgfm in https://github.com/frgfm/torch-cam/pull/169
    • docs: Fixed README badge and updated documentation by @frgfm in https://github.com/frgfm/torch-cam/pull/170
    • ci: Updated release job by @frgfm in https://github.com/frgfm/torch-cam/pull/173
    • refactor: Improved target resolution by @frgfm in https://github.com/frgfm/torch-cam/pull/174
    • ci: Updated the trigger for the release job by @frgfm in https://github.com/frgfm/torch-cam/pull/176
    • docs: Updated landing page screenshot by @frgfm in https://github.com/frgfm/torch-cam/pull/179

    Full Changelog: https://github.com/frgfm/torch-cam/compare/v0.3.1...v0.3.2

    Source code(tar.gz)
    Source code(zip)
  • v0.3.1(Oct 31, 2021)

    This patch release adds new features to the demo and reorganizes the package for a clearer hierarchy.

    Note: TorchCAM 0.3.1 requires PyTorch 1.5.1 or higher.

    Highlights

    CAM fusion is coming to the demo :rocket:

    With release 0.3.0, the support of multiple target layers was added as well as CAM fusion. The demo was updated to automatically fuse CAMs when you hooked multiple layers (add a "+" separator between each layer name):

    demo

    Breaking changes

    Submodule renaming

    To anticipate further developments of the library, modules were renamed:

    • torchcam.cams was renamed into torchcam.methods
    • torchcam.cams.utils was renamed and made private (torchcam.methods._utils) since it's API may evolve quickly
    • activation-based CAM methods are now implemented in torchcam.methods.activation rather than torchcam.cams.cam
    • gradient-based CAM methods are now implemented in torchcam.methods.gradient rather than torchcam.cams.gradcam

    0.3.0 | 0.3.1 -- | -- >>> from torchcam.cams import LayerCAM | >>> from torchcam.methods import LayerCAM |

    What's Changed

    • chore: Made post release modifications by @frgfm in https://github.com/frgfm/torch-cam/pull/103
    • docs: Updated changelog by @frgfm in https://github.com/frgfm/torch-cam/pull/104
    • feat: Added possibility to retrieve multiple CAMs in demo by @frgfm in https://github.com/frgfm/torch-cam/pull/105
    • refactor: Reorganized package hierarchy by @frgfm in https://github.com/frgfm/torch-cam/pull/106
    • docs: Fixed LaTeX syntax in docstrings by @frgfm in https://github.com/frgfm/torch-cam/pull/107

    Full Changelog: https://github.com/frgfm/torch-cam/compare/v0.3.0...v0.3.1

    Source code(tar.gz)
    Source code(zip)
    carbon.png(166.06 KB)
    example.png(698.66 KB)
  • v0.3.0(Oct 31, 2021)

    This release extends CAM methods with Layer-CAM, greatly improves the core features (CAM computation for multiple layers at once, CAM fusion, support of torch.nn.Module), while improving accessibility for entry users.

    Note: TorchCAM 0.3.0 requires PyTorch 1.5.1 or higher.

    Highlights

    Enters Layer-CAM

    The previous release saw the introduction of Score-CAM variants, and this one introduces you to Layer-CAM, which is meant to be considerably faster, while offering very competitive localization cues!

    Just like any other CAM methods, you can now use it as follows:

    from torchcam.cams import LayerCAM
    # model = ....
    # Hook the model
    cam_extractor = LayerCAM(model)
    

    Consequently, the illustration of visual outputs for all CAM methods has been updated so that you can better choose the option that suits you:

    cam_example

    Computing CAMs for multiple layers & CAM fusion

    A class activation map is specific to a given layer in a model. To fully capture the influence of visual traits on your classification output, you might want to explore the CAMs for multiple layers.

    For instance, here are the CAMs on the layers "layer2", "layer3" and "layer4" of a resnet18:

    from torchvision.io.image import read_image
    from torchvision.models import resnet18
    from torchvision.transforms.functional import normalize, resize, to_pil_image
    import matplotlib.pyplot as plt
    
    from torchcam.cams import LayerCAM
    from torchcam.utils import overlay_mask
    
    # Download an image
    !wget https://www.woopets.fr/assets/races/000/066/big-portrait/border-collie.jpg
    # Set this to your image path if you wish to run it on your own data
    img_path = "border-collie.jpg"
    
    # Get your input
    img = read_image(img_path)
    # Preprocess it for your chosen model
    input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    # Get your model
    model = resnet18(pretrained=True).eval()
    # Hook the model
    cam_extractor = LayerCAM(model, ["layer2", "layer3", "layer4"])
    
    out = model(input_tensor.unsqueeze(0))
    cams = cam_extractor(out.squeeze(0).argmax().item(), out)
    # Plot the CAMs
    _, axes = plt.subplots(1, len(cam_extractor.target_names))
    for idx, name, cam in zip(range(len(cam_extractor.target_names)), cam_extractor.target_names, cams):
      axes[idx].imshow(cam.numpy()); axes[idx].axis('off'); axes[idx].set_title(name);
    plt.show()
    

    multi_cams

    Now, the way you would combine those together is up to you. By default, most approaches use an element-wise maximum. But, LayerCAM has its own fusion method:

    # Let's fuse them
    fused_cam = cam_extractor.fuse_cams(cams)
    # Plot the raw version
    plt.imshow(fused_cam.numpy()); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()
    

    fused_cams

    # Overlay it on the image
    result = overlay_mask(to_pil_image(img), to_pil_image(fused_cam, mode='F'), alpha=0.5)
    # Plot the result
    plt.imshow(result); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()
    

    fused_overlay

    Support of torch.nn.Module as target_layer

    While making the API more robust, CAM constructors now also accept torch.nn.Module as target_layer. Previously, you had to pass the name of the layer as string, but you can now pass the object reference directly if you prefer:

    from torchcam.cams import LayerCAM
    # model = ....
    # Hook the model
    cam_extractor = LayerCAM(model, model.layer4)
    

    :zap: Latency benchmark :zap:

    Since CAMs can be used from localization or production pipelines, it is important to consider latency along with pure visual output quality. For this reason, a latency evaluation script has been included in this release along with a full benchmark table.

    Should you wish to have latency metrics on your dedicated hardware, you can run the script on your own:

    python scripts/eval_latency.py SmoothGradCAMpp --size 224
    

    Notebooks :play_or_pause_button:

    Do you prefer to only run code rather than write it? Perhaps you only want to tweak a few things? Then enjoy the brand new Jupyter notebooks than you can either run locally or on Google Colab!

    :hugs: Live demo :hugs:

    The ML community was recently blessed by HuggingFace with their beta of Spaces, which let you host free-of-charge your ML demos!

    Previously, you were able to run the demo locally on deploy it on your own, but now, you can enjoy the live demo of TorchCAM :art:

    Breaking changes

    Multiple CAM output

    Since CAM extractor can now compute the resulting maps for multiple layer at a time, the return type of all CAMs has been changed from torch.Tensor to List[torch.Tensor] with N elements, where N is the number of target layers.

    0.2.0 | 0.3.0 -- | -- >>> from torchcam.cams import SmoothGradCAMpp
    >>> extractor = SmoothGradCAMpp(model)
    >>> out = model(input_tensor.unsqueeze(0))
    >>> print(type(cam_extractor(out.squeeze(0).argmax().item(), out)))
    <class 'torch.Tensor'> | >>> from torchcam.cams import SmoothGradCAMpp
    >>> extractor = SmoothGradCAMpp(model)
    >>> out = model(input_tensor.unsqueeze(0))
    >>> print(type(cam_extractor(out.squeeze(0).argmax().item(), out)))
    <class 'list'> |

    New features

    CAMs

    Implementations of CAM method

    • Added support of conv1x1 as FC candidate in base CAM #69 (@frgfm)
    • Added support of LayerCAM #77 (@frgfm)
    • Added support of torch.nn.Module as target_layer or fc_layer #83 (@frgfm)
    • Added support of multiple target layers for all CAM methods #89 #92 (@frgfm)
    • Added layer-specific CAM fusion method #93 (@frgfm)

    Scripts

    Side scripts to make the most out of TorchCAM

    • Added latency evaluation script #95 (@frgfm)

    Test

    Verifications of the package well-being before release

    • Added unittests to verify that conv1x1 can be used as FC in base CAM #69 (@frgfm)
    • Added unittest for LayerCAM #77 (@frgfm)
    • Added unittest for gradient-based CAM method for models with in-place ops #80 (@frgfm)
    • Added unittest to check support of torch.nn.Module as target_layer in CAM constructor #83 #88 (@frgfm)
    • Added unittest for CAM fusion #93 (@frgfm)

    Documentation

    Online resources for potential users

    • Added LayerCAM ref in the README and in the documentation #77 (@frgfm)
    • Added CODE_OF_CONDUCT #86 (@frgfm)
    • Added changelog to the documentation #91 (@frgfm)
    • Added latency benchmark & GIF illustration of CAM on a video in README #95 (@frgfm)
    • Added documentation of .fuse_cams method #93 (@frgfm)
    • Added ref to HF Space demo in README and documentation #96 (@frgfm)
    • Added tutorial notebooks and reference page in the documentation #99 #100 #101 #102 (@frgfm)

    Others

    Other tools and implementations

    • Added class_idx & target_layer selection in the demo #67 (@frgfm)
    • Added CI jobs to build on different OS & Python versions, to validate the demo, and the example script #73 #74 (@frgfm)
    • Added LayerCAM to the demo #77 (@frgfm)
    • Added an environment collection script #78 (@frgfm)
    • Added CI check for the latency evaluation script #95 (@frgfm)

    Bug fixes

    CAMs

    • Fixes backward hook mechanism for in-place operations #80 (@frgfm)

    Documentation

    • Fixed docutils version constraint for documentation building #98 (@frgfm)

    Others

    • Fixed CI job to build documentation #64 (@frgfm)
    • Fixed Pillow version constraint #73 #84 (@frgfm)

    Improvements

    CAMs

    • Improved weight broadcasting for all CAMs #77 (@frgfm)
    • Refactored hook enabling #80 (@frgfm)
    • Improved the warning message for target automatic resolution #87 #92 (@frgfm)
    • Improved arg type checking for CAM constructor #88 (@frgfm)

    Scripts

    • Improved the layout option of the example script #66 (@frgfm)
    • Refactored example script #80 #94 (@frgfm)
    • Updated all scripts for support of multiple target layers #89 (@frgfm)

    Test

    • Updated unittests for multiple target layer support #89 (@frgfm)

    Documentation

    • Added latest release doc version & updated README badge #63 (@frgfm)
    • Added demo screenshot in the README #67 (@frgfm)
    • Updated instructions in README #89 (@frgfm)
    • Improved documentation landing page #91 (@frgfm)
    • Updated contribution guidelines #94 (@frgfm)
    • Updated documentation requirements #99 (@frgfm)

    Others

    • Updated package version and fixed CI jobs to validate release publish #63 (@frgfm)
    • Updated license from MIT to Apache 2.0 #70 (@frgfm)
    • Refactored CI jobs #73 (@frgfm)
    • Improved bug report template #78 (@frgfm)
    • Updated streamlit syntax in demo #94 (@frgfm)
    • Added isort config and CI job #97 (@frgfm)
    • Added CI job for sanity check of the documentation build #98 (@frgfm)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Apr 10, 2021)

    This release extends TorchCAM compatibility to 3D inputs, and improves documentation.

    Note: TorchCAM 0.2.0 requires PyTorch 1.5.1 or higher.

    Highlights

    Compatibility for inputs with more than 2 spatial dimensions

    The first papers about CAM methods were built for classification models using 2D (spatially) inputs. However, the latest methods can be extrapolated to higher dimension inputs and it's now live:

    import torch
    from torchcam.cams import SmoothGradCAMpp
    # Define your model compatible with 3D inputs
    video_model = ...
    extractor = SmoothGradCAMpp(video_model)
    # Forward your input
    scores = model(torch.rand((1, 3, 32, 224, 224)))
    # Retrieve the CAM
    cam = extractor(scores[0].argmax().item(), scores)
    

    Multi-version documentation

    While documentation was up-to-date with the latest commit on the main branch, previously if you were running an older release of the library, you had no corresponding documentation.

    As of now, you can select the version of the documentation you wish to access (stable releases or latest commit): torchcam_doc

    Demo app

    Since spatial information is at the very core of TorchCAM, a minimal Streamlit demo app was added to explore the activation of your favorite models. You can run the demo with the following commands:

    streamlit run demo/app.py
    

    Here is how it renders retrieving the heatmap using SmoothGradCAMpp on a pretrained resnet18: torchcam_demo

    New features

    CAMs

    Implementations of CAM method

    • Enabled CAM compatibility for inputs with more than 2 spatial dimensions #45 (@frgfm)
    • Added support of XGradCAM #47 (@frgfm)

    Test

    Verifications of the package well-being before release

    • Added unittests for XGradCAM #47 (@frgfm)

    Documentation

    Online resources for potential users

    • Added references to XGradCAM in README and documentation #47 (@frgfm)
    • Added multi-version documentation & added github star button #53, #54, #55, #56 (@frgfm)
    • Revamped README #59 (@frgfm) focusing on short easy code snippets
    • Improved documentation #60 (@frgfm)

    Others

    Other tools and implementations

    • Added issue templates for bug report and feature request #49 (@frgfm)
    • Added option to specify a single CAM method in example script #52 (@frgfm)
    • Added minimal demo app #59 (@frgfm)

    Bug fixes

    CAMs

    • Fixed automatic layer resolution on GPU #41 (@frgfm)
    • Fixed backward hook warnings for Pytorch >= 1.8.0 #58 (@frgfm)

    Utils

    • Fixed RGBA -> RGB conversion in overlay_mask #38 (@alexandrosstergiou)

    Test

    • Fixed overlay_mask unittest #38 (@alexandrosstergiou)

    Documentation

    • Fixed codacy badge in README #46 (@frgfm)
    • Fixed typo in documentation #62 (@frgfm)

    Others

    • Fixed CI job for conda build #34 (@frgfm)
    • Fixed model mode in example script #37 (@frgfm)
    • Fixed sphinx version #40 (@frgfm)
    • Fixed usage instructions in README #43 (@frgfm)
    • Fixed example script for local image input #51 (@frgfm)

    Improvements

    CAMs

    • Added NaN check in gradcams #37 (@frgfm)

    Test

    • Added NaN check unittest for gradcam #37 (@frgfm)
    • Switched from unittest to pytest #45 (@frgfm) and split test files by module

    Documentation

    • Updated README badges #34, illustration #39 and usage instructions #41 (@frgfm)
    • Added instructions to run all CI checks locally in CONTRIBUTING #34, #45 (@frgfm)
    • Updated project hierarchy description in CONTRIBUTING #43 (@frgfm)
    • Added minimal code snippet in documentation #41 (@frgfm)

    Others

    • Updated version in setup #34 and requirements #61 (@frgfm)
    • Leveraged automatic layer resolution in example script #41 (@frgfm)
    • Updated CI job to run unittests #45 (@frgfm)
    Source code(tar.gz)
    Source code(zip)
    cam_example.png(328.50 KB)
    cam_example_2rows.png(671.00 KB)
    torchcam_demo.png(733.68 KB)
    video_example_wallaby.gif(6.30 MB)
  • v0.1.2(Dec 27, 2020)

    This release adds an implementation of IS-CAM and greatly improves interface.

    Note: torchcam 0.1.2 requires PyTorch 1.1 or newer.

    Highlights

    CAMs

    Implementation of CAM extractor New

    • Add an IS-CAM implementation #13 (@frgfm)
    • Added automatic target layer resolution #32 (@frgfm)

    Improvements

    • Added support for submodule hooking #21 (@pkmandke)

    Fixes

    • Fixed hooking mechanism edge case #23 (@frgfm)

    Test

    Verifications of the package well-being before release New

    • Updated test for torchcam.cams #13, #30 (@frgfm)

    Improvements

    • Removed pretrained model loading in unittests #25 (@frgfm)
    • Switched all models to eval, removed gradient when not required, and changed to simpler models #33 (@frgfm)

    Documentation

    Online resources for potential users New

    • Added entry for IS-CAM #13, #30 (@frgfm)

    Fixes

    • Fixed examples in docstrings of gradient-based CAMs #28, #33 (@frgfm)

    Others

    Other tools and implementations New

    • Added annotation typing to the codebase & mypy verification CI job #19 (@frgfm)
    • Added package publishing verification jobs #12 (@frgfm)

    Improvements

    • Improved example script #15 (@frgfm)
    • Optimized CI cache #20 (@frgfm)

    Fixes

    • Fixed coverage upload job #16 (@frgfm)
    • Fixed doc deployment job #24 (@frgfm)
    • Fixed conda recipe #29 (@frgfm)
    Source code(tar.gz)
    Source code(zip)
    base_snippet.png(72.58 KB)
    overlayed_heatmap.png(227.91 KB)
    raw_heatmap.png(2.41 KB)
    torchcam_demo.png(734.79 KB)
  • v0.1.1(Aug 3, 2020)

    This release adds implementations of SmoothGradCAM++, Score-CAM and SS-CAM.

    Note: torchcam 0.1.1 requires PyTorch 1.1 or newer.

    brought to you by @frgfm

    Highlights

    CAMs

    Implementation of CAM extractor New

    • Add a SmoothGradCAM++ implementation (#4)
    • Add a Score-CAM implementation (#5)
    • Add a SS-CAM implementation (#11).

    Improvements

    • Refactor CAM extractor for better code reusability (#6)

    Test

    Verifications of the package well-being before release New

    • Updated test for torchcam.cams (#4, #5, #11)

    Documentation

    Online resources for potential users Improvements

    • Add detailed explanation of CAM computation (#8, #11)
    • Add websearch referencing of documentation (#7)

    Others

    Other tools and implementations

    • Fixed conda upload job (#3)
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Mar 24, 2020)

    This release adds implementations of CAM, GradCAM and GradCAM++.

    Note: torchcam 0.1.0 requires PyTorch 1.1 or newer.

    brought to you by @frgfm

    Highlights

    GradCAM

    Implementation of gradient-based CAM extractor New

    • Add a CAM implementation (#2)
    • Add Grad-CAM and Grad-CAM++ implementations (#1, #2).

    Test

    Verifications of the package well-being before release New

    • Add test for torchcam.cams (#1, #2)
    • Add test for torschscan.utils (#1)

    Documentation

    Online resources for potential users New

    • Add sphinx automatic documentation build for existing features (#1, #2)
    • Add contribution guidelines (#1)
    • Add installation, usage, and benchmark in readme (#1, #2)

    Others

    Other tools and implementations

    • Add ̀overlay_mask` to easily overlay mask on images (#1).
    Source code(tar.gz)
    Source code(zip)
Owner
F-G Fernandez
Research Engineer in Computer Vision
F-G Fernandez
Tensorflow Implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE

SMU A Tensorflow Implementation of SMU: SMOOTH ACTIVATION FUNCTION FOR DEEP NETWORKS USING SMOOTHING MAXIMUM TECHNIQUE arXiv https://arxiv.org/abs/211

Fuhang 5 Jan 18, 2022
Rational Activation Functions - Replacing Padé Activation Units

Rational Activations - Learnable Rational Activation Functions First introduce as PAU in Padé Activation Units: End-to-end Learning of Activation Func

ml-research@TUDarmstadt 38 Nov 22, 2022
Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

CCAM (Unsupervised) Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localizati

Computer Vision Insitute, SZU 113 Dec 27, 2022
Code for paper: Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks

Group-CAM By Zhang, Qinglong and Rao, Lu and Yang, Yubin [State Key Laboratory for Novel Software Technology at Nanjing University] This repo is the o

zhql 98 Nov 16, 2022
With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function

With this package, you can generate mixed-integer linear programming (MIP) models of trained artificial neural networks (ANNs) using the rectified linear unit (ReLU) activation function. At the moment, only TensorFlow sequential models are supported. Interfaces to either the Pyomo or Gurobi modeling environments are offered.

ChemEngAI 40 Dec 27, 2022
Pytorch implementation of "Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech"

GradTTS Unofficial Pytorch implementation of "Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech" (arxiv) About this repo This is an unoffic

HeyangXue1997 103 Dec 23, 2022
PyTorch implementation of ''Background Activation Suppression for Weakly Supervised Object Localization''.

Background Activation Suppression for Weakly Supervised Object Localization PyTorch implementation of ''Background Activation Suppression for Weakly S

null 35 Jan 6, 2023
Repository for the "Gotta Go Fast When Generating Data with Score-Based Models" paper

Gotta Go Fast When Generating Data with Score-Based Models This repo contains the official implementation for the paper Gotta Go Fast When Generating

Alexia Jolicoeur-Martineau 89 Nov 9, 2022
Generative Autoregressive, Normalized Flows, VAEs, Score-based models (GANVAS)

GANVAS-models This is an implementation of various generative models. It contains implementations of the following: Autoregressive Models: PixelCNN, G

MRSAIL (Mini Robotics, Software & AI Lab) 6 Nov 26, 2022
Callable PyTrees and filtered JIT/grad transformations => neural networks in JAX.

Equinox Callable PyTrees and filtered JIT/grad transformations => neural networks in JAX Equinox brings more power to your model building in JAX. Repr

Patrick Kidger 909 Dec 30, 2022
Official Repsoitory for "Activate or Not: Learning Customized Activation." [CVPR 2021]

CVPR 2021 | Activate or Not: Learning Customized Activation. This repository contains the official Pytorch implementation of the paper Activate or Not

null 184 Dec 27, 2022
ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training

ActNN : Activation Compressed Training This is the official project repository for ActNN: Reducing Training Memory Footprint via 2-Bit Activation Comp

UC Berkeley RISE 178 Jan 5, 2023
FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery (TGRS)

FactSeg: Foreground Activation Driven Small Object Semantic Segmentation in Large-Scale Remote Sensing Imagery by Ailong Ma, Junjue Wang*, Yanfei Zhon

Kingdrone 43 Jan 5, 2023
Source for the paper "Universal Activation Function for machine learning"

Universal Activation Function Tensorflow and Pytorch source code for the paper Yuen, Brosnan, Minh Tu Hoang, Xiaodai Dong, and Tao Lu. "Universal acti

null 4 Dec 3, 2022
Implementation of parameterized soft-exponential activation function.

Soft-Exponential-Activation-Function: Implementation of parameterized soft-exponential activation function. In this implementation, the parameters are

Shuvrajeet Das 1 Feb 23, 2022
Unofficial Tensorflow 2 implementation of the paper Implicit Neural Representations with Periodic Activation Functions

Siren: Implicit Neural Representations with Periodic Activation Functions The unofficial Tensorflow 2 implementation of the paper Implicit Neural Repr

Seyma Yucer 2 Jun 27, 2022
NeuroGen: activation optimized image synthesis for discovery neuroscience

NeuroGen: activation optimized image synthesis for discovery neuroscience NeuroGen is a framework for synthesizing images that control brain activatio

null 3 Aug 17, 2022
Some toy examples of score matching algorithms written in PyTorch

toy_gradlogp This repo implements some toy examples of the following score matching algorithms in PyTorch: ssm-vr: sliced score matching with variance

Ending Hsiao 21 Dec 26, 2022