Deep Inside Convolutional Networks - This is a caffe implementation to visualize the learnt model

Last update: Apr 15, 2022

Overview

Deep Inside Convolutional Networks

This is a caffe implementation to visualize the learnt model.

Part of a class project at Georgia Tech
Problem Statement Pdf

Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: Visualising image classification models and saliency maps Pdf

###Results:

Class Model visualization of Cat
In this exercise, we will use the method suggested in the “Deep inside convolutional networks: Visualising image classification models and saliency maps” to visualize the class model learnt by a convolutional network. We will use caffe for this exercise and visualize the class model learnt by the “bvlc_reference_caffenet.caffemodel”. Another aspect pointed out by the paper is that, the unnormalized Image score needs to be maximized instead of the probability. For this reason, we will be drop the final softmax layer(as the output here is the probability) and maximize the score at the inner product layer “fc8”.

Class Saliency extraction
The core idea behind this approach is to use the gradients at the image layer for a given image and class, to find the pixels which need to be changed the least i.e, the pixels for which the gradients have the smallest values. Also since our image is a 3 channel image, for each pixel, there will three different gradients. The maximum of these three will be considered the class saliency extraction.

Understanding backpropagation Here we simply visuzlize the gradients at different layers

Instructions:

Install Caffe-rc2 tag
Copy the deploy_fc8.prototxt file to /models/bvlc_reference_caffenet/
Copy all the py files to /examples/
the just run python visualize.py

Comments

could not broadcast input array from shape (1000,1,1) into shape (1,1000)

When i run the visualize.py code, i got the above error as follows:-

Traceback (most recent call last): File "visualize.py", line 69, in bw = net.backward(**{net.outputs[0]: caffeLabel}) File "../python/caffe/pycaffe.py", line 167, in _Net_backward self.blobs[top].diff[...] = diff ValueError: could not broadcast input array from shape (1000,1,1) into shape (1,1000)

and when i change the net.backward line as: bw = net.backward(), it runs perfectly. please tell me what is the problem??

opened by gauticse2015 6
IndexError: list index out of range

Traceback (most recent call last): File "class_saliency_extraction.py", line 25, in image_dims=(224, 224)) File "D:/Projects/caffe-windows/caffe/python\caffe\classifier.py", line 29, in init in_ = self.inputs[0] IndexError: list index out of range

opened by billhhh 3
Absolute value of gradients

As described in Section 3.1 of the paper, you need to take the absolute value of gradients before normalizing it. If you are not doing it, then you are not visualizing those pixels which decrease the class score the most with a small change (this is also a important descriptor).

opened by saiprabhakar 3
ValueError: could not broadcast input array from shape(1000,1,1) into shape (1,1000)

hi, I am trying to use your code for the same purpose. But there is a error in bw=net.backward(**{net.outputs[0]: caffeLabel}) ValueError: could not broadcast input array from shape(1000,1,1) into shape (1,1000). Can you tell me the reason. Thank you. Looking forward to your reply.

opened by zhoafan 2
how to set the Hyperparameter?

In visualize.py file, I see the learning_rate is 10000,it confused me!why doing this, also,In the optimization procedure,The sentence "caffe_data = caffe_data + learning_rate*diff " seems not used any L2 Paradigm，but in the paper, It is not like this! thanks a lot!

opened by kilakila-heart 0
How to use it with multiple regression outputs.

Hi, I am using it for a multi-task model. I am using a customized alexnet for a multi-task model for aesthetics prediction and classification. I have a normalized score ( 0 to 1 ) for aesthetics prediction and a single label for classification. I want to check which areas are more salient for aesthetics. After doing net.forward() I am setting net.blobs['aesthetics'].data[0][0] = 1 Then I am running bw = net.backward() But I am getting only zero values as input image. I have set force_backward: true in my model prototxt file. I think I am messing up the backward function, let me know if you have any idea about how to use it for regression problems.

opened by gautamMalu 1

Deep Inside Convolutional Networks - This is a caffe implementation to visualize the learnt model

Related tags

Overview

Deep Inside Convolutional Networks

Instructions:

Comments

could not broadcast input array from shape (1000,1,1) into shape (1,1000)

IndexError: list index out of range

Absolute value of gradients

ValueError: could not broadcast input array from shape(1000,1,1) into shape (1,1000)

how to set the Hyperparameter?

How to use it with multiple regression outputs.

Owner

Jigar

MMdnn is a set of tools to help users inter-operate among different deep learning frameworks. E.g. model conversion and visualization. Convert models between Caffe, Keras, MXNet, Tensorflow, CNTK, PyTorch Onnx and CoreML.

Caffe: a fast open framework for deep learning.

hipCaffe: the HIP port of Caffe

a reimplementation of LiteFlowNet in PyTorch that matches the official Caffe version

The original weights of some Caffe models, ported to PyTorch.

Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.

A PyTorch implementation of " EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks."

This project is a loose implementation of paper "Algorithmic Financial Trading with Deep Convolutional Neural Networks: Time Series to Image Conversion Approach"

A PyTorch implementation of "Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks" (KDD 2019).

A PyTorch implementation of Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks

Visualize Camera's Pose Using Extrinsic Parameter by Plotting Pyramid Model on 3D Space

Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

Code for paper: Group-CAM: Group Score-Weighted Visual Explanations for Deep Convolutional Networks

Generating Anime Images by Implementing Deep Convolutional Generative Adversarial Networks paper

PyTorch framework, for reproducing experiments from the paper Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

Cancer-and-Tumor-Detection-Using-Inception-model - In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks, specifically here the Inception model by google.

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

This is a model made out of Neural Network specifically a Convolutional Neural Network model

Allows including an action inside another action (by preprocessing the Yaml file). This is how composite actions should have worked.