Band-Adaptive Spectral-Spatial Feature Learning Neural Network for Hyperspectral Image Classification

Overview

This repository has been ⛔️ DEPRECATED. Please take a look at our fairly recent work:

Band-Adaptive Spectral-Spatial Feature Learning Neural Network for Hyperspectral Image Classification [paper] [Code]

Deep Learning for Land-cover Classification in Hyperspectral Images

Hyperspectral images are images captured in multiple bands of the electromagnetic spectrum. This project is focussed at the development of Deep Learned Artificial Neural Networks for robust landcover classification in hyperspectral images. Land-cover classification is the task of assigning to every pixel, a class label that represents the type of land-cover present in the location of the pixel. It is an image segmentation/scene labeling task. The following diagram describes the task.



This website describes our explorations with the performance of Multi-Layer Perceptrons and Convolutional Neural Networks at the task of Land-cover Classification in Hyperspectral Images. Currently we perform pixel-wise classification.


Dataset =======

We have performed our experiments on the Indian Pines Dataset. The following are the particulars of the dataset:

  • Source: AVIRIS sensor
  • Region: Indian Pines test site over north-western Indiana
  • Time of the year: June
  • Wavelength range: 0.4 – 2.5 micron
  • Number of spectral bands: 220
  • Size of image: 145x145 pixel
  • Number of land-cover classes: 16

Input data format =================

Each pixel is described by an NxN patch centered at the pixel. N denotes the size of spatial context used for making the inference about a given pixel.

The input data was divided into training set (75%) and a test set (25%).

Hardware used

The neural networks were trained on a machine with dual Intel Xeon E5-2630 v2 CPUs, 32 GB RAM and NVIDIA Tesla K-20C GPU.


Multi-Layer Perceptron

Multi-Layer Perceptron (MLP) is an artificial neural network with one or more hidden layers of neurons. MLP is capable of modelling highly non-linear functions between the input and output and forms the basis of Deep-learning Neural Network (DNN) models.

Architecture of Multi-Layer Perceptron used

input- [affine - relu] x 3 - affine - softmax

(Schematic representation below)

Ndenotes the size of the input patch.


Specifics of the learning algorithm

The following are the details of the learning algorithm used:

  • Parameter update algorithm used: Adagrad

    • Batch size: 200
    • Learning rate: 0.01
  • Number of steps: until best validation performance


Performance

Decoding generated for different input patch sizes:


Convolutional Neural Network

(CNN or ConvNet) are a special category of artificial neural networks designed for processing data with a gridlike structure. The ConvNet architecture is based on sparse interactions and parameter sharing and is highly effective for efficient learning of spatial invariances in images. There are four kinds of layers in a typical ConvNet architecture: convolutional (conv), pooling (pool), fullyconnected (affine) and rectifying linear unit (ReLU). Each convolutional layer transforms one set of feature maps into another set of feature maps by convolution with a set of filters.

Architecture of Convolutional Neural Network used

input- [conv - relu - maxpool] x 2 - [affine - relu] x 2 - affine - softmax

(Schematic representation below)

Ndenotes the size of the input patch.


Specifics of the learning algorithm

The following are the details of the learning algorithm used:

  • Parameter update algorithm used: Adagrad

    • Batch size: 100
    • Learning rate: 0.01
  • Number of steps: until best validation performance


Performance

Decoding generated for different input patch sizes:



Description of the repository

  • IndianPines_DataSet_Preparation_Without_Augmentation.ipynb - does the following operations:

    • Loads the Indian Pines dataset
    • Scales the input between [0,1]
    • Mean normalizes the channels
    • Makes training and test splits
    • Extracts patches of given size
    • Oversamples the training set for balancing the classes
  • Spatial_dataset.py - provides a highly flexible Dataset class for handling the Indian Pines data.

  • patch_size.py - specify the required patch-size here.

  • IndianPinesCNN.ipynb- builds the TensorFlow Convolutional Neural Network and defines the training and evaluation ops:

    • inference() - builds the model as far as is required for running the network forward to make predictions.
    • loss() - adds to the inference model the layers required to generate loss.
    • training() - adds to the loss model the Ops required to generate and apply gradients.
    • evaluation() - calcuates the classification accuracy
  • CNN_feed.ipynb - trains and evaluates the Neural Network using a feed dictionary

  • Decoder_Spatial_CNN.ipynb - generates the landcover classification of an input hyperspectral image for a given trained network

  • IndianPinesMLP.py - builds the TensorFlow Multi-layer Perceptron and defines the training and evaluation ops:

    • inference() - builds the model as far as is required for running the network forward to make predictions.
    • loss() - adds to the inference model the layers required to generate loss.
    • training() - adds to the loss model the Ops required to generate and apply gradients.
    • evaluation() - calcuates the classification accuracy
  • MLP_feed.ipynb - trains and evaluates the MLP using a feed dictionary

  • Decoder_Spatial_MLP.ipynb - generates the landcover classification of an input hyperspectral image for a given trained network

  • credibility.ipynb - summarizes the predictions of an ensemble and produces the land-cover classification and class-wise confusion matrix.


Setting up the experiment

  • Download the Indian Pines data-set from here.
  • Make a directory named Data within the current working directory and copy the downloaded .mat files Indian_pines.mat and Indian_pines_gt.mat in this directory.

In order to make sure all codes run smoothly, you should have the following directory subtree structure under your current working directory:

|-- IndianPines_DataSet_Preparation_Without_Augmentation.ipynb
|-- Decoder_Spatial_CNN.ipynb
|-- Decoder_Spatial_MLP.ipynb
|-- IndianPinesCNN.ipynb
|-- CNN_feed.ipynb
|-- MLP_feed.ipynb
|-- credibility.ipynb
|-- IndianPinesCNN.py
|-- IndianPinesMLP.py
|-- Spatial_dataset.py
|-- patch_size.py
|-- Data
|   |-- Indian_pines_gt.mat
|   |-- Indian_pines.mat


  • Set the required patch-size value (eg. 11, 21, etc) in patch_size.py and run the following notebooks in order:
    1. IndianPines_DataSet_Preparation_Without_Augmentation.ipynb
    2. CNN_feed.ipynb OR MLP_feed.ipynb (specify the number of fragments in the training and test data in the variables TRAIN_FILES and TEST_FILES)
    3. Decoder_Spatial_CNN.ipynb OR Decoder_Spatial_MLP.ipynb (set the required checkpoint to be used for decoding in the model_name variable)

Outputs will be displayed in the notebooks.


Acknowledgement

This repository was developed by Anirban Santara, Ankit Singh, Pranoot Hatwar and Kaustubh Mani under the supervision of Prof. Pabitra Mitra during June-July, 2016 at the Department of Computer Science and Engineering, Indian Institute of Technology Kharagpur, India. The project is funded by Satellite Applications Centre, Indian Space Research Organization (SAC-ISRO).

Issues
  • ValueError: Only call `sparse_softmax_cross_entropy_with_logits` with named arguments (labels=..., logits=..., ...)

    ValueError: Only call `sparse_softmax_cross_entropy_with_logits` with named arguments (labels=..., logits=..., ...)

    Hi, thanks for your code. When I run the "CNN_feed"or "MLP_feed", I meet this error, and I cannot fix it. Do you know the solution?Can you help me? Thanks very much. My platform is Ubuntu 16.04, Anaconda2(python2.7), tensorflow1.0.1.

    opened by Strider20151 1
  • ValueError: Only call `sparse_softmax_cross_entropy_with_logits` with named arguments (labels=..., logits=..., ...)

    ValueError: Only call `sparse_softmax_cross_entropy_with_logits` with named arguments (labels=..., logits=..., ...)

    I am trying to implement Deep Learning for Land-cover Classification in Hyper spectral Images

    In CNN_feed.ipynb file line number-9 run_training() function the following error encountered

    ValueError: Only call sparse_softmax_cross_entropy_with_logits with named arguments (labels=..., logits=..., ...)

    Thank you in advance for your help.

    opened by SimoneDeGasperis 0
  • Proposed Solution for Issue #2

    Proposed Solution for Issue #2

    I faced the same error while using the project. Thanks to @ishiry for mentioning it in Issue 2 The error occurs because default devision in python outputs a float. At line number 6 in In[8] curr_tar = target_mat[i + (PATCH_SIZE - 1)/2, j + (PATCH_SIZE - 1)/2] the division by 2 operation creates a float output which is not accepted by target_mat[] variable because it needs integer as indexes. I have rectified the issue by adding explicit type conversion for the float output to integer by making the line curr_tar = target_mat[i + int((PATCH_SIZE - 1)/2), j + int((PATCH_SIZE - 1)/2)]\n After making the change, the code is running fine on my machine

    @Project-Admin please review the changes and accept the PR if it is correct.

    opened by thisisashukla 0
  • 您好,在运行CNN-feed.py这个代码时遇到了问题,恳请获得帮助。

    您好,在运行CNN-feed.py这个代码时遇到了问题,恳请获得帮助。

    您好,在运行CNN-feed.py这个代码时遇到了问题,恳请获得帮助。 ValueError: Cannot feed value of shape (100, 24200) for Tensor 'Placeholder:0', which has shape '(100, 26620)'

    Originally posted by @wjks-zxy in https://github.com/KGPML/Hyperspectral/issues/1#issuecomment-529747444

    opened by wjks-zxy 0
  • TypeError: 'float' object cannot be interpreted as an integer

    TypeError: 'float' object cannot be interpreted as an integer

    I have taken patch_size=21 in the "patch_size.py" file (as suggested in the README.md) and the length of train_patches are obtained as follows: 100

    Now upon running the following code I get the error as given below: 101

    What should I do?

    opened by saikosis 1
  • "ValueError: Only call `sparse_softmax_cross_entropy_with_logits` with named arguments" encountered in the training loop

    Hi,

    I'm using TF1.3 with Cuda 8.0 and CuDNN 6.0. May I know if this error could be caused by incompatible Tensorflow version? If so, could you share with us the version of TF/Cuda/CuDnn that you used to run the code?

    image

    opened by bernt73 3
  • Fail to run Decoder_Spatial_MLP.ipynb

    Fail to run Decoder_Spatial_MLP.ipynb

    when I run Decoder_Spatial_MLP.ipynb, errors arise like "NotFoundError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to find any matching files for model/model-MLP-1X1.ckpt-49999" . I am wondering how to solve it?

    opened by daifeng2016 0
  • add samples

    add samples

    rotating the original samples by 90°, 180°and 270° and flipping the data where the original samples are flipped along the horizontal and vertical directions can increase the number of training data. But a question is that the label should remain unchanged or changed ?

    opened by FunkyGod 0
The implementation of "Optimizing Shoulder to Shoulder: A Coordinated Sub-Band Fusion Model for Real-Time Full-Band Speech Enhancement"

SF-Net for fullband SE This is the repo of the manuscript "Optimizing Shoulder to Shoulder: A Coordinated Sub-Band Fusion Model for Real-Time Full-Ban

Guochen Yu 28 Jul 27, 2022
paper: Hyperspectral Remote Sensing Image Classification Using Deep Convolutional Capsule Network

DC-CapsNet This is a tensorflow and keras based implementation of DC-CapsNet for HSI in the Remote Sensing Letters R. Lei et al., "Hyperspectral Remot

LEI 6 May 5, 2022
FPGA: Fast Patch-Free Global Learning Framework for Fully End-to-End Hyperspectral Image Classification

FPGA & FreeNet Fast Patch-Free Global Learning Framework for Fully End-to-End Hyperspectral Image Classification by Zhuo Zheng, Yanfei Zhong, Ailong M

Zhuo Zheng 85 Jul 30, 2022
Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot. Graph Convolutional Networks for Hyperspectral Image Classification, IEEE TGRS, 2021.

Graph Convolutional Networks for Hyperspectral Image Classification Danfeng Hong, Lianru Gao, Jing Yao, Bing Zhang, Antonio Plaza, Jocelyn Chanussot T

Danfeng Hong 137 Aug 12, 2022
Semi-Supervised Graph Prototypical Networks for Hyperspectral Image Classification, IGARSS, 2021.

Semi-Supervised Graph Prototypical Networks for Hyperspectral Image Classification, IGARSS, 2021. Bobo Xi, Jiaojiao Li, Yunsong Li and Qian Du. Code f

Bobo Xi 6 Jun 2, 2022
PyTorch implementation of our method for adversarial attacks and defenses in hyperspectral image classification.

Self-Attention Context Network for Hyperspectral Image Classification PyTorch implementation of our method for adversarial attacks and defenses in hyp

null 21 Jul 31, 2022
Spectralformer: Rethinking hyperspectral image classification with transformers

The code in this toolbox implements the "Spectralformer: Rethinking hyperspectral image classification with transformers". More specifically, it is detailed as follow.

Danfeng Hong 70 Aug 4, 2022
Deep learning toolbox based on PyTorch for hyperspectral data classification.

Deep learning toolbox based on PyTorch for hyperspectral data classification.

Nicolas 280 Aug 14, 2022
Graph Regularized Residual Subspace Clustering Network for hyperspectral image clustering

Graph Regularized Residual Subspace Clustering Network for hyperspectral image clustering

Yaoming Cai 5 Jul 18, 2022
Spectral Temporal Graph Neural Network (StemGNN in short) for Multivariate Time-series Forecasting

Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting This repository is the official implementation of Spectral Temporal Gr

Microsoft 260 Aug 6, 2022
A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering.

DeepFilterNet A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) based on Deep Filtering. libDF contains Rust code used for dat

Hendrik Schröter 152 Aug 10, 2022
Spatial Contrastive Learning for Few-Shot Classification (SCL)

This repo contains the official implementation of Spatial Contrastive Learning for Few-Shot Classification (SCL), which presents of a novel contrastive learning method applied to few-shot image classification in order to learn more general purpose embeddings, and facilitate the test-time adaptation to novel visual categories.

Yassine 26 Jul 30, 2022
Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

Xavier Bresson 288 Jul 9, 2022
This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection, built on SECOND.

3D-CVF This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object

YecheolKim 95 Jul 20, 2022
Code for paper Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting

Decoupled Spatial-Temporal Graph Neural Networks Code for our paper: Decoupled Dynamic Spatial-Temporal Graph Neural Network for Traffic Forecasting.

S22 10 Aug 3, 2022
Unofficial pytorch implementation of the paper "Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution"

DFSA Unofficial pytorch implementation of the ICCV 2021 paper "Dynamic High-Pass Filtering and Multi-Spectral Attention for Image Super-Resolution" (p

null 2 Nov 15, 2021
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022) https://arxiv.org/abs/2203.09388 Jianqi Ma, Zheto

MA Jianqi, shiki 83 Aug 1, 2022
This is an official PyTorch implementation of Task-Adaptive Neural Network Search with Meta-Contrastive Learning (NeurIPS 2021, Spotlight).

NeurIPS 2021 (Spotlight): Task-Adaptive Neural Network Search with Meta-Contrastive Learning This is an official PyTorch implementation of Task-Adapti

Wonyong Jeong 10 Mar 17, 2022
Scripts for training an AI to play the endless runner Subway Surfers using a supervised machine learning approach by imitation and a convolutional neural network (CNN) for image classification

About subwAI subwAI - a project for training an AI to play the endless runner Subway Surfers using a supervised machine learning approach by imitation

null 77 Aug 6, 2022