PyTorch implementation of U-TAE and PaPs for satellite image time series panoptic segmentation.

Overview

Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks (ICCV 2021)

This repository is the official implementation of Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks .

PWC

PWC

Contents

This repository contains the following PyTorch code:

  • Implementation of U-TAE spatio-temporal encoding architecture for satellite image time series UTAE
  • Implementation of Parcels-as-Points (PaPs) module for panoptic segmentation of agricultural parcels PaPs
  • Code for reproduction of the paper's results for panoptic and semantic segmentation.

Results

Our model achieves the following performance on :

PASTIS - Panoptic segmentation

Our spatio-temporal encoder U-TAE combined with our PaPs instance segmentation module achieves 40.4 Panoptic Quality (PQ) on PASTIS for panoptic segmentation. When replacing U-TAE with a convolutional LSTM the performance drops to 33.4 PQ.

Model name SQ RQ PQ
U-TAE + PaPs (ours) 81.3 49.2 40.4
UConvLSTM+PaPs 80.9 40.8 33.4

PASTIS - Semantic segmentation

Our spatio-temporal encoder U-TAE yields a semantic segmentation score of 63.1 mIoU on PASTIS, achieving an improvement of approximately 5 points compared to the best existing methods that we re-implemented (Unet-3d, Unet+ConvLSTM and Feature Pyramid+Unet). See the paper for more details.

Model name #Params OA mIoU
U-TAE (ours) 1.1M 83.2% 63.1%
Unet-3d 1.6M 81.3% 58.4%
Unet-ConvLSTM 1.5M 82.1% 57.8%
FPN-ConvLSTM 1.3M 81.6% 57.1%

Requirements

PASTIS Dataset download

The Dataset is freely available for download here.

Python requirements

To install requirements:

pip install -r requirements.txt

(torch_scatter is required for the panoptic experiments. Installing this library requires a little more effort, see the official repo)

Inference with pre-trained models

Panoptic segmentation

Pre-trained weights of U-TAE+Paps are available here

To perform inference of the pre-trained model on the test set of PASTIS run:

python test_panoptic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER

Semantic segmentation

Pre-trained weights of U-TAE are available here

To perform inference of the pre-trained model on the test set of PASTIS run:

python test_semantic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER

Training models from scratch

Panoptic segmentation

To reproduce the main result for panoptic segmentation (with U-TAE+PaPs) run the following :

python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR

Options are also provided in train_panoptic.py to reproduce the other results of Table 2:

python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_NoCNN --no_mask_conv
python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --backbone uconvlstm
python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_shape24 --shape_size 24

Note: By default this script runs the 5 folds of the cross validation, which can be quite long (~12 hours per fold on a Tesla V100). Use the fold argument to execute one of the 5 folds only (e.g. for the 3rd fold : python train_panoptic.py --fold 3 --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR).

Semantic segmentation

To reproduce results for semantic segmentation (with U-TAE) run the following :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR

And in order to obtain the results of the competing methods presented in Table 1 :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UNET3d --model unet3d
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --model uconvlstm
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_FPN --model fpn
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BUConvLSTM --model buconvlstm
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_COnvGRU --model convgru
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_ConvLSTM --model convlstm

Finally, to reproduce the ablation study presented in Table 1 :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_MeanAttention --agg_mode att_mean
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SkipMeanConv --agg_mode mean
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BatchNorm --encoder_norm batch
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SingleDate --mono_date "08-01-2019"

Reference

Please include a citation to the following paper if you use the U-TAE, PaPs or the PASTIS benchmark.

@article{garnot2021panoptic,
  title={Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks},
  author={Sainte Fare Garnot, Vivien  and Landrieu, Loic },
  journal={ICCV},
  year={2021}
}

Credits

  • This work was partly supported by ASP, the French Payment Agency.

  • Code for the presented methods and dataset is original code by Vivien Sainte Fare Garnot, competing methods and some utility functions were adapted from existing repositories which are credited in the corresponding files.

Comments
  • Optimization problem: missing m_hat parameters

    Optimization problem: missing m_hat parameters

    Hi, how is it going? @VSainteuf @loicland

    In equation (6), which is apply to extract the centerness map, I would like to ask what is the estimated parameters $\hat{I}{p}$ and $\hat{J}{p}$.

    I'm asking since I have implemented this optimization problem and I didn't got those parameters:

    def loss(params):
        i_measured, j_measured, sigma_h, sigma_w = params
        term_i = np.power((i_measured), 2) / np.power((2*sigma_h), 2)
        term_j = np.power((j_measured), 2) / np.power((2*sigma_w), 2)
    
        return np.exp(-(term_i + term_j))
    

    Thank you so much!

    opened by GustavoMourao 15
  • multi GPU training problem

    multi GPU training problem

    When I use multiple GPUs for training, the inconsistent size of tensor types in instance_masks leads to data parallelism errors. How can I solve this error?

    opened by MiSsU-HH 7
  • Runtime error during testing panoptic segmentation (tensors on different devices)

    Runtime error during testing panoptic segmentation (tensors on different devices)

    During testing panoptic segmentation, with the following command:

    python3 test_panoptic.py --dataset_folder "../PASTIS" --weight_folder "../UTAE_PAPs"
    

    I ran into the following error:

      File "test_panoptic.py", line 142, in <module>
        main(config)
      File "test_panoptic.py", line 125, in main
        device=device,
      File "/utae-paps/train_panoptic.py", line 243, in iterate
        pano_meter.add(predictions, y)
      File "/utae-paps/src/panoptic/metrics.py", line 126, in add
        self.cumulative_ious[i] += torch.stack(ious).sum()
    RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
    

    It seems that tensor self.cumulative_ious is on cpu while the other tensor is on cuda.

    The following change in file /utae-paps/src/panoptic/metrics.py:

       self.cumulative_ious[i] += torch.stack(ious).sum().to(device='cpu')
    

    manages to fix this issue. Kindly confirm the validity of this fix.

    Kindly also note that I also tried transferring self.cumulative_ious to cuda, but it runs into errors later during execution.

    opened by watch24hrs-iiitd 7
  • [missing input argment]

    [missing input argment]

    https://github.com/VSainteuf/utae-paps/blob/2b9fae182f7271cabe59e5837057c7c1b0b40f39/test_semantic.py#L23

    Please, add:

    parser.add_argument(
        "--model",
        default="utae",
        type=str,
        help="Type of architecture to use. Can be one of: (utae/unet3d/fpn/convlstm/convgru/uconvlstm/buconvlstm)",
    )
    

    Since we'll get this error:

    AttributeError: 'Namespace' object has no attribute 'model'
    
    opened by GustavoMourao 4
  • A couple of questions about the data set

    A couple of questions about the data set

    Hi, I want to replace the original dataset with my own, but I don't know how some NPY files are generated (e.g. INSTANCE_Id file, it seems to add some boundaries and determine the background), could you explain

    opened by Lzqlzq123123 3
  • How to perform predictions for a single image. (parcels predicted)

    How to perform predictions for a single image. (parcels predicted)

    Hi,

    There is any function/script in the code to perform predictions from the model using a single input image? I want to reproduce the qualitative results of the panoptic process (panoptic mask parcels predict) show in figure 12 in the paper. thanks

    opened by alexanderDuenas 3
  • Data types

    Data types

    Hello! I have a question about the nature of the data if you may. In my case, I have as input data "GeoTiff" and the labels are GeoJson. So my question here is that can I use this type of data in order to train the model, os I should use, strictly, npy and Json as data. And if I can, Is there anything to change in the code? Thank you!

    opened by NourSoltani 2
  • Data loading logic

    Data loading logic

    Hey!! NIce paper and well organized code. Thanks for sharing the code. I read your paper and also checked the code but bit confused with the way you are loading data. This is first encounter with sattelite data so may be I am asking silly question. So, when I checked the data its time series and that is fine, but how you use a reference data and use mono date is not very clear to me.My questions are:

    1. What is the reference date means here? When I use a mono data that means I am taking data for only that date?
    2. If I want to use whole time series data, do I need to give reference date and also mono data as argument?
    3. For the panoptic segmentation there is no condition for that in the dataset wraper function. Instance segmentation is same as panoptic? bit confused with that.

    Hope you understand my questions.

    opened by nazib 2
  • Question regarding pixel_to_object_mapping (zones) array

    Question regarding pixel_to_object_mapping (zones) array

    Hello, I observed that some pixels characterized as belonging to some parcel in instance_ids array are mapped to a different one in pixel_to_object_mapping. May I ask what was the reasoning behind that?

    opened by andrster1 2
  • Problem with filtering out multiple void instances

    Problem with filtering out multiple void instances

    I run some tests with your implementation for the panoptic metric and got a curious result.

    I made a simple test case with two instance who overlap the void class -> they should get filtered out.

    if I have only one of those instances it works properly and I don’t get FP. If I now have both instances they don’t get filtered out and I get two FP

    It might have to do with https://github.com/VSainteuf/utae-paps/blob/9e54d3c330b6d207f951796ea1656be8ebfad562/src/panoptic/metrics.py#L55

    since you here do the unique over basicly a binary mask if I understand correctly. I supect you wanted to do here instead of void_mask -> instance_ture[batch_idx]

    opened by marckatzenmaier 2
  • Question about visualization

    Question about visualization

    Hi, Thank you very much for the visualization code you shared, I am getting the following error when I run it

    TypeError                                 Traceback (most recent call last)
    Input In [50], in <cell line: 8>()
          5 alpha=.5
          8 for b in range(batch_size):
          9     # Plot S2 background
    ---> 10     im = get_rgb(x,b=b, t_show=t)
         11     axes[b,0].imshow(im)
         12     axes[b,2].imshow(im)
    
    Input In [48], in get_rgb(x, b, t_show)
         17 def get_rgb(x,b=0,t_show=6):
         18     """Gets an observation from a time series and normalises it for visualisation."""
    ---> 19     im = x[b,t_show,[2,1,0]].cpu().numpy()
         20     mx = im.max(axis=(1,2))
         21     mi = im.min(axis=(1,2))   
    
    TypeError: 'int' object is not subscriptable
    

    How can I fix this error please?

    opened by Auroralyxa 2
  • Inference

    Inference

    Greetings Mr @VSainteuf

    During the inference, I've been following the instructions in order to do panoptic segmentation. However, after I wrote the instruction (python test_panoptic.py --dataset_folder data --weight_folder weights/UTAE_PAPs --res_dir Output) It prompted this: Traceback (most recent call last): File "test_panoptic.py", line 148, in main(config) File "test_panoptic.py", line 97, in main dt_test = PASTIS_Dataset(**dt_args, folds=test_fold) File "/home/nextav/LULC/U-TAE-main/src/dataset.py", line 80, in init if "-" in mono_date TypeError: argument of type 'NoneType' is not iterable

    So, I made some modifications; More precisely I changed the If condition by: mono_date=config.mono_date; I'm sure it's not pretty right, but it works :) After some minutes of doing inference(and the gpu memory reached 10GB), I've faced this new error, and the Output folder stills empty:

    Step [100/124], Loss: 7.8338, SQ 81.0, RQ 53.0 , PQ 43.4 Epoch time : 289.6s Traceback (most recent call last): File "test_panoptic.py", line 148, in main(config) File "test_panoptic.py", line 134, in main save_results(fold + 1, test_metrics, tables, config) File "/home/nextav/LULC/U-TAE-main/train_panoptic.py", line 329, in save_results with open( FileNotFoundError: [Errno 2] No such file or directory: 'Output/Fold_1/test_metrics.json'

    Do you have any idea for where the error could be? And thank you in advance.

    opened by NourSoltani 0
  • Background class supervision problem

    Background class supervision problem

    from the code train_semantic.py: parser.add_argument("--ignore_index", default=-1, type=int) classes=20 weights[config.ignore_index] = 0 iou_meter = IoU( num_classes=config.num_classes, ignore_index=config.ignore_index, cm_device=config.device, ) Background class loss is not calculated during training, and background IoU evaluation is ignored during evaluation?

    opened by YangPanHZAU 1
  • Things and stuff id confusion

    Things and stuff id confusion

    Hi @VSainteuf, I was trying to train detectron panoptic segmentation model using PASTIS dataset. We are basically doing an eavluation of five different panoptic segmentation models for multispectrum sattelite data. We choose PASTIS as our primary data since its clean and well curated, (Thanks for clean data buildup). Most of themodels we choose can easily be trained using coco format. We decided to make a coco formated jeson file for PASTIS so that we can easily train those models on PASTIS. But here is a thing which confuses use while creating jeson in coco panoptic format. COCO format requires 'stuff" id and 'things' id in its format. When I check your paper and code we can easily relate the nomenclature you provided as stuff id or category_id but how should we define the instance id. I checked the code in your data loader and seems like 'zone' files are acting as instance id. Can you pls clarify how to map 19 class category_id to things id or 'id' in coco format.?

    kind regards

    opened by nazib 0
Owner
null
Time-series-deep-learning - Developing Deep learning LSTM, BiLSTM models, and NeuralProphet for multi-step time-series forecasting of stock price.

Stock Price Prediction Using Deep Learning Univariate Time Series Predicting stock price using historical data of a company using Neural networks for

Abdultawwab Safarji 7 Nov 27, 2022
An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

Abhinav Atrishi 11 Nov 25, 2022
Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020). This codebase contains the essential ingredients of PCV, including various spatial discretizati

Haochen 23 Oct 25, 2022
[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021) PyTorch implementation for EOPSN. We propose open-set panoptic segmentation t

Jaedong Hwang 49 Dec 30, 2022
[CVPR 2021] Forecasting the panoptic segmentation of future video frames

Panoptic Segmentation Forecasting Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021 [Link to paper] We propose

Niantic Labs 44 Nov 29, 2022
PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

null 63 Dec 16, 2022
A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

Biomedical Computer Vision @ Uniandes 52 Dec 19, 2022
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 360 Dec 10, 2022
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 359 Jan 5, 2023
Implementation for Panoptic-PolarNet (CVPR 2021)

Panoptic-PolarNet This is the official implementation of Panoptic-PolarNet. [ArXiv paper] Introduction Panoptic-PolarNet is a fast and robust LiDAR po

Zixiang Zhou 126 Jan 1, 2023
[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Interactive Scene Reconstruction Project Page | Paper This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D

null 97 Dec 28, 2022
This project is a loose implementation of paper "Algorithmic Financial Trading with Deep Convolutional Neural Networks: Time Series to Image Conversion Approach"

Stock Market Buy/Sell/Hold prediction Using convolutional Neural Network This repo is an attempt to implement the research paper titled "Algorithmic F

Asutosh Nayak 136 Dec 28, 2022
PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Soft DTW Loss Function for PyTorch in CUDA This is a Pytorch Implementation of Soft-DTW: a Differentiable Loss Function for Time-Series which is batch

Keon Lee 76 Dec 20, 2022
Implementation of ETSformer, state of the art time-series Transformer, in Pytorch

ETSformer - Pytorch Implementation of ETSformer, state of the art time-series Transformer, in Pytorch Install $ pip install etsformer-pytorch Usage im

Phil Wang 121 Dec 30, 2022
1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

Lihe Yang 209 Jan 1, 2023
Multiband spectro-radiometric satellite image analysis with K-means cluster algorithm

Multi-band Spectro Radiomertric Image Analysis with K-means Cluster Algorithm Overview Multi-band Spectro Radiomertric images are images comprising of

Chibueze Henry 6 Mar 16, 2022
SatelliteNeRF - PyTorch-based Neural Radiance Fields adapted to satellite domain

SatelliteNeRF PyTorch-based Neural Radiance Fields adapted to satellite domain.

Kai Zhang 46 Nov 20, 2022
U-Net Implementation: Convolutional Networks for Biomedical Image Segmentation" using the Carvana Image Masking Dataset in PyTorch

U-Net Implementation By Christopher Ley This is my interpretation and implementation of the famous paper "U-Net: Convolutional Networks for Biomedical

Christopher Ley 1 Jan 6, 2022
tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

timeseriesAI 2.8k Jan 8, 2023