# Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks (ICCV 2021)

This repository is the official implementation of Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks .

## Contents

This repository contains the following PyTorch code:

• Implementation of U-TAE spatio-temporal encoding architecture for satellite image time series
• Implementation of Parcels-as-Points (PaPs) module for panoptic segmentation of agricultural parcels
• Code for reproduction of the paper's results for panoptic and semantic segmentation.

## Results

Our model achieves the following performance on :

### PASTIS - Panoptic segmentation

Our spatio-temporal encoder U-TAE combined with our PaPs instance segmentation module achieves 40.4 Panoptic Quality (PQ) on PASTIS for panoptic segmentation. When replacing U-TAE with a convolutional LSTM the performance drops to 33.4 PQ.

Model name SQ RQ PQ
U-TAE + PaPs (ours) 81.3 49.2 40.4
UConvLSTM+PaPs 80.9 40.8 33.4

### PASTIS - Semantic segmentation

Our spatio-temporal encoder U-TAE yields a semantic segmentation score of 63.1 mIoU on PASTIS, achieving an improvement of approximately 5 points compared to the best existing methods that we re-implemented (Unet-3d, Unet+ConvLSTM and Feature Pyramid+Unet). See the paper for more details.

Model name #Params OA mIoU
U-TAE (ours) 1.1M 83.2% 63.1%
Unet-3d 1.6M 81.3% 58.4%
Unet-ConvLSTM 1.5M 82.1% 57.8%
FPN-ConvLSTM 1.3M 81.6% 57.1%

## Requirements

### Python requirements

To install requirements:

pip install -r requirements.txt


(torch_scatter is required for the panoptic experiments. Installing this library requires a little more effort, see the official repo)

## Inference with pre-trained models

### Panoptic segmentation

Pre-trained weights of U-TAE+Paps are available here

To perform inference of the pre-trained model on the test set of PASTIS run:

python test_panoptic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER


### Semantic segmentation

Pre-trained weights of U-TAE are available here

To perform inference of the pre-trained model on the test set of PASTIS run:

python test_semantic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER


## Training models from scratch

### Panoptic segmentation

To reproduce the main result for panoptic segmentation (with U-TAE+PaPs) run the following :

python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR


Options are also provided in train_panoptic.py to reproduce the other results of Table 2:

python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_NoCNN --no_mask_conv
python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --backbone uconvlstm
python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_shape24 --shape_size 24


Note: By default this script runs the 5 folds of the cross validation, which can be quite long (~12 hours per fold on a Tesla V100). Use the fold argument to execute one of the 5 folds only (e.g. for the 3rd fold : python train_panoptic.py --fold 3 --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR).

### Semantic segmentation

To reproduce results for semantic segmentation (with U-TAE) run the following :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR


And in order to obtain the results of the competing methods presented in Table 1 :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UNET3d --model unet3d
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --model uconvlstm
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_FPN --model fpn
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BUConvLSTM --model buconvlstm
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_COnvGRU --model convgru
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_ConvLSTM --model convlstm



Finally, to reproduce the ablation study presented in Table 1 :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_MeanAttention --agg_mode att_mean
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SkipMeanConv --agg_mode mean
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BatchNorm --encoder_norm batch
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SingleDate --mono_date "08-01-2019"



### Reference

Please include a citation to the following paper if you use the U-TAE, PaPs or the PASTIS benchmark.

@article{garnot2021panoptic,
title={Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks},
author={Sainte Fare Garnot, Vivien  and Landrieu, Loic },
journal={ICCV},
year={2021}
}



### Credits

• This work was partly supported by ASP, the French Payment Agency.

• Code for the presented methods and dataset is original code by Vivien Sainte Fare Garnot, competing methods and some utility functions were adapted from existing repositories which are credited in the corresponding files.

• #### Optimization problem: missing m_hat parameters

Hi, how is it going? @VSainteuf @loicland

In equation (6), which is apply to extract the centerness map, I would like to ask what is the estimated parameters $\hat{I}{p}$ and $\hat{J}{p}$.

I'm asking since I have implemented this optimization problem and I didn't got those parameters:

def loss(params):
i_measured, j_measured, sigma_h, sigma_w = params
term_i = np.power((i_measured), 2) / np.power((2*sigma_h), 2)
term_j = np.power((j_measured), 2) / np.power((2*sigma_w), 2)

return np.exp(-(term_i + term_j))


Thank you so much!

opened by GustavoMourao 15
• #### multi GPU training problem

When I use multiple GPUs for training, the inconsistent size of tensor types in instance_masks leads to data parallelism errors. How can I solve this error?

opened by MiSsU-HH 7
• #### Runtime error during testing panoptic segmentation (tensors on different devices)

During testing panoptic segmentation, with the following command:

python3 test_panoptic.py --dataset_folder "../PASTIS" --weight_folder "../UTAE_PAPs"


I ran into the following error:

  File "test_panoptic.py", line 142, in <module>
main(config)
File "test_panoptic.py", line 125, in main
device=device,
File "/utae-paps/train_panoptic.py", line 243, in iterate
File "/utae-paps/src/panoptic/metrics.py", line 126, in add
self.cumulative_ious[i] += torch.stack(ious).sum()
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!


It seems that tensor self.cumulative_ious is on cpu while the other tensor is on cuda.

The following change in file /utae-paps/src/panoptic/metrics.py:

   self.cumulative_ious[i] += torch.stack(ious).sum().to(device='cpu')


manages to fix this issue. Kindly confirm the validity of this fix.

Kindly also note that I also tried transferring self.cumulative_ious to cuda, but it runs into errors later during execution.

opened by watch24hrs-iiitd 7
• #### [missing input argment]

https://github.com/VSainteuf/utae-paps/blob/2b9fae182f7271cabe59e5837057c7c1b0b40f39/test_semantic.py#L23

parser.add_argument(
"--model",
default="utae",
type=str,
help="Type of architecture to use. Can be one of: (utae/unet3d/fpn/convlstm/convgru/uconvlstm/buconvlstm)",
)


Since we'll get this error:

AttributeError: 'Namespace' object has no attribute 'model'

opened by GustavoMourao 4
• #### A couple of questions about the data set

Hi, I want to replace the original dataset with my own, but I don't know how some NPY files are generated (e.g. INSTANCE_Id file, it seems to add some boundaries and determine the background), could you explain

opened by Lzqlzq123123 3
• #### How to perform predictions for a single image. (parcels predicted)

Hi,

There is any function/script in the code to perform predictions from the model using a single input image? I want to reproduce the qualitative results of the panoptic process (panoptic mask parcels predict) show in figure 12 in the paper. thanks

opened by alexanderDuenas 3
• #### Data types

Hello! I have a question about the nature of the data if you may. In my case, I have as input data "GeoTiff" and the labels are GeoJson. So my question here is that can I use this type of data in order to train the model, os I should use, strictly, npy and Json as data. And if I can, Is there anything to change in the code? Thank you!

opened by NourSoltani 2

Hey!! NIce paper and well organized code. Thanks for sharing the code. I read your paper and also checked the code but bit confused with the way you are loading data. This is first encounter with sattelite data so may be I am asking silly question. So, when I checked the data its time series and that is fine, but how you use a reference data and use mono date is not very clear to me.My questions are:

1. What is the reference date means here? When I use a mono data that means I am taking data for only that date?
2. If I want to use whole time series data, do I need to give reference date and also mono data as argument?
3. For the panoptic segmentation there is no condition for that in the dataset wraper function. Instance segmentation is same as panoptic? bit confused with that.

Hope you understand my questions.

opened by nazib 2
• #### Question regarding pixel_to_object_mapping (zones) array

Hello, I observed that some pixels characterized as belonging to some parcel in instance_ids array are mapped to a different one in pixel_to_object_mapping. May I ask what was the reasoning behind that?

opened by andrster1 2
• #### Problem with filtering out multiple void instances

I run some tests with your implementation for the panoptic metric and got a curious result.

I made a simple test case with two instance who overlap the void class -> they should get filtered out.

if I have only one of those instances it works properly and I don’t get FP. If I now have both instances they don’t get filtered out and I get two FP

It might have to do with https://github.com/VSainteuf/utae-paps/blob/9e54d3c330b6d207f951796ea1656be8ebfad562/src/panoptic/metrics.py#L55

since you here do the unique over basicly a binary mask if I understand correctly. I supect you wanted to do here instead of void_mask -> instance_ture[batch_idx]

opened by marckatzenmaier 2

Hi, Thank you very much for the visualization code you shared, I am getting the following error when I run it

TypeError                                 Traceback (most recent call last)
Input In [50], in <cell line: 8>()
5 alpha=.5
8 for b in range(batch_size):
9     # Plot S2 background
---> 10     im = get_rgb(x,b=b, t_show=t)
11     axes[b,0].imshow(im)
12     axes[b,2].imshow(im)

Input In [48], in get_rgb(x, b, t_show)
17 def get_rgb(x,b=0,t_show=6):
18     """Gets an observation from a time series and normalises it for visualisation."""
---> 19     im = x[b,t_show,[2,1,0]].cpu().numpy()
20     mx = im.max(axis=(1,2))
21     mi = im.min(axis=(1,2))

TypeError: 'int' object is not subscriptable


How can I fix this error please?

opened by Auroralyxa 2
• #### Background class supervision problem

from the code train_semantic.py: parser.add_argument("--ignore_index", default=-1, type=int) classes=20 weights[config.ignore_index] = 0 iou_meter = IoU( num_classes=config.num_classes, ignore_index=config.ignore_index, cm_device=config.device, ) Background class loss is not calculated during training, and background IoU evaluation is ignored during evaluation?

opened by YangPanHZAU 1
• #### Things and stuff id confusion

Hi @VSainteuf, I was trying to train detectron panoptic segmentation model using PASTIS dataset. We are basically doing an eavluation of five different panoptic segmentation models for multispectrum sattelite data. We choose PASTIS as our primary data since its clean and well curated, (Thanks for clean data buildup). Most of themodels we choose can easily be trained using coco format. We decided to make a coco formated jeson file for PASTIS so that we can easily train those models on PASTIS. But here is a thing which confuses use while creating jeson in coco panoptic format. COCO format requires 'stuff" id and 'things' id in its format. When I check your paper and code we can easily relate the nomenclature you provided as stuff id or category_id but how should we define the instance id. I checked the code in your data loader and seems like 'zone' files are acting as instance id. Can you pls clarify how to map 19 class category_id to things id or 'id' in coco format.?

kind regards

opened by nazib 0
###### Time-series-deep-learning - Developing Deep learning LSTM, BiLSTM models, and NeuralProphet for multi-step time-series forecasting of stock price.

Stock Price Prediction Using Deep Learning Univariate Time Series Predicting stock price using historical data of a company using Neural networks for

7 Nov 27, 2022
###### An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

11 Nov 25, 2022
###### Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020). This codebase contains the essential ingredients of PCV, including various spatial discretizati

23 Oct 25, 2022
###### [CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021) PyTorch implementation for EOPSN. We propose open-set panoptic segmentation t

48 Oct 28, 2022
###### [CVPR 2021] Forecasting the panoptic segmentation of future video frames

Panoptic Segmentation Forecasting Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021 [Link to paper] We propose

43 Oct 28, 2022
###### PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

61 Nov 30, 2022
###### A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

49 Sep 22, 2022
###### Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Nov 7, 2022
###### Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

359 Nov 7, 2022
###### Implementation for Panoptic-PolarNet (CVPR 2021)

Panoptic-PolarNet This is the official implementation of Panoptic-PolarNet. [ArXiv paper] Introduction Panoptic-PolarNet is a fast and robust LiDAR po

122 Nov 3, 2022
###### [ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Interactive Scene Reconstruction Project Page | Paper This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D

91 Nov 28, 2022
###### This project is a loose implementation of paper "Algorithmic Financial Trading with Deep Convolutional Neural Networks: Time Series to Image Conversion Approach"

Stock Market Buy/Sell/Hold prediction Using convolutional Neural Network This repo is an attempt to implement the research paper titled "Algorithmic F

135 Dec 2, 2022
###### PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Soft DTW Loss Function for PyTorch in CUDA This is a Pytorch Implementation of Soft-DTW: a Differentiable Loss Function for Time-Series which is batch

75 Nov 9, 2022
###### Implementation of ETSformer, state of the art time-series Transformer, in Pytorch

ETSformer - Pytorch Implementation of ETSformer, state of the art time-series Transformer, in Pytorch Install \$ pip install etsformer-pytorch Usage im

117 Nov 28, 2022
###### 1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

206 Nov 24, 2022
###### Multiband spectro-radiometric satellite image analysis with K-means cluster algorithm

Multi-band Spectro Radiomertric Image Analysis with K-means Cluster Algorithm Overview Multi-band Spectro Radiomertric images are images comprising of

6 Mar 16, 2022