PyTorch implementation of U-TAE and PaPs for satellite image time series panoptic segmentation.

Overview

Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks (ICCV 2021)

This repository is the official implementation of Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks .

PWC

PWC

Contents

This repository contains the following PyTorch code:

  • Implementation of U-TAE spatio-temporal encoding architecture for satellite image time series UTAE
  • Implementation of Parcels-as-Points (PaPs) module for panoptic segmentation of agricultural parcels PaPs
  • Code for reproduction of the paper's results for panoptic and semantic segmentation.

Results

Our model achieves the following performance on :

PASTIS - Panoptic segmentation

Our spatio-temporal encoder U-TAE combined with our PaPs instance segmentation module achieves 40.4 Panoptic Quality (PQ) on PASTIS for panoptic segmentation. When replacing U-TAE with a convolutional LSTM the performance drops to 33.4 PQ.

Model name SQ RQ PQ
U-TAE + PaPs (ours) 81.3 49.2 40.4
UConvLSTM+PaPs 80.9 40.8 33.4

PASTIS - Semantic segmentation

Our spatio-temporal encoder U-TAE yields a semantic segmentation score of 63.1 mIoU on PASTIS, achieving an improvement of approximately 5 points compared to the best existing methods that we re-implemented (Unet-3d, Unet+ConvLSTM and Feature Pyramid+Unet). See the paper for more details.

Model name #Params OA mIoU
U-TAE (ours) 1.1M 83.2% 63.1%
Unet-3d 1.6M 81.3% 58.4%
Unet-ConvLSTM 1.5M 82.1% 57.8%
FPN-ConvLSTM 1.3M 81.6% 57.1%

Requirements

PASTIS Dataset download

The Dataset is freely available for download here.

Python requirements

To install requirements:

pip install -r requirements.txt

(torch_scatter is required for the panoptic experiments. Installing this library requires a little more effort, see the official repo)

Inference with pre-trained models

Panoptic segmentation

Pre-trained weights of U-TAE+Paps are available here

To perform inference of the pre-trained model on the test set of PASTIS run:

python test_panoptic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER

Semantic segmentation

Pre-trained weights of U-TAE are available here

To perform inference of the pre-trained model on the test set of PASTIS run:

python test_semantic.py --dataset_folder PATH_TO_DATASET --weight_folder PATH_TO_WEIGHT_FOLDER

Training models from scratch

Panoptic segmentation

To reproduce the main result for panoptic segmentation (with U-TAE+PaPs) run the following :

python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR

Options are also provided in train_panoptic.py to reproduce the other results of Table 2:

python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_NoCNN --no_mask_conv
python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --backbone uconvlstm
python train_panoptic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_shape24 --shape_size 24

Note: By default this script runs the 5 folds of the cross validation, which can be quite long (~12 hours per fold on a Tesla V100). Use the fold argument to execute one of the 5 folds only (e.g. for the 3rd fold : python train_panoptic.py --fold 3 --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR).

Semantic segmentation

To reproduce results for semantic segmentation (with U-TAE) run the following :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR

And in order to obtain the results of the competing methods presented in Table 1 :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UNET3d --model unet3d
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_UConvLSTM --model uconvlstm
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_FPN --model fpn
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BUConvLSTM --model buconvlstm
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_COnvGRU --model convgru
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_ConvLSTM --model convlstm

Finally, to reproduce the ablation study presented in Table 1 :

python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_MeanAttention --agg_mode att_mean
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SkipMeanConv --agg_mode mean
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_BatchNorm --encoder_norm batch
python train_semantic.py --dataset_folder PATH_TO_DATASET --res_dir OUT_DIR_SingleDate --mono_date "08-01-2019"

Reference

Please include a citation to the following paper if you use the U-TAE, PaPs or the PASTIS benchmark.

@article{garnot2021panoptic,
  title={Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks},
  author={Sainte Fare Garnot, Vivien  and Landrieu, Loic },
  journal={ICCV},
  year={2021}
}

Credits

  • This work was partly supported by ASP, the French Payment Agency.

  • Code for the presented methods and dataset is original code by Vivien Sainte Fare Garnot, competing methods and some utility functions were adapted from existing repositories which are credited in the corresponding files.

Issues
  • Optimization problem: missing m_hat parameters

    Optimization problem: missing m_hat parameters

    Hi, how is it going? @VSainteuf @loicland

    In equation (6), which is apply to extract the centerness map, I would like to ask what is the estimated parameters $\hat{I}{p}$ and $\hat{J}{p}$.

    I'm asking since I have implemented this optimization problem and I didn't got those parameters:

    def loss(params):
        i_measured, j_measured, sigma_h, sigma_w = params
        term_i = np.power((i_measured), 2) / np.power((2*sigma_h), 2)
        term_j = np.power((j_measured), 2) / np.power((2*sigma_w), 2)
    
        return np.exp(-(term_i + term_j))
    

    Thank you so much!

    opened by GustavoMourao 15
  • multi GPU training problem

    multi GPU training problem

    When I use multiple GPUs for training, the inconsistent size of tensor types in instance_masks leads to data parallelism errors. How can I solve this error?

    opened by MiSsU-HH 7
  • Runtime error during testing panoptic segmentation (tensors on different devices)

    Runtime error during testing panoptic segmentation (tensors on different devices)

    During testing panoptic segmentation, with the following command:

    python3 test_panoptic.py --dataset_folder "../PASTIS" --weight_folder "../UTAE_PAPs"
    

    I ran into the following error:

      File "test_panoptic.py", line 142, in <module>
        main(config)
      File "test_panoptic.py", line 125, in main
        device=device,
      File "/utae-paps/train_panoptic.py", line 243, in iterate
        pano_meter.add(predictions, y)
      File "/utae-paps/src/panoptic/metrics.py", line 126, in add
        self.cumulative_ious[i] += torch.stack(ious).sum()
    RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
    

    It seems that tensor self.cumulative_ious is on cpu while the other tensor is on cuda.

    The following change in file /utae-paps/src/panoptic/metrics.py:

       self.cumulative_ious[i] += torch.stack(ious).sum().to(device='cpu')
    

    manages to fix this issue. Kindly confirm the validity of this fix.

    Kindly also note that I also tried transferring self.cumulative_ious to cuda, but it runs into errors later during execution.

    opened by watch24hrs-iiitd 7
  • A couple of questions about the data set

    A couple of questions about the data set

    Hi, I want to replace the original dataset with my own, but I don't know how some NPY files are generated (e.g. INSTANCE_Id file, it seems to add some boundaries and determine the background), could you explain

    opened by Lzqlzq123123 3
  • How to perform predictions for a single image. (parcels predicted)

    How to perform predictions for a single image. (parcels predicted)

    Hi,

    There is any function/script in the code to perform predictions from the model using a single input image? I want to reproduce the qualitative results of the panoptic process (panoptic mask parcels predict) show in figure 12 in the paper. thanks

    opened by alexanderDuenas 3
  • Problem with filtering out multiple void instances

    Problem with filtering out multiple void instances

    I run some tests with your implementation for the panoptic metric and got a curious result.

    I made a simple test case with two instance who overlap the void class -> they should get filtered out.

    if I have only one of those instances it works properly and I don’t get FP. If I now have both instances they don’t get filtered out and I get two FP

    It might have to do with https://github.com/VSainteuf/utae-paps/blob/9e54d3c330b6d207f951796ea1656be8ebfad562/src/panoptic/metrics.py#L55

    since you here do the unique over basicly a binary mask if I understand correctly. I supect you wanted to do here instead of void_mask -> instance_ture[batch_idx]

    opened by marckatzenmaier 2
  • Question about visualization

    Question about visualization

    Hi, Thank you very much for the visualization code you shared, I am getting the following error when I run it

    TypeError                                 Traceback (most recent call last)
    Input In [50], in <cell line: 8>()
          5 alpha=.5
          8 for b in range(batch_size):
          9     # Plot S2 background
    ---> 10     im = get_rgb(x,b=b, t_show=t)
         11     axes[b,0].imshow(im)
         12     axes[b,2].imshow(im)
    
    Input In [48], in get_rgb(x, b, t_show)
         17 def get_rgb(x,b=0,t_show=6):
         18     """Gets an observation from a time series and normalises it for visualisation."""
    ---> 19     im = x[b,t_show,[2,1,0]].cpu().numpy()
         20     mx = im.max(axis=(1,2))
         21     mi = im.min(axis=(1,2))   
    
    TypeError: 'int' object is not subscriptable
    

    How can I fix this error please?

    opened by Auroralyxa 2
  • Dataset format: Zones

    Dataset format: Zones

    @VSainteuf @watch24hrs-iiitd, Hello

    it is possible to do a new training with a new dataset. I am studying the implementation and I find that the file train_panoptic.py needs the zone parameters for this. What are zones and what type of data are they.

    Than you

    if mode != "train": with torch.no_grad(): predictions = model( x, batch_positions=dates, pseudo_nms=compute_metrics, heatmap_only=heatmap_only, ) else: zones = y[:, :, :, 2] if config.supmax else None optimizer.zero_grad() predictions = model( x,

    Originally posted by @jhonjam in https://github.com/VSainteuf/utae-paps/issues/3#issuecomment-995907734

    opened by VSainteuf 2
  • Config mismatch during testing

    Config mismatch during testing

    For testing panoptic segmentation, (using downloaded UTAE-PAPs weights and PASTIS dataset from Zenodo).

    I ran the following command (as mentioned in readme):

    python3 test_panoptic.py --dataset_folder "./PASTIS" --weight_folder "./UTAE_PAPs"
    

    Despite the dataset_folder explicitly given as ./PASTIS, the dataset_folder is assigned the value /home/DATA/PASTIS.

    On further investigation, in test_panoptic.py,

    if __name__ == "__main__":
        test_config = parser.parse_args()
    
        with open(os.path.join(test_config.weight_folder, "conf.json")) as file:
            model_config = json.loads(file.read())
    
        config = {**vars(test_config), **model_config}
        config = argparse.Namespace(**config)
        config.fold = test_config.fold
       
        pprint.pprint(config)
        main(config)
    

    Here the variable config.dataset_folder is assigned the value of model_config.dataset_folder (which is /home/DATA/PASTIS) rather than test_config.dataset_folder (which is ./PASTIS), thus resulting in configuration mismatch.

    opened by watch24hrs-iiitd 2
  • Panoptic Metrics Bugfix: correctly ignoring void segments

    Panoptic Metrics Bugfix: correctly ignoring void segments

    Fix for issue #11

    Problem

    Void instances where not correctly matched to predicted instances. Predicted instances matched with void instances were counted as False Positives instead of being ignored. This results in an artificial decrease of the Recognition Quality.

    Fix

    Iterate over the correct tensor in l.55 of src/panoptic/metrics.py.

    Impact

    • The metrics reported in the original paper are impacted by this error. I re-runed the evaluation of the different methods of the paper. Across methods, the bug fix entails a ~2 point increase of Panoptic Quality. The results will be updated in a new Arxiv version.
    • The panoptic metrics are not involved in the training loss so this bug does not affect training procedure. Models that were trained with the previous implementation just need to be re-evaluated with the corrected metrics.
    opened by VSainteuf 0
  • [missing input argment]

    [missing input argment]

    https://github.com/VSainteuf/utae-paps/blob/2b9fae182f7271cabe59e5837057c7c1b0b40f39/test_semantic.py#L23

    Please, add:

    parser.add_argument(
        "--model",
        default="utae",
        type=str,
        help="Type of architecture to use. Can be one of: (utae/unet3d/fpn/convlstm/convgru/uconvlstm/buconvlstm)",
    )
    

    Since we'll get this error:

    AttributeError: 'Namespace' object has no attribute 'model'
    
    opened by GustavoMourao 4
An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

Abhinav Atrishi 10 Jun 1, 2022
Time-series-deep-learning - Developing Deep learning LSTM, BiLSTM models, and NeuralProphet for multi-step time-series forecasting of stock price.

Stock Price Prediction Using Deep Learning Univariate Time Series Predicting stock price using historical data of a company using Neural networks for

Abdultawwab Safarji 3 Jun 11, 2022
Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020). This codebase contains the essential ingredients of PCV, including various spatial discretizati

Haochen 23 Mar 23, 2022
[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021) PyTorch implementation for EOPSN. We propose open-set panoptic segmentation t

Jaedong Hwang 45 May 31, 2022
[CVPR 2021] Forecasting the panoptic segmentation of future video frames

Panoptic Segmentation Forecasting Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021 [Link to paper] We propose

Niantic Labs 44 May 19, 2022
PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

null 40 Jun 16, 2022
A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

Biomedical Computer Vision @ Uniandes 45 Jan 25, 2022
Implementation for Panoptic-PolarNet (CVPR 2021)

Panoptic-PolarNet This is the official implementation of Panoptic-PolarNet. [ArXiv paper] Introduction Panoptic-PolarNet is a fast and robust LiDAR po

Zixiang Zhou 108 Jun 23, 2022
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 359 May 28, 2022
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

Mamy Ratsimbazafy 360 Jun 28, 2022
[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Interactive Scene Reconstruction Project Page | Paper This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D

null 72 Jun 15, 2022
This project is a loose implementation of paper "Algorithmic Financial Trading with Deep Convolutional Neural Networks: Time Series to Image Conversion Approach"

Stock Market Buy/Sell/Hold prediction Using convolutional Neural Network This repo is an attempt to implement the research paper titled "Algorithmic F

Asutosh Nayak 131 May 12, 2022
PyTorch implementation of Soft-DTW: a Differentiable Loss Function for Time-Series in CUDA

Soft DTW Loss Function for PyTorch in CUDA This is a Pytorch Implementation of Soft-DTW: a Differentiable Loss Function for Time-Series which is batch

Keon Lee 38 Jun 14, 2022
Implementation of ETSformer, state of the art time-series Transformer, in Pytorch

ETSformer - Pytorch Implementation of ETSformer, state of the art time-series Transformer, in Pytorch Install $ pip install etsformer-pytorch Usage im

Phil Wang 103 Jun 18, 2022
1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

1st place solution to the Satellite Image Change Detection Challenge hosted by SenseTime

Lihe Yang 172 Jun 15, 2022
Multiband spectro-radiometric satellite image analysis with K-means cluster algorithm

Multi-band Spectro Radiomertric Image Analysis with K-means Cluster Algorithm Overview Multi-band Spectro Radiomertric images are images comprising of

Chibueze Henry 6 Mar 16, 2022
SatelliteNeRF - PyTorch-based Neural Radiance Fields adapted to satellite domain

SatelliteNeRF PyTorch-based Neural Radiance Fields adapted to satellite domain.

Kai Zhang 27 Jun 22, 2022
U-Net Implementation: Convolutional Networks for Biomedical Image Segmentation" using the Carvana Image Masking Dataset in PyTorch

U-Net Implementation By Christopher Ley This is my interpretation and implementation of the famous paper "U-Net: Convolutional Networks for Biomedical

Christopher Ley 1 Jan 6, 2022
tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

timeseriesAI 2.2k Jun 28, 2022