PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation (TPAMI).

Related tags

Deep Learning segmentation few-shot pami-2020
Overview

PFENet

This is the implementation of our paper PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation that has been accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

Get Started

Environment

  • torch==1.4.0 (torch version >= 1.0.1.post2 should be okay to run this repo)
  • numpy==1.18.4
  • tensorboardX==1.8
  • cv2==4.2.0

Datasets and Data Preparation

Please download the following datasets:

  • PASCAL-5i is based on the PASCAL VOC 2012 and SBD where the val images should be excluded from the list of training samples.

  • COCO 2014.

This code reads data from .txt files where each line contains the paths for image and the correcponding label respectively. Image and label paths are seperated by a space. Example is as follows:

image_path_1 label_path_1
image_path_2 label_path_2
image_path_3 label_path_3
...
image_path_n label_path_n

Then update the train/val/test list paths in the config files.

[Update] We have uploaded the lists we use in our paper.

  • The train/val lists for COCO contain 82081 and 40137 images respectively. They are the default train/val splits of COCO.
  • The train/val lists for PASCAL5i contain 5953 and 1449 images respectively. The train list should be voc_sbd_merge_noduplicate.txt and the val list is the original val list of pascal voc (val.txt).
To get voc_sbd_merge_noduplicate.txt:
  • We first merge the original VOC (voc_original_train.txt) and SBD (sbd_data.txt) training data.
  • [Important] sbd_data.txt does not overlap with the PASCALVOC 2012 validation data.
  • The merged list (voc_sbd_merge.txt) is then processed by the script (duplicate_removal.py) to remove the duplicate images and labels.

Run Demo / Test with Pretrained Models

  • Please download the pretrained models.

  • We provide 8 pre-trained models: 4 ResNet-50 based models for PASCAL-5i and 4 VGG-16 based models for COCO.

  • Update the config file by speficifying the target split and path (weights) for loading the checkpoint.

  • Execute mkdir initmodel at the root directory.

  • Download the ImageNet pretrained backbones and put them into the initmodel directory.

  • Then execute the command:

    sh test.sh {*dataset*} {*model_config*}

Example: Test PFENet with ResNet50 on the split 0 of PASCAL-5i:

sh test.sh pascal split0_resnet50

Train

Execute this command at the root directory:

sh train.sh {*dataset*} {*model_config*}

Related Repositories

This project is built upon a very early version of SemSeg: https://github.com/hszhao/semseg.

Other projects in few-shot segmentation:

Many thanks to their greak work!

Citation

If you find this project useful, please consider citing:

@article{tian2020pfenet,
  title={Prior Guided Feature Enrichment Network for Few-Shot Segmentation},
  author={Tian, Zhuotao and Zhao, Hengshuang and Shu, Michelle and Yang, Zhicheng and Li, Ruiyu and Jia, Jiaya},
  journal={TPAMI},
  year={2020}
}
Comments
  • remove small objects

    remove small objects

    Thanks for sharing your work! I noticed that you remove all objects whose area are smaller than 2x32x32, following the practice of OSLSM. However, in some recent works, such as PANet(ICCV'19) and CANet(CVPR'19), they include all objects in their source code whatever their size is. So I am wondering whether it is somewhat unfair to compare with these methods directly? Or are there any details that I have neglected?

    opened by LiheYoung 9
  • question about train.py

    question about train.py

    Why subtract one here (subcls-1)? I understand subcls here is the index of class_chosen in sublist ,so it should start from 0,but subtracting one cause it become a negative number subcls = subcls[0].cpu().numpy()[0] class_intersection_meter[(subcls-1)%split_gap] += intersection[1] class_union_meter[(subcls-1)%split_gap] += union[1]

    opened by ily666666 8
  • May be a wrong code ?

    May be a wrong code ?

    In train.py the validate function. image but i find the len(subcls) == 5, len(subcls[0]) == 20, I set 5-shot, and batch_size_val = 20, split is 1. I can't get the purpose of ' 'subcls[0].cpu().numpy()[0]', I think the operator subcls[0].cpu().numpy()[0] is wrong,

    opened by zhijiejia 7
  • n way setting

    n way setting

    hello, thanks for your excellent work. I didn't find any parameter setting number of way, is the batch size of support image actually define the n_way? thanks

    opened by tymatfd 7
  • Problem about dataset setting and baseline

    Problem about dataset setting and baseline

    Thanks for sharing your work! I would like to know why the PASCAL training set only contains about 5900 images. I mean in the setting of CANet or OSLSM, the number of training images is more than that, right?

    opened by roywithfiringblade 6
  • results obtained fluctuate greatly even set all random seeds

    results obtained fluctuate greatly even set all random seeds

    Hi, thanks for sharing your work! There's a question that troubles me. I set the same random seed every time, but the results obtained fluctuate greatly (about 1%). Is this due to the "Dropout " operation, or something I neglected?

    Regards, Lang

    good first issue 
    opened by chunbolang 5
  • Question regarding evaluation with ignore_label

    Question regarding evaluation with ignore_label

    Hello, I have a question regarding evaluation

    In line 52-64 in PFENet/util/util.py (function intersectionAndUnion), the function computes intersection & union between gt-mask and predicted mask. I found that the code uses ignore_label=255, to refine the prediction right before computing the intersection & union. Why did you adopt this kind of refinement process (using mask boundary, e.g., ignore_label)?

    This refinement would make sense if the code tries to ignore the zero-padded regions (which is set to ignore_label, e.g., 255) in both predicted and gt masks. However, because the object boundary is also set as ignore_label=255, the prediction is further refined.

    In the paper, the mIoU of PFENet in 1-shot setting (with resnet50 backbone) is 60.8% on PASCAL-5i dataset but when I reproduced the model without using object boundary (ignore label), the mIoU drops to 56.2%. May I ask why did you adopt such prediction refinement using gt boundary?

    a

    I brought example predictions: Top image visualizes naive prediction by the model while the bottom image visualizes refined prediction using gt-boundary.

    opened by juhongm999 5
  • how to generate SegmentationClassAug？

    how to generate SegmentationClassAug？

    I have downloaded the PASCAL VOC 2012 dataset and SBD dataset from their official website respectively.But I donnot know how to use them in the code .Could you show me your dataset folder structure or the method to generate SegmentationClassAug? Thanks , bubble from hfut!

    opened by ily666666 5
  • Problem about reproduced accuracy

    Problem about reproduced accuracy

    Hi, thanks for sharing the code! I have trained the model without any modification, but the results are always about 1% worse than the reported accuracy. Here are some reproduced results with reported results on Pascalvoc dataset: Fold-0 60.8 (61.7), Fold-1 68.5 (69.5), Fold-2 53.9 (55.4) So I wonder if I miss some tricks to reach the reported result? do I need to keep fine-tuning the model?

    good first issue 
    opened by Reagan1311 5
  • Can not download pretrained model

    Can not download pretrained model

    The pretrained model cannot be downloaded from the given link: https://mycuhk-my.sharepoint.com/:u:/g/personal/1155122171_link_cuhk_edu_hk/EW20i_eiTINDgJDqUqikNR4Bo-7kVFkLBkxGZ2_uorOJcw?e=4%3aSIRlwD&at=9

    opened by jingliang95 5
  • Experiment detail

    Experiment detail

    Hi, this project is very helpful. But I would like to know some details:

    1. the backbone you provided named resnet50_v2,pth and resnet101_v2.pth. Did you train this backbone from init for few-shot segmentation task?
    2. I get same result with pre-trained model. But the model training by myself is lower about 2% mIoU. Is the config same with the paper setting?

    Thanks.

    opened by 2448845600 4
  • Problem in code

    Problem in code

    Please check this error Traceback (most recent call last): File "test.py", line 263, in main() File "test.py", line 86, in main main_worker(args.train_gpu, args.ngpus_per_node, args) File "test.py", line 144, in main_worker loss_val, mIoU_val, mAcc_val, allAcc_val, class_miou = validate(val_loader, model, criterion) File "test.py", line 237, in validate allAcc = sum(intersection_meter.sum) / (sum(target_meter.sum) + 1e-10) TypeError: 'int' object is not iterable

    opened by titoriya007 0
  • Training Details

    Training Details

    Hello，

    It's a good job for FSS，and I am very interested in your work and tried to train myself.

    But the test result of the trained model is a little lower than what you gave, so I would like to ask you about the details of the model training or what is the cause?

    Thank you!

    opened by liushengnan1998 1
  • 分割结果可视化

    分割结果可视化

    Hi,Thank you for your work . I tried to follow this method https://github.com/hszhao/semseg/blob/master/tool/test.py#L216 to get the segmentation result , but due to the random horizontal rotation and mirroring processing set in the experiment, the obtained results cannot be superimposed on the original image.And I can't find any other solution because I'm too clumsy。 Could you please share the visualization code you use, it would be greatly appreciated。

    opened by XBPFS 1
  • visualization

    visualization

    hello,Thank you for your work! I used this link https://github.com/hszhao/semseg/blob/master/tool/test.py#L216 to visualize images,But I never got it right,Can you provide your visual code?thanks.

    opened by zhthing 1
Owner
DV Lab
Deep Vision Lab
DV Lab
GitHub
The implementation of PEMP in paper "Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes"

Prior-Enhanced network with Meta-Prototypes (PEMP) This is the PyTorch implementation of PEMP. Overview of PEMP Meta-Prototypes & Adaptive Prototypes

Jianwei ZHANG 8 Oct 14, 2021
Few-NERD: Not Only a Few-shot NER Dataset

Few-NERD: Not Only a Few-shot NER Dataset This is the source code of the ACL-IJCNLP 2021 paper: Few-NERD: A Few-shot Named Entity Recognition Dataset.

THUNLP 319 Dec 30, 2022
Code for T-Few from "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning"

T-Few This repository contains the official code for the paper: "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learni

null 220 Dec 31, 2022
Sample Prior Guided Robust Model Learning to Suppress Noisy Labels

PGDF This repo is the official implementation of our paper "Sample Prior Guided Robust Model Learning to Suppress Noisy Labels ". Citation If you use

CVSM Group - email: czhu@bupt.edu.cn 22 Dec 23, 2022
PyTorch implemention of ICCV'21 paper SGPA: Structure-Guided Prior Adaptation for Category-Level 6D Object Pose Estimation

SGPA: Structure-Guided Prior Adaptation for Category-Level 6D Object Pose Estimation This is the PyTorch implemention of ICCV'21 paper SGPA: Structure

Chen Kai 24 Dec 5, 2022
Original code for "Zero-Shot Domain Adaptation with a Physics Prior"

Zero-Shot Domain Adaptation with a Physics Prior [arXiv] [sup. material] - ICCV 2021 Oral paper, by Attila Lengyel, Sourav Garg, Michael Milford and J

Attila Lengyel 40 Dec 21, 2022
[NeurIPS 2021] A weak-shot object detection approach by transferring semantic similarity and mask prior.

[NeurIPS 2021] A weak-shot object detection approach by transferring semantic similarity and mask prior.

BCMI 49 Jul 27, 2022
Models Supported: AlbUNet [18, 34, 50, 101, 152] (1D and 2D versions for Single and Multiclass Segmentation, Feature Extraction with supports for Deep Supervision and Guided Attention)

AlbUNet-1D-2D-Tensorflow-Keras This repository contains 1D and 2D Signal Segmentation Model Builder for AlbUNet and several of its variants developed

Sakib Mahmud 1 Nov 15, 2021
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Region Proportion Regularized Inference (RePRI) for Few-Shot Segmentation In this repo, we provide the code for our paper : "Few-Shot Segmentation Wit

Malik Boudiaf 138 Dec 12, 2022
Official PyTorch Implementation of Hypercorrelation Squeeze for Few-Shot Segmentation, arXiv 2021

Hypercorrelation Squeeze for Few-Shot Segmentation This is the implementation of the paper "Hypercorrelation Squeeze for Few-Shot Segmentation" by Juh

Juhong Min 165 Dec 28, 2022
[CVPR 2021] Few-shot 3D Point Cloud Semantic Segmentation

Few-shot 3D Point Cloud Semantic Segmentation Created by Na Zhao from National University of Singapore Introduction This repository contains the PyTor

null 117 Dec 27, 2022
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (CVPR 2021)

ASGNet The code is for the paper "Adaptive Prototype Learning and Allocation for Few-Shot Segmentation" (accepted to CVPR 2021) [arxiv] Overview data/

Gen Li 91 Dec 23, 2022
Official code for "Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021".

Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021. Introduction We proposed a novel model training paradi

Lucas 103 Dec 14, 2022
The code is for the paper "A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation"

SD-AANet The code is for the paper "A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation" [arxiv] Overview confi

cv516Buaa 9 Nov 7, 2022
Pytorch Implementation for CVPR2018 Paper: Learning to Compare: Relation Network for Few-Shot Learning

LearningToCompare Pytorch Implementation for Paper: Learning to Compare: Relation Network for Few-Shot Learning Howto download mini-imagenet and make

Jackie Loong 246 Dec 19, 2022
Task-related Saliency Network For Few-shot learning

Task-related Saliency Network For Few-shot learning This is an official implementation in Tensorflow of TRSN. Abstract An essential cue of human wisdo

null 1 Nov 18, 2021
Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging, ICCV2021 [PyTorch Code]

Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging, ICCV2021 [PyTorch Code]

Jian Zhang 20 Oct 24, 2022
An official implementation of "SFNet: Learning Object-aware Semantic Correspondence" (CVPR 2019, TPAMI 2020) in PyTorch.

PyTorch implementation of SFNet This is the implementation of the paper "SFNet: Learning Object-aware Semantic Correspondence". For more information,

CV Lab @ Yonsei University 87 Dec 30, 2022
(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Res2Net The official pytorch implemention of the paper "Res2Net: A New Multi-scale Backbone Architecture" Our paper is accepted by IEEE Transactions o

Res2Net Applications 928 Dec 29, 2022