PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation (TPAMI).

Last update: Dec 31, 2022

Overview

PFENet

This is the implementation of our paper PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation that has been accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).

Get Started

Environment

torch==1.4.0 (torch version >= 1.0.1.post2 should be okay to run this repo)
numpy==1.18.4
tensorboardX==1.8
cv2==4.2.0

Datasets and Data Preparation

Please download the following datasets:

PASCAL-5i is based on the PASCAL VOC 2012 and SBD where the val images should be excluded from the list of training samples.
COCO 2014.

This code reads data from .txt files where each line contains the paths for image and the correcponding label respectively. Image and label paths are seperated by a space. Example is as follows:

image_path_1 label_path_1
image_path_2 label_path_2
image_path_3 label_path_3
...
image_path_n label_path_n

Then update the train/val/test list paths in the config files.

[Update] We have uploaded the lists we use in our paper.

The train/val lists for COCO contain 82081 and 40137 images respectively. They are the default train/val splits of COCO.
The train/val lists for PASCAL5i contain 5953 and 1449 images respectively. The train list should be voc_sbd_merge_noduplicate.txt and the val list is the original val list of pascal voc (val.txt).

To get voc_sbd_merge_noduplicate.txt:

We first merge the original VOC (voc_original_train.txt) and SBD (sbd_data.txt) training data.
[Important] sbd_data.txt does not overlap with the PASCALVOC 2012 validation data.
The merged list (voc_sbd_merge.txt) is then processed by the script (duplicate_removal.py) to remove the duplicate images and labels.

Run Demo / Test with Pretrained Models

Please download the pretrained models.
We provide 8 pre-trained models: 4 ResNet-50 based models for PASCAL-5i and 4 VGG-16 based models for COCO.
Update the config file by speficifying the target split and path (weights) for loading the checkpoint.
Execute mkdir initmodel at the root directory.
Download the ImageNet pretrained backbones and put them into the initmodel directory.
Then execute the command:

sh test.sh {*dataset*} {*model_config*}

Example: Test PFENet with ResNet50 on the split 0 of PASCAL-5i:

sh test.sh pascal split0_resnet50

Train

Execute this command at the root directory:

sh train.sh {*dataset*} {*model_config*}

Related Repositories

This project is built upon a very early version of SemSeg: https://github.com/hszhao/semseg.

Other projects in few-shot segmentation:

OSLSM: https://github.com/lzzcd001/OSLSM
CANet: https://github.com/icoz69/CaNet
PANet: https://github.com/kaixin96/PANet
FSS-1000: https://github.com/HKUSTCV/FSS-1000
AMP: https://github.com/MSiam/AdaptiveMaskedProxies
On the Texture Bias for FS Seg: https://github.com/rezazad68/fewshot-segmentation
SG-One: https://github.com/xiaomengyc/SG-One
FS Seg Propogation with Guided Networks: https://github.com/shelhamer/revolver

Many thanks to their greak work!

Citation

If you find this project useful, please consider citing:

@article{tian2020pfenet,
  title={Prior Guided Feature Enrichment Network for Few-Shot Segmentation},
  author={Tian, Zhuotao and Zhao, Hengshuang and Shu, Michelle and Yang, Zhicheng and Li, Ruiyu and Jia, Jiaya},
  journal={TPAMI},
  year={2020}
}

Comments

remove small objects

Thanks for sharing your work! I noticed that you remove all objects whose area are smaller than 2x32x32, following the practice of OSLSM. However, in some recent works, such as PANet(ICCV'19) and CANet(CVPR'19), they include all objects in their source code whatever their size is. So I am wondering whether it is somewhat unfair to compare with these methods directly? Or are there any details that I have neglected?

opened by LiheYoung 9
question about train.py

Why subtract one here (subcls-1)? I understand subcls here is the index of class_chosen in sublist ,so it should start from 0,but subtracting one cause it become a negative number subcls = subcls[0].cpu().numpy()[0] class_intersection_meter[(subcls-1)%split_gap] += intersection[1] class_union_meter[(subcls-1)%split_gap] += union[1]

opened by ily666666 8
May be a wrong code ?

In train.py the validate function. but i find the len(subcls) == 5, len(subcls[0]) == 20, I set 5-shot, and batch_size_val = 20, split is 1. I can't get the purpose of ' 'subcls[0].cpu().numpy()[0]', I think the operator subcls[0].cpu().numpy()[0] is wrong,

opened by zhijiejia 7
n way setting

hello, thanks for your excellent work. I didn't find any parameter setting number of way, is the batch size of support image actually define the n_way? thanks

opened by tymatfd 7
Problem about dataset setting and baseline

Thanks for sharing your work! I would like to know why the PASCAL training set only contains about 5900 images. I mean in the setting of CANet or OSLSM, the number of training images is more than that, right?

opened by roywithfiringblade 6
results obtained fluctuate greatly even set all random seeds

Hi, thanks for sharing your work! There's a question that troubles me. I set the same random seed every time, but the results obtained fluctuate greatly (about 1%). Is this due to the "Dropout " operation, or something I neglected?

Regards, Lang
good first issue

opened by chunbolang 5
Question regarding evaluation with ignore_label

Hello, I have a question regarding evaluation

In line 52-64 in PFENet/util/util.py (function intersectionAndUnion), the function computes intersection & union between gt-mask and predicted mask. I found that the code uses ignore_label=255, to refine the prediction right before computing the intersection & union. Why did you adopt this kind of refinement process (using mask boundary, e.g., ignore_label)?

This refinement would make sense if the code tries to ignore the zero-padded regions (which is set to ignore_label, e.g., 255) in both predicted and gt masks. However, because the object boundary is also set as ignore_label=255, the prediction is further refined.

In the paper, the mIoU of PFENet in 1-shot setting (with resnet50 backbone) is 60.8% on PASCAL-5i dataset but when I reproduced the model without using object boundary (ignore label), the mIoU drops to 56.2%. May I ask why did you adopt such prediction refinement using gt boundary?

I brought example predictions: Top image visualizes naive prediction by the model while the bottom image visualizes refined prediction using gt-boundary.

opened by juhongm999 5
how to generate SegmentationClassAug？

I have downloaded the PASCAL VOC 2012 dataset and SBD dataset from their official website respectively.But I donnot know how to use them in the code .Could you show me your dataset folder structure or the method to generate SegmentationClassAug? Thanks , bubble from hfut!

opened by ily666666 5
Problem about reproduced accuracy

Hi, thanks for sharing the code! I have trained the model without any modification, but the results are always about 1% worse than the reported accuracy. Here are some reproduced results with reported results on Pascalvoc dataset: Fold-0 60.8 (61.7), Fold-1 68.5 (69.5), Fold-2 53.9 (55.4) So I wonder if I miss some tricks to reach the reported result? do I need to keep fine-tuning the model?
good first issue

opened by Reagan1311 5
Can not download pretrained model

The pretrained model cannot be downloaded from the given link: https://mycuhk-my.sharepoint.com/:u:/g/personal/1155122171_link_cuhk_edu_hk/EW20i_eiTINDgJDqUqikNR4Bo-7kVFkLBkxGZ2_uorOJcw?e=4%3aSIRlwD&at=9

opened by jingliang95 5
Experiment detail
Hi, this project is very helpful. But I would like to know some details:

the backbone you provided named resnet50_v2,pth and resnet101_v2.pth. Did you train this backbone from init for few-shot segmentation task?

I get same result with pre-trained model. But the model training by myself is lower about 2% mIoU. Is the config same with the paper setting?

Thanks.
opened by 2448845600 4
Problem in code

Please check this error Traceback (most recent call last): File "test.py", line 263, in main() File "test.py", line 86, in main main_worker(args.train_gpu, args.ngpus_per_node, args) File "test.py", line 144, in main_worker loss_val, mIoU_val, mAcc_val, allAcc_val, class_miou = validate(val_loader, model, criterion) File "test.py", line 237, in validate allAcc = sum(intersection_meter.sum) / (sum(target_meter.sum) + 1e-10) TypeError: 'int' object is not iterable

opened by titoriya007 0
Training Details

Hello，

It's a good job for FSS，and I am very interested in your work and tried to train myself.

But the test result of the trained model is a little lower than what you gave, so I would like to ask you about the details of the model training or what is the cause?

Thank you!

opened by liushengnan1998 1
分割结果可视化

Hi,Thank you for your work . I tried to follow this method https://github.com/hszhao/semseg/blob/master/tool/test.py#L216 to get the segmentation result , but due to the random horizontal rotation and mirroring processing set in the experiment, the obtained results cannot be superimposed on the original image.And I can't find any other solution because I'm too clumsy。 Could you please share the visualization code you use, it would be greatly appreciated。

opened by XBPFS 1
visualization

hello,Thank you for your work! I used this link https://github.com/hszhao/semseg/blob/master/tool/test.py#L216 to visualize images,But I never got it right,Can you provide your visual code?thanks.

opened by zhthing 1

Owner

DV Lab

Deep Vision Lab

GitHub

The implementation of PEMP in paper "Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes"

Prior-Enhanced network with Meta-Prototypes (PEMP) This is the PyTorch implementation of PEMP. Overview of PEMP Meta-Prototypes & Adaptive Prototypes

8 Oct 14, 2021

Few-NERD: Not Only a Few-shot NER Dataset

Few-NERD: Not Only a Few-shot NER Dataset This is the source code of the ACL-IJCNLP 2021 paper: Few-NERD: A Few-shot Named Entity Recognition Dataset.

319 Dec 30, 2022

Code for T-Few from "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning"

T-Few This repository contains the official code for the paper: "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learni

220 Dec 31, 2022

Sample Prior Guided Robust Model Learning to Suppress Noisy Labels

PGDF This repo is the official implementation of our paper "Sample Prior Guided Robust Model Learning to Suppress Noisy Labels ". Citation If you use

22 Dec 23, 2022

PyTorch implemention of ICCV'21 paper SGPA: Structure-Guided Prior Adaptation for Category-Level 6D Object Pose Estimation

SGPA: Structure-Guided Prior Adaptation for Category-Level 6D Object Pose Estimation This is the PyTorch implemention of ICCV'21 paper SGPA: Structure

24 Dec 5, 2022

Original code for "Zero-Shot Domain Adaptation with a Physics Prior"

Zero-Shot Domain Adaptation with a Physics Prior [arXiv] [sup. material] - ICCV 2021 Oral paper, by Attila Lengyel, Sourav Garg, Michael Milford and J

40 Dec 21, 2022

[NeurIPS 2021] A weak-shot object detection approach by transferring semantic similarity and mask prior.

49 Jul 27, 2022

Models Supported: AlbUNet [18, 34, 50, 101, 152] (1D and 2D versions for Single and Multiclass Segmentation, Feature Extraction with supports for Deep Supervision and Guided Attention)

AlbUNet-1D-2D-Tensorflow-Keras This repository contains 1D and 2D Signal Segmentation Model Builder for AlbUNet and several of its variants developed

1 Nov 15, 2021

(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Res2Net The official pytorch implemention of the paper "Res2Net: A New Multi-scale Backbone Architecture" Our paper is accepted by IEEE Transactions o

928 Dec 29, 2022

PFENet: Prior Guided Feature Enrichment Network for Few-shot Segmentation (TPAMI).

Related tags

Overview

PFENet

Get Started

Environment

Datasets and Data Preparation

[Update] We have uploaded the lists we use in our paper.

To get voc_sbd_merge_noduplicate.txt:

Run Demo / Test with Pretrained Models

Train

Related Repositories

Citation

Comments

Owner

DV Lab

The implementation of PEMP in paper "Prior-Enhanced Few-Shot Segmentation with Meta-Prototypes"

Few-NERD: Not Only a Few-shot NER Dataset

Code for T-Few from "Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning"

Sample Prior Guided Robust Model Learning to Suppress Noisy Labels

PyTorch implemention of ICCV'21 paper SGPA: Structure-Guided Prior Adaptation for Category-Level 6D Object Pose Estimation

Original code for "Zero-Shot Domain Adaptation with a Physics Prior"

[NeurIPS 2021] A weak-shot object detection approach by transferring semantic similarity and mask prior.

Models Supported: AlbUNet [18, 34, 50, 101, 152] (1D and 2D versions for Single and Multiclass Segmentation, Feature Extraction with supports for Deep Supervision and Guided Attention)

Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Official PyTorch Implementation of Hypercorrelation Squeeze for Few-Shot Segmentation, arXiv 2021

[CVPR 2021] Few-shot 3D Point Cloud Semantic Segmentation

Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (CVPR 2021)

Official code for "Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer. ICCV2021".

The code is for the paper "A Self-Distillation Embedded Supervised Affinity Attention Model for Few-Shot Segmentation"

Pytorch Implementation for CVPR2018 Paper: Learning to Compare: Relation Network for Few-Shot Learning

Task-related Saliency Network For Few-shot learning

Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging, ICCV2021 [PyTorch Code]

An official implementation of "SFNet: Learning Object-aware Semantic Correspondence" (CVPR 2019, TPAMI 2020) in PyTorch.

(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"