(AAAI 2021) Progressive One-shot Human Parsing

Overview

End-to-end One-shot Human Parsing

This is the official repository for our two papers:


Introduction:

In the two papers, we propose a new task named One-shot Human Parsing (OSHP). OSHP requires parsing humans in a query image into an open set of reference classes defined by any single reference example (i.e., a support image) during testing, no matter whether they have been annotated during training (base classes) or not (novel classes). This new task mainly aims to accommodate human parsing into a wider range of applications that seek to parse flexible fashion/clothing classes that are not pre-defined in previous large-scale datasets.

Progressive One-shot Human Parsing (AAAI 2021) applies a progressive training scheme and is separated into three stages.

End-to-end One-shot Human Parsing (journal version) is a one-stage end-to-end training method, which has higher performance and FPS.


Main results:

You can find the well-trained models together with the performance in the following table.

EOPNet ATR-OS, Kway F1 ATR-OS, Kway Fold F2 LIP-OS, Kway F1 LIP-OS, Kway F2 CIHP-OS, Kway F1 CIHP-OS Kway F2
Novel mIoU 31.1 34.6 25.7 30.4 20.5 25.1
Human mIoU 61.9 63.3 43.0 45.7 49.1 45.5
Model Model Coming Soon Model Model Model Model

You can find the well-trained models together with the performance in the following table.

EOPNet ATR-OS, 1way F1 ATR-OS, 1way F2 LIP-OS, 1way F1 LIP-OS, 1way F2 CIHP-OS, 1way F1 CIHP-OS 1way F2
Novel mIoU 53.0 41.4 42.0 46.2 25.4 36.4
Human mIoU 68.2 69.5 57.0 58.0 53.8 55.4
Model Coming Soon

Getting started:

Data preparation:

First, please download ATR, LIP and CIHP dataset from source. Then, use the following commands to link the data into our project folder. Please also remember to download the atr flipped labels and cihp flipped labels.

# ATR dataset
$ ln -s YOUR_ATR_PATH/JPEGImages/* YOUR_PROJECT_ROOT/ATR_OS/trainval_images
$ ln -s YOUR_ATR_PATH/SegmentationClassAug/* YOUR_PROJECT_ROOT/ATR_OS/trainval_classes
$ ln -s YOUR_ATR_PATH/SegmentationClassAug_rev/* YOUR_PROJECT_ROOT/ATR_OS/Category_rev_ids


# LIP dataset
$ ln -s YOUR_LIP_PATH/TrainVal_images/TrainVal_images/train_images/* YOUR_PROJECT_ROOT/LIP_OS/trainval_images
$ ln -s YOUR_LIP_PATH/TrainVal_images/TrainVal_images/val_images/* YOUR_PROJECT_ROOT/LIP_OS/trainval_images
$ ln -s YOUR_LIP_PATH/TrainVal_parsing_annotations/TrainVal_parsing_annotations/train_segmentations/* YOUR_PROJECT_ROOT/LIP_OS/trainval_classes
$ ln -s YOUR_LIP_PATH/TrainVal_parsing_annotations/TrainVal_parsing_annotations/val_segmentations/* YOUR_PROJECT_ROOT/LIP_OS/trainval_classes
$ ln -s YOUR_LIP_PATH/Train_parsing_reversed_labels/TrainVal_parsing_annotations/* YOUR_PROJECT_ROOT/LIP_OS/Category_rev_ids
$ ln -s YOUR_LIP_PATH/val_segmentations_reversed/* YOUR_PROJECT_ROOT/LIP_OS/Category_rev_ids


# CIHP dataset
$ ln -s YOUR_CIHP_PATH/Training/Images/* YOUR_PROJECT_ROOT/CIHP_OS/trainval_images
$ ln -s YOUR_CIHP_PATH/Validation/Images/* YOUR_PROJECT_ROOT/CIHP_OS/trainval_images
$ ln -s YOUR_CIHP_PATH/Training/Category_ids/* YOUR_PROJECT_ROOT/CIHP_OS/trainval_classes
$ ln -s YOUR_CIHP_PATH/Validation/Category_ids/* YOUR_PROJECT_ROOT/CIHP_OS/trainval_classes
$ ln -s YOUR_CIHP_PATH/Category_rev_ids/* YOUR_PROJECT_ROOT/CIHP_OS/Category_rev_ids

Please also download our generated support .pkl files from source, which contains each class's image IDs. You can also generate support files on your own by controlling dtrain_dtest_split in oshp_loader.py, however, the training and validation list might be different from our paper.

Finally, your data folder should look like this:

${PROJECT ROOT}
|-- data
|   |--datasets
|       |-- ATR_OS
|       |   |-- list
|       |   |   |-- meta_train_id.txt
|       |   |   `-- meta_test_id.txt
|       |   |-- support
|       |   |   |-- meta_train_atr_supports.pkl
|       |   |   `-- meta_test_atr_supports.pkl
|       |   |-- trainval_images
|       |   |   |-- 997-1.jpg
|       |   |   |-- 997-2.jpg
|       |   |   `-- ...
|       |   |-- trainval_classes
|       |   |   |-- 997-1.png
|       |   |   |-- 997-2.png
|       |   |   `-- ... 
|       |   `-- Category_rev_ids
|       |       |-- 997-1.png
|       |       |-- 997-2.png
|       |       `-- ... 
|       |-- LIP_OS
|       |   |-- list
|       |   |   |-- meta_train_id.txt
|       |   |   |-- meta_test_id.txt
|       |   |-- support
|       |   |   |-- meta_train_lip_supports.pkl
|       |   |   `-- meta_test_lip_supports.pkl
|       |   |-- trainval_images
|       |   |   |-- ...
|       |   |-- trainval_classes
|       |   |   |-- ... 
|       |   `-- Category_rev_ids
|       |       |-- ... 
|       `-- CIHP_OS
|           |-- list
|           |   |-- meta_train_id.txt
|           |   |-- meta_test_id.txt
|           |-- support
|           |   |-- meta_train_cihp_supports.pkl
|           |   `-- meta_test_cihp_supports.pkl
|           |-- trainval_images
|           |   |-- ...
|           |-- trainval_classes
|           |   |-- ... 
|           `-- Category_rev_ids
|               |-- ... 

Finally, please download the DeepLab V3+ pretrained model (pretrained on COCO dataset) from source and put it into the data folder:

${PROJECT ROOT}
|-- data
|   |--pretrained_model
|       |--deeplab_v3plus_v3.pth

Installation:

Please make sure your current environment has Python >= 3.7.0 and pytorch >= 1.1.0. The pytorch can be downloaded from source.

Then, clone the repository and install the dependencies from the following commands:

git clone https://github.com/Charleshhy/One-shot-Human-Parsing.git
cd One-shot-Human-Parsing
pip install -r requirements.txt

Training:

To train EOPNet in End-to-end One-shot Human Parsing (journal version), run:

# OSHP kway on ATR-OS fold 1
bash scripts/atr_eop_kwf1.sh

Validation:

To evaluate EOPNet in End-to-end One-shot Human Parsing (journal version), run:

# OSHP kway on ATR-OS fold 1
bash scripts/evaluate_atr_eop_kwf1.sh

TODO:

  • Release training/validation code for POPNet
  • Release well-trained EOPNet 1-way models

Citation:

If you find our papers or this repository useful, please consider cite our papers:

@inproceedings{he2021progressive,
title={Progressive One-shot Human Parsing},
author={He, Haoyu and Zhang, Jing and Thuraisingham, Bhavani and Tao, Dacheng},
booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
year={2021}
}

@article{he2021end,
title={End-to-end One-shot Human Parsing},
author={He, Haoyu and Zhang, Jing and Zhuang, Bohan and Cai, Jianfei and Tao, Dacheng},
journal={arXiv preprint arXiv:2105.01241},
year={2021}
}

Acknowledgement:

This repository is mainly developed basing on Graphonomy and Grapy-ML.

Comments
  • some questions about metrics

    some questions about metrics

    I successfully reproduced your code on the atr dataset, but the effect did not reach the effect mentioned in your article, as shown in the figure below, epoch=50, batch=2, gpus=2, because batch=2 will report an error, I made a simple modification to the code, what else can I do to achieve the effect of your article. Also, which part of the code output should I look at, agm or npm, where mean miou is the human miou in the paper? Looking forward to your reply!

    opened by dandanAD 3
  • Data sets usage license

    Data sets usage license

    Hello Thanks for the awesome work

    In the README, there are link to datasets that can be downloaded via google drive, can you specify what are the licenses for using these datasets

    b.r

    opened by AlkaSaliss 2
  • Inference using pretrained models

    Inference using pretrained models

    Hi, first of all, thanks a lot for sharing your work. Really impressive! I have 2 questions please:

    1. I am not sure if I get the purpose/capabilities of this approach correctly but, is the model able to handle one single input image at inference time? e.g. during training it seems that both query and support images/masks are needed. For inference, what we want though, is to be able to submit a single (or batch) picture of a person and have the mask as output (image IN -> mask OUT). Is this supported?
    2. If yes, do you have a code snippet providing preprocessing+inference+postprocessing?

    Thanks and have a great day

    opened by FraPochetti 2
  • Download dataset

    Download dataset

    Hello, I am running the project recently. I hope you can provide it. Please pass the application. My account number is [email protected] Thank you very much.

    opened by dandanAD 1
  • ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

    ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

    When I am running the project from the following command:

    bash scripts/evaluate_atr_kwf1.sh
    

    I am getting the following error:

    Traceback (most recent call last):
      File "./exp/train.py", line 14, in <module>
        from torchvision import transforms
      File "/home/mayank/anaconda3/envs/iniy_oneshothumanparsing_041221/lib/python3.7/site-packages/torchvision/__init__.py", line 1, in <module>
        from torchvision import models
      File "/home/mayank/anaconda3/envs/iniy_oneshothumanparsing_041221/lib/python3.7/site-packages/torchvision/models/__init__.py", line 11, in <module>
        from . import detection
      File "/home/mayank/anaconda3/envs/iniy_oneshothumanparsing_041221/lib/python3.7/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
        from .faster_rcnn import *
      File "/home/mayank/anaconda3/envs/iniy_oneshothumanparsing_041221/lib/python3.7/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
        from torchvision.ops import misc as misc_nn_ops
      File "/home/mayank/anaconda3/envs/iniy_oneshothumanparsing_041221/lib/python3.7/site-packages/torchvision/ops/__init__.py", line 1, in <module>
        from .boxes import nms, box_iou
      File "/home/mayank/anaconda3/envs/iniy_oneshothumanparsing_041221/lib/python3.7/site-packages/torchvision/ops/boxes.py", line 2, in <module>
        from torchvision import _C
    ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
    
    

    This is the structure of the project:

    .
    ├── data
    │   ├── datasets
    │   │   ├── ATR_OS
    │   │   │   ├── list
    │   │   │   │   ├── meta_test_id.txt
    │   │   │   │   └── meta_train_id.txt
    │   │   │   └── support
    │   │   │       ├── meta_test_atr_supports.pkl
    │   │   │       └── meta_train_atr_supports.pkl
    │   │   ├── CIHP_OS
    │   │   │   └── list
    │   │   │       ├── meta_test_id.txt
    │   │   │       └── meta_train_id.txt
    │   │   └── LIP_OS
    │   │       └── list
    │   │           ├── meta_test_id.txt
    │   │           └── meta_train_id.txt
    │   └── pretrained_model
    │       └── deeplab_v3plus_v3.pth
    ├── dataloaders
    │   ├── mypath_atr.py
    │   ├── mypath_mhp.py
    │   ├── mypath.py
    │   ├── oshp_loader.py
    │   └── transforms.py
    ├── exp
    │   ├── __init__.py
    │   └── train.py
    ├── how_to_install.txt
    ├── imgs
    │   ├── Figure1.jpg
    │   └── Figure1.pdf
    ├── LICENSE
    ├── networks
    │   ├── deeplab_xception_synBN.py
    │   ├── eopnet.py
    │   ├── __init__.py
    │   └── popnet.py
    ├── README.md
    ├── requirements.txt
    ├── scripts
    │   ├── atr_eop_1wf1.sh
    │   ├── atr_eop_1wf2.sh
    │   ├── atr_eop_kwf1.sh
    │   ├── atr_eop_kwf2.sh
    │   ├── cihp_eop_kwf1.sh
    │   ├── cihp_eop_kwf2.sh
    │   ├── evaluate_atr_1wf1.sh
    │   ├── evaluate_atr_kwf1.sh
    │   ├── lip_etop_kw_f1.sh
    │   └── lip_etop_kw_f2.sh
    ├── sync_batchnorm
    │   ├── batchnorm.py
    │   ├── comm.py
    │   ├── __init__.py
    │   ├── replicate.py
    │   └── unittest.py
    └── utils
        ├── __init__.py
        ├── metric.py
        ├── sampler.py
        ├── test_human.py
        └── util.py
    

    Please help me.

    opened by mayanktiwariiiitdmj 1
  • size mismatch for prototype

    size mismatch for prototype

    bash scripts/evaluate_atr_kwf1.sh

    the content of evaluate_atr_kwf1.sh: export PYTHONPATH="${PYTHONPATH}:$(pwd)" python ./exp/train.py
    --batch 2
    --gpus 1
    --resume_model ./data/pretrained_model/atr_kwf1.pth
    --resume_epoch 39
    --lr 0.001
    --numworker 6
    --testInterval 10
    --hidden_layers 256
    --epochs 50
    --dataset atr
    --test_classes 18
    --fold 1
    --way 'kway'
    --structure kway_oshp
    --size 512
    --prototype_warmup 25
    --fg_weight 1.0
    --contrast_weight 1.0
    --testing_screening
    --feature_lvl mid
    --temperature 0.1
    --validate

    result: => Structure:kway_oshp, dataset:atr, way:kway, fold:1 => Saving log to: /home/fuzhi-dlj/workspace_big_bang/workspace/infant/human_parse/One-shot-Human-Parsing/exp/run_cihp/run_6 Constructing DeepLabv3+ model... Output stride: 16 Number of Input Channels: 3 => No model load! opts.resume_model ./data/pretrained_model/atr_kwf1.pth Traceback (most recent call last): File "./exp/train.py", line 560, in main(opts) File "./exp/train.py", line 381, in main net_.load_state_dict(x) File "/home/fuzhi-dlj/workspace/conda_list/pyt1736/lib/python3.6/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict self.class.name, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for EOPNet_kway: size mismatch for prototype: copying a param with shape torch.Size([20, 256]) from checkpoint, the shape in current model is torch.Size([18, 256]).

    opened by xddlj 0
Owner
null
:hot_pepper: R²SQL: "Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing." (AAAI 2021)

R²SQL The PyTorch implementation of paper Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing. (AAAI 2021) Requirement

huybery 60 Dec 31, 2022
Implementation of fast algorithms for Maximum Spanning Tree (MST) parsing that includes fast ArcMax+Reweighting+Tarjan algorithm for single-root dependency parsing.

Fast MST Algorithm Implementation of fast algorithms for (Maximum Spanning Tree) MST parsing that includes fast ArcMax+Reweighting+Tarjan algorithm fo

Miloš Stanojević 11 Oct 14, 2022
Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph".

multilingual-mrc-isdg Code for the AAAI 2022 paper "Zero-Shot Cross-Lingual Machine Reading Comprehension via Inter-Sentence Dependency Graph". This r

Liyan 5 Dec 7, 2022
Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

Vision Transformer with Progressive Sampling This is the official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

yuexy 123 Jan 1, 2023
Pytorch implementation of the paper Progressive Growing of Points with Tree-structured Generators (BMVC 2021)

PGpoints Pytorch implementation of the paper Progressive Growing of Points with Tree-structured Generators (BMVC 2021) Hyeontae Son, Young Min Kim Pre

Hyeontae Son 9 Jun 6, 2022
(AAAI2020)Grapy-ML: Graph Pyramid Mutual Learning for Cross-dataset Human Parsing

Grapy-ML: Graph Pyramid Mutual Learning for Cross-dataset Human Parsing This repository contains pytorch source code for AAAI2020 oral paper: Grapy-ML

null 54 Aug 4, 2022
[CVPR 2021] 'Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator'

[CVPR2021] Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator Overview This is the entire codebase for the paper

null 35 Dec 1, 2022
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
One implementation of the paper "DMRST: A Joint Framework for Document-Level Multilingual RST Discourse Segmentation and Parsing".

Introduction One implementation of the paper "DMRST: A Joint Framework for Document-Level Multilingual RST Discourse Segmentation and Parsing". Users

seq-to-mind 18 Dec 11, 2022
Code for the paper One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation, CVPR 2021.

One Thing One Click One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation (CVPR2021) Code for the paper One Thi

null 44 Dec 12, 2022
Multi-Stage Progressive Image Restoration

Multi-Stage Progressive Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Sh

Syed Waqas Zamir 859 Dec 22, 2022
Progressive Domain Adaptation for Object Detection

Progressive Domain Adaptation for Object Detection Implementation of our paper Progressive Domain Adaptation for Object Detection, based on pytorch-fa

null 96 Nov 25, 2022
Code for Referring Image Segmentation via Cross-Modal Progressive Comprehension, CVPR2020.

CMPC-Refseg Code of our CVPR 2020 paper Referring Image Segmentation via Cross-Modal Progressive Comprehension. Shaofei Huang*, Tianrui Hui*, Si Liu,

spyflying 55 Dec 1, 2022
Official code of paper "PGT: A Progressive Method for Training Models on Long Videos" on CVPR2021

PGT Code for paper PGT: A Progressive Method for Training Models on Long Videos. Install Run pip install -r requirements.txt. Run python setup.py buil

Bo Pang 27 Mar 30, 2022
PyTorch implementation of Progressive Growing of GANs for Improved Quality, Stability, and Variation.

PyTorch implementation of Progressive Growing of GANs for Improved Quality, Stability, and Variation. Warning: the master branch might collapse. To ob

null 559 Dec 14, 2022
PyTorch inference for "Progressive Growing of GANs" with CelebA snapshot

Progressive Growing of GANs inference in PyTorch with CelebA training snapshot Description This is an inference sample written in PyTorch of the origi

null 320 Nov 21, 2022
Source code for "Progressive Transformers for End-to-End Sign Language Production" (ECCV 2020)

Progressive Transformers for End-to-End Sign Language Production Source code for "Progressive Transformers for End-to-End Sign Language Production" (B

null 58 Dec 21, 2022
Progressive Coordinate Transforms for Monocular 3D Object Detection

Progressive Coordinate Transforms for Monocular 3D Object Detection This repository is the official implementation of PCT. Introduction In this paper,

null 58 Nov 6, 2022
Efficient Conformer: Progressive Downsampling and Grouped Attention for Automatic Speech Recognition

Efficient Conformer: Progressive Downsampling and Grouped Attention for Automatic Speech Recognition Official implementation of the Efficient Conforme

Maxime Burchi 145 Dec 30, 2022