A PyTorch-based library for semi-supervised learning

Related tags

Deep Learning toolkit pytorch semi-supervised-learning codebase flexmatch fixmatch
Overview

News

If you want to join TorchSSL team, please e-mail Yidong Wang ([email protected]; [email protected]) for more information. We plan to add more SSL algorithms and expand TorchSSL from CV to NLP and Speech.

TorchSSL: A PyTorch-based Toolbox for Semi-Supervised Learning

An all-in-one toolkit based on PyTorch for semi-supervised learning (SSL). We implmented 9 popular SSL algorithms to enable fair comparison and boost the development of SSL algorithms.

FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling(https://arxiv.org/abs/2110.08263)

Supported algorithms

We support fully supervised training + 9 popular SSL algorithms as listed below:

  1. Pi-Model [1]
  2. MeanTeacher [2]
  3. Pseudo-Label [3]
  4. VAT [4]
  5. MixMatch [5]
  6. UDA [6]
  7. ReMixMatch [7]
  8. FixMatch [8]
  9. FlexMatch [9]

Besides, we implement our Curriculum Pseudo Labeling (CPL) method for Pseudo-Label (Flex-Pseudo-Label) and UDA (Flex-UDA).

Supported datasets

We support 5 popular datasets in SSL research as listed below:

  1. CIFAR-10
  2. CIFAR-100
  3. STL-10
  4. SVHN
  5. ImageNet

Installation

  1. Prepare conda
  2. Run conda env create -f environment.yml

Usage

It is convenient to perform experiment with TorchSSL. For example, if you want to perform FlexMatch algorithm:

  1. Modify the config file in config/flexmatch/flexmatch.yaml as you need
  2. Run python flexmatch --c config/flexmatch/flexmatch.yaml

Customization

If you want to write your own algorithm, please follow the following steps:

  1. Create a directory for your algorithm, e.g., SSL, write your own model file SSl/SSL.py in it.
  2. Write the training file in SSL.py
  3. Write the config file in config/SSL/SSL.yaml

Results

avatar avatar avatar avatar

Citation

If you think this toolkit or the results are helpful to you and your research, please cite our paper:

@article{zhang2021flexmatch},
  title={FlexMatch: Boosting Semi-supervised Learning with Curriculum Pseudo Labeling},
  author={Zhang, Bowen and Wang, Yidong and Hou Wenxin and Wu, Hao and Wang, Jindong and Okumura, Manabu and Shinozaki, Takahiro},
  booktitle={Neural Information Processing Systems (NeurIPS)},
  year={2021}
}

Maintainer

Yidong Wang1, Hao Wu2, Bowen Zhang1, Wenxin Hou1,3, Jindong Wang3

Shinozaki Lab1 http://www.ts.ip.titech.ac.jp/

Okumura Lab2 http://lr-www.pi.titech.ac.jp/wp/

Microsoft Research Asia3

References

[1] Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi-supervised learning with ladder networks. InNeurIPS, pages 3546–3554, 2015.

[2] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagedconsistency targets improve semi-supervised deep learning results. InNeurIPS, pages 1195–1204, 2017.

[3] Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning methodfor deep neural networks. InWorkshop on challenges in representation learning, ICML,volume 3, 2013.

[4] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training:a regularization method for supervised and semi-supervised learning.IEEE TPAMI, 41(8):1979–1993, 2018.

[5] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and ColinRaffel. Mixmatch: A holistic approach to semi-supervised learning.NeurIPS, page 5050–5060,2019.

[6] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmen-tation for consistency training.NeurIPS, 33, 2020.

[7] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang,and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching andaugmentation anchoring. InICLR, 2019.

[8] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raf-fel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence.NeurIPS, 33, 2020.

[9] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao wu, Jindong Wang, Okumura Manabu, and Shinozaki Takahiro. FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling. NeurIPS, 2021.

Comments
  • Ask for the code

    Ask for the code

    Hi, I am interested in your work. research was very interesting. I want to know if 'Flexmatch' in the corresponding code reproduces the results of the paper. I tried CIFAR100_400 case, but the top-1 score doesn't go up.

    I look at the code and have a question related to it. Is it right that classwise_acc all starts at 0.? Doesn't this mean that the threshold of each class is zero? This seems to be a factor in lowering the score by breaking the model. Is it the intention of the paper to raise the threshold from zero? (I'm sorry if I didn't understand.)

    Is there any other way to train the model properly?

    Best Regards, Harim

    opened by harimkang 22
  • Release of training log or tensorboard for FlexMatch on CIFAR100

    Release of training log or tensorboard for FlexMatch on CIFAR100

    Dear authors,

    Thanks for the impressive work FlexMatch and the awesome codebase!

    Recently, I am trying to conduct some experiments on CIFAR100 based on this repo and I just found that FlexMatch training may cost around 50min for 5k iterations (3 * NVIDIA RTX2080Ti) which may take 7 days for the whole training. It is quite a long period.

    I wonder if it is possible for you to release the training log or tensorboard for FlexMatch method on CIFAR100 to provide me a reference which would help me a lot. Many many thanks~

    Cheers, Haiming

    opened by HeimingX 10
  • tensorboard not working

    tensorboard not working

    Hello, and many thanks for creating this repo!

    I've run several experiments, the log gives the expected results but when i run tensorboard I consistently get "No dashboards active". Am I doing something wrong?

    opened by nikoskaraliolios 8
  • Some questions about multi-gpu training

    Some questions about multi-gpu training

    Firstly, thanks a lot for open source this code base which will help the development of semi-supervised field. I have questions about multi-gpu training and look forward to your reply: As far as I understand, this code supports multi-gpu training and I want to know have you ever tested different parameters in multi-gpu environment, for example, increasing batch size when using more gpu. And how did you set the parameters in .yaml when you have multiple gpus in one machine?

    opened by wanghao14 7
  • Reproduce Numbers on CIFAR100

    Reproduce Numbers on CIFAR100

    Hi,

    Thanks for the great work. I tried to reproduce FlexMatch number on CIFAR100 with 400 labels. I followed the instructions to create a conda env and ran python flexmatch.py --c config/flexmatch/flexmatch_cifar100_400_0.yaml with 3 Tesla V100.

    Unfortunately, the best top-1 accuracy I got was 60.65, which is 6.91% lower than the reported number in the paper. There seems to be a sharp performance drop at around iteration 600K. But I couldn't pinpoint the issue from the training statistics. I wonder if you also have observed similar behaviors. It would be great if you could offer some insights here. Thanks! image

    image

    Here is the tensorboard file: tf_logs.zip

    Besides, the curve of the mask ratio looks a bit strange to me. Because 1.0 - mask.detach() is actually logged in the code, so, shouldn't it start from 1 and then decrease? Any intuition why it starts from 0 and increases at the very beginning? Thanks!

    opened by YUE-FAN 6
  • Results in the paper

    Results in the paper

    Thanks for your great work!

    Just one quick question, how many GPUs were used to obtain the results in the paper? I didn't seem to find the specification on this.

    Best

    opened by ZhuoranYu 5
  • DataLoader worker (pid 12847) is killed by signal: Killed

    DataLoader worker (pid 12847) is killed by signal: Killed

    When I run python flexmatch.py --c ./config/flexmatch/flexmatch_cifar100_400_1.yaml, I always get the following error. Can you help me to fix the mistake?

    1635170601

    opened by ljjcoder 5
  • AttrubuteError

    AttrubuteError

    I put the custom data set using the same data loader as Imagenet, and organize my data as: "imagenet"/{train or val}/class name/*.jpg
    image

    But this situation still appears, how should I solve it?
    image

    and there is the [isic.yaml.] setting. isic.txt

    opened by tangwwwwww 4
  • supervised to semi-supervised ?

    supervised to semi-supervised ?

    Hi, I just discovered your repo, its great ! However, I was wondering if it was possible to modify for example a supervised training loop that i have to do semi-supervised learning ? How would one adapt and use your repo to achieve such task ? for example if i am currently using detectron2 module to do training, is there a way i can modify train.py to do semi-supervised learning ?

    Thank you so much !

    opened by an99990 4
  • Validation set for CIFAR10

    Validation set for CIFAR10

    Thanks for your great work!

    But it seems there is no validation set sampling part in this repository.

    As far as I know, in general, 5,000 images are subtracted from the training set and used as validation. [1] Official FixMatch, validation set size options, fixmatch [2] A. Oliver, et al., "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms"

    I tried to find it in the datasets/ssl_dataset.py, but I couldn't. May I know where the validation set part, or why it's not there? Thanks!

    opened by Holim0711 4
  • Regarding the warmup in the snippet.

    Regarding the warmup in the snippet.

    Hi, Thanks a lot for your contribution.

    I wanted to ask about the warmup thing. In your flexmatch training code there is args.warmup inside the if condition if max(pseudo_counter.values()) < len(self.ulb_dset):

    Will it not everytime go in this if condition and args.warmup if condition? Because I printed the if statement and through the training it goes in args.warmup if condition.

    So ACC to paper the warmup should get over after some time? Is there a issue in a code or am I understanding it in a wrong way?

    Thanks a lot, Shreejal

    opened by shreejalt 3
  • Fix the save_model() bug.

    Fix the save_model() bug.

    I think there's a small bug when saving the model.

    The model is saved before the self.it is updated, so when training is resumed, the model starts with the same self.it. However, it should be self.it + 1

    https://github.com/TorchSSL/TorchSSL/blob/f2f46076cbea1b6f6c9b3c1c45609502c6576250/models/fixmatch/fixmatch.py#L191-L196

    https://github.com/TorchSSL/TorchSSL/blob/f2f46076cbea1b6f6c9b3c1c45609502c6576250/models/fixmatch/fixmatch.py#L220

    In my workaround, take note that you can only use save_model() after updating the model and before updating self.it.

    opened by PM25 0
  • Drop in accuracy on resumt

    Drop in accuracy on resumt

    I am running this code on a server that has a time limit of one day per job. So, I need to resume the code. I see a drop in accuracy when the training is resuming. Could you please comment on what might be causing such drops?

    opened by ShuvenduRoy 4
  • selected_label

    selected_label

    Hello, I read the code you open source to Github and found that the size of selected_label is the size of the entire unlabeled dataset. When updating selected_label later, the index used is the current batch size index, for example, batch is 64, X_ULb_idx is the index range of 0-63. Or the index of the original dataset, I don't know if you can understand this description?

    opened by lzw1997lzw 1
  • question about different version of code.

    question about different version of code.

    Thanks for your excellent work. I found that the results in cifar100 are different in different versions of paper. I noticed that your code also changed five months ago. I downloaded your code on 2021.10.19 and experimented on it. Is the code of 2021.10.19 different from the current one? Do I need to redo the experiment on the current code?

    opened by ljjcoder 1
Owner
null
GitHub
PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning This is the PyTorch implementation of our paper: FeatMatch: Feature-Based Augmentat

null 43 Nov 19, 2022
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 5 Dec 10, 2022
Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

SSRL-for-image-classification Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Feng 2 Nov 19, 2021
This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

CG3 This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning]. R

null 12 Oct 28, 2022
CVPR2022 paper "Dense Learning based Semi-Supervised Object Detection"

[CVPR2022] DSL: Dense Learning based Semi-Supervised Object Detection DSL is the first work on Anchor-Free detector for Semi-Supervised Object Detecti

Bhchen 69 Dec 8, 2022
CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder arXiv p

Fraunhofer SCAI 10 Oct 11, 2022
code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning Overview This code is for paper: Not All Unlabeled Data are Equa

Jason Ren 22 Nov 23, 2022
Source codes for the paper "Local Additivity Based Data Augmentation for Semi-supervised NER"

LADA This repo contains codes for the following paper: Jiaao Chen*, Zhenghui Wang*, Ran Tian, Zichao Yang, Diyi Yang: Local Additivity Based Data Augm

GT-SALT 36 Dec 2, 2022
Group R-CNN for Point-based Weakly Semi-supervised Object Detection (CVPR2022)

Group R-CNN for Point-based Weakly Semi-supervised Object Detection (CVPR2022) By Shilong Zhang*, Zhuoran Yu*, Liyang Liu*, Xinjiang Wang, Aojun Zhou,

Shilong Zhang 129 Dec 24, 2022
PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

Unbiased Teacher for Semi-Supervised Object Detection This is the PyTorch implementation of our paper: Unbiased Teacher for Semi-Supervised Object Detection

Facebook Research 366 Dec 28, 2022
Semi-supervised Video Deraining with Dynamical Rain Generator (CVPR, 2021, Pytorch)

S2VD Semi-supervised Video Deraining with Dynamical Rain Generator (CVPR, 2021) Requirements and Dependencies Ubuntu 16.04, cuda 10.0 Python 3.6.10, P

Zongsheng Yue 53 Nov 23, 2022
A PyTorch implementation of "Semi-Supervised Graph Classification: A Hierarchical Graph Perspective" (WWW 2019)

SEAL ⠀⠀⠀ A PyTorch implementation of Semi-Supervised Graph Classification: A Hierarchical Graph Perspective (WWW 2019) Abstract Node classification an

Benedek Rozemberczki 202 Dec 27, 2022
Official pytorch code for SSC-GAN: Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation(ICCV 2021)

SSC-GAN_repo Pytorch implementation for 'Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation'.PDF SSC-GAN:Sem

tyty 4 Aug 28, 2022
Code for CoMatch: Semi-supervised Learning with Contrastive Graph Regularization

CoMatch: Semi-supervised Learning with Contrastive Graph Regularization (Salesforce Research) This is a PyTorch implementation of the CoMatch paper [B

Salesforce 107 Dec 14, 2022
Semi-supervised Learning for Sentiment Analysis

Neural-Semi-supervised-Learning-for-Text-Classification-Under-Large-Scale-Pretraining Code, models and Datasets for《Neural Semi-supervised Learning fo

null 47 Jan 1, 2023
The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

DS3L This is the code for paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020. Setups The code is implem

Guolz 36 Oct 19, 2022
noisy labels; missing labels; semi-supervised learning; entropy; uncertainty; robustness and generalisation.

ProSelfLC: CVPR 2021 ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks For any specific discussion or potential fu

amos_xwang 57 Dec 4, 2022
Semi-Supervised Learning, Object Detection, ICCV2021

End-to-End Semi-Supervised Object Detection with Soft Teacher By Mengde Xu*, Zheng Zhang*, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai,

Microsoft 789 Dec 27, 2022
Semi Supervised Learning for Medical Image Segmentation, a collection of literature reviews and code implementations.

Semi-supervised-learning-for-medical-image-segmentation. Recently, semi-supervised image segmentation has become a hot topic in medical image computin

Healthcare Intelligence Laboratory 1.3k Jan 3, 2023