Good Semi-Supervised Learning That Requires a Bad GAN

Zhilin Yang

Last update: Dec 12, 2022

Related tags

Deep Learning ssl_bad_gan

Overview

Good Semi-Supervised Learning that Requires a Bad GAN

This is the code we used in our paper

Good Semi-supervised Learning that Requires a Bad GAN

Zihang Dai*, Zhilin Yang*, Fan Yang, William W. Cohen, Ruslan Salakhutdinov (*: equal contribution)

NIPS 2017

Requirements

The repo supports python 2.7 + pytorch 0.1.12. To install pytorch 0.1.12, run conda install pytorch=0.1.12 cuda80 -c soumith.

Get Pretrained PixelCNN Model

mkdir model
cd model
wget http://kimi.ml.cmu.edu/mnist.True.3.best.pixel

Run the Code

To reproduce our results on MNIST

python mnist_trainer.py

To reproduce our results on SVHN

python svhn_trainer.py

To reproduce our results on CIFAR-10

python cifar_trainer.py

Results

Here is a comparison of different models using standard architectures without ensembles (100 labels on MNIST, 1000 labels on SVHN, and 4000 labels on CIFAR):

Method	MNIST (# errors)	SVHN (% errors)	CIFAR (% errors)
CatGAN	191 +/- 10	-	19.58 +/- 0.46
SDGM	132 +/- 7	16.61 +/- 0.24	-
Ladder Network	106 +/- 37	-	20.40 +/- 0.47
ADGM	96 +/- 2	22.86	-
FM	93 +/- 6.5	8.11 +/- 1.3	18.63 +/- 2.32
ALI	-	7.42 +/- 0.65	17.99 +/- 1.62
VAT small	136	6.83	14.87
Ours	79.5 +/- 9.8	4.25 +/- 0.03	14.41 +/- 0.30

You might also like...

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning This is the PyTorch implementation of our paper: FeatMatch: Feature-Based Augmentat

43 Nov 19, 2022

Code for CoMatch: Semi-supervised Learning with Contrastive Graph Regularization

CoMatch: Semi-supervised Learning with Contrastive Graph Regularization (Salesforce Research) This is a PyTorch implementation of the CoMatch paper [B

107 Dec 14, 2022

Semi-supervised Learning for Sentiment Analysis

Neural-Semi-supervised-Learning-for-Text-Classification-Under-Large-Scale-Pretraining Code, models and Datasets for《Neural Semi-supervised Learning fo

47 Jan 1, 2023

The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

DS3L This is the code for paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020. Setups The code is implem

36 Oct 19, 2022

noisy labels; missing labels; semi-supervised learning; entropy; uncertainty; robustness and generalisation.

ProSelfLC: CVPR 2021 ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks For any specific discussion or potential fu

57 Dec 4, 2022

Semi-Supervised Learning, Object Detection, ICCV2021

End-to-End Semi-Supervised Object Detection with Soft Teacher By Mengde Xu*, Zheng Zhang*, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai,

789 Dec 27, 2022

Semi Supervised Learning for Medical Image Segmentation, a collection of literature reviews and code implementations.

Semi-supervised-learning-for-medical-image-segmentation. Recently, semi-supervised image segmentation has become a hot topic in medical image computin

1.3k Jan 3, 2023

The Rich Get Richer: Disparate Impact of Semi-Supervised Learning

The Rich Get Richer: Disparate Impact of Semi-Supervised Learning Preprocess file of the dataset used in implicit sub-populations: (Demographic groups

4 Oct 14, 2022

Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

SSRL-for-image-classification Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

2 Nov 19, 2021

Comments

model encoder implemented as CNN?

If I understand correctly, the encoder is there to implement the Variational Inference term for enhancing entropy of the feature space. The latent z variable is represented by diagonal Gaussian.

However, in the model.py code, I see a sequence of CNNs defined. I am not able to make the connection, how you implement this VI using a chain of CNNs?

opened by bhomass 0
expand_as error

I cloned the repo and made the modifications to run with python 3.6. When running mnist_train.py, I get this error RuntimeError: The expanded size of the tensor (3) must match the existing size (160) at non-singleton dimension 3. Target sizes: [160, 2, 2, 3]. Tensor sizes: [160, 1, 1, 160]

on norm_weight = self.weight * (weight_scale[:,None,None,None] / torch.sqrt((self.weight ** 2).sum(3).sum(2).sum(1) + 1e-6)).expand_as(self.weight)

I checked all the dimensions leading to this statement self.weight.shape = [160, 2, 2, 3] weight_scale[:,None,None,None] shape = [160, 1, 1, 1] the torch.sqrt term shape = [160] but the division came out to weight_scale[:,None,None,None] / torch.sqrt((self.weight ** 2).sum(3).sum(2).sum(1) + 1e-6) shape = [160, 1, 1, 160]

This I am quite puzzled. Why would a shape [160, 1, 1, 1] divided by shape [160] lead to [160, 1, 1, 160]?

This shape, in term, fails to expand_as to [160, 2, 2, 3], which is probably reasonable.

opened by bhomass 0
Hi,after I replace the code ,please help

Hi, It is the problem of tensor dimension. Just replace the code norm_weight = self.weight * (weight_scale[None,:,None,None] / torch.sqrt((self.weight ** 2).sum(3).sum(2).sum(0) + 1e-6)).expand_as(self.weight) with norm_weight = self.weight * (weight_scale[None,:,None,None] / torch.sqrt((self.weight ** 2).sum(3).sum(2).sum(0) + 1e-6).reshape([-1, 1, 1, 1])).expand_as(self.weight)

Originally posted by @zcwang0702 in https://github.com/kimiyoung/ssl_bad_gan/issues/6#issuecomment-456633881

but after I replace the code new error: RuntimeError: The expanded size of the tensor (128) must match the existing size (3) at non-singleton dimension 0

original code error : File "/home/gis/PycharmProjects/guo/ssl_bad_gan-master/model.py", line 165, in forward norm_weight = self.weight * (weight_scale[None,:,None,None] / torch.sqrt((self.weight ** 2).sum(3).sum(2).sum(0) + 1e-6)).expand_as(self.weight) RuntimeError: The expanded size of the tensor (5) must match the existing size (3) at non-singleton dimension 3 Thank you!

opened by gden138 2
question about cifar10

I run the code cifar_trainer.py . No changes to code, data and parameters. And the pytorch version is 0.1.12. But the result does not reach the 14.41 that you published in the paper until #400000 iterations. Could you tell me how the hyper parameters is set? Is the discriminator learning rate 6e-4 or 3e-4 by default?

opened by congxin513 0

Owner

Zhilin Yang

GitHub

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

49 Jan 2, 2023

Code to reproduce experiments in the paper "Explainability Requires Interactivity".

Explainability Requires Interactivity This repository contains the code to train all custom models used in the paper Explainability Requires Interacti

5 Apr 7, 2022

Updated for TTS(CE) = Also Known as TTN V3. The code requires the first server to be 'ttn' protocol.

Updated Updated for TTS(CE) = Also Known as TTN V3. The code requires the first server to be 'ttn' protocol. Introduction This balenaCloud (previously

1 Oct 17, 2021

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

5 Dec 10, 2022

FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

191 Dec 31, 2022

DR-GAN: Automatic Radial Distortion Rectification Using Conditional GAN in Real-Time

DR-GAN: Automatic Radial Distortion Rectification Using Conditional GAN in Real-Time Introduction This is official implementation for DR-GAN (IEEE TCS

18 Dec 23, 2022

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder arXiv p

10 Oct 11, 2022

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.

Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning Overview This code is for paper: Not All Unlabeled Data are Equa

22 Nov 23, 2022

This is the official implementation for "Do Transformers Really Perform Bad for Graph Representation?".

Graphormer By Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng*, Guolin Ke, Di He*, Yanming Shen and Tie-Yan Liu. This repo is the official impl

1.3k Dec 29, 2022

This is the official implementation for "Do Transformers Really Perform Bad for Graph Representation?".

Graphormer By Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng*, Guolin Ke, Di He*, Yanming Shen and Tie-Yan Liu. This repo is the official impl

1.3k Dec 26, 2022

Good Semi-Supervised Learning That Requires a Bad GAN

Related tags

Overview

Good Semi-Supervised Learning that Requires a Bad GAN

Requirements

Get Pretrained PixelCNN Model

Run the Code

Results

You might also like...

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

Code for CoMatch: Semi-supervised Learning with Contrastive Graph Regularization

Semi-supervised Learning for Sentiment Analysis

The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

noisy labels; missing labels; semi-supervised learning; entropy; uncertainty; robustness and generalisation.

Semi-Supervised Learning, Object Detection, ICCV2021

Semi Supervised Learning for Medical Image Segmentation, a collection of literature reviews and code implementations.

The Rich Get Richer: Disparate Impact of Semi-Supervised Learning

Semi-supervised Representation Learning for Remote Sensing Image Classification Based on Generative Adversarial Networks

Comments

model encoder implemented as CNN?

expand_as error

Hi,after I replace the code ,please help

question about cifar10

Owner

Zhilin Yang

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

Code to reproduce experiments in the paper "Explainability Requires Interactivity".

Updated for TTS(CE) = Also Known as TTN V3. The code requires the first server to be 'ttn' protocol.

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

DR-GAN: Automatic Radial Distortion Rectification Using Conditional GAN in Real-Time

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

This is the official implementation for "Do Transformers Really Perform Bad for Graph Representation?".

This is the official implementation for "Do Transformers Really Perform Bad for Graph Representation?".

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.