Pytorch implementation of paper Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

Hrishikesh Kamath

Last update: Nov 20, 2022

Related tags

Overview

PATE Example

The scripts in the folder allow you to train a MNIST model using PATE diffrential privacy framework. While running this example would give you an accurate implementation of a PATE implementation. An accurate analysis of DP guarantees is still an work in progress.

[Sorry, I wasn't a very good coder when I wrote this. But, not maintaining this anymore]

Requirements:

Pytorch
PySyft

$ python Main.py

Scripts present

data: Consists of functions for loading datasets
Main: The file to be run for a complete PATE model
Model: PyTorch model definition. The same model is used for student and teacher.
Student: Class to handle student functionality such as training and making predictions
Teacher: Class to handle teacher functionality such as training and making noisy predictions. All the Teacher ensembles are handled in this Class
util: Functions

This training loop is then executed for a given number of epochs. The performance on the test set of MNIST is shown after each epoch.

You might also like...

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

5 Dec 10, 2022

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

47 Nov 6, 2022

Source codes for the paper "Local Additivity Based Data Augmentation for Semi-supervised NER"

LADA This repo contains codes for the following paper: Jiaao Chen*, Zhenghui Wang*, Ran Tian, Zichao Yang, Diyi Yang: Local Additivity Based Data Augm

36 Dec 2, 2022

[CVPR 2021] MiVOS - Mask Propagation module. Reproduced STM (and better) with training code :star2:. Semi-supervised video object segmentation evaluation.

MiVOS (CVPR 2021) - Mask Propagation Ho Kei Cheng, Yu-Wing Tai, Chi-Keung Tang [arXiv] [Paper PDF] [Project Page] [Papers with Code] This repo impleme

106 Jan 3, 2023

ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

ST++ This is the official PyTorch implementation of our paper: ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation. Lihe Ya

147 Jan 3, 2023

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

UniSpeech The family of UniSpeech: UniSpeech (ICML 2021): Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR UniSpeech-

282 Jan 9, 2023

PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

Unbiased Teacher for Semi-Supervised Object Detection This is the PyTorch implementation of our paper: Unbiased Teacher for Semi-Supervised Object Detection

366 Dec 28, 2022

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos. By adopting a unified pipeline-based API design, PyKale enforces standardization and minimalism, via reusing existing resources, reducing repetitions and redundancy, and recycling learning models across areas.

370 Dec 27, 2022

This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

CG3 This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning]. R

12 Oct 28, 2022

Comments

Can't seem to finish running Main.py (not SHVN)

Hey,

Love the repo, but I have an issue, whenever I run it it seems to get stuck at the last two lines of Main.py it does not show an error, but after a while i press force it to stop and its stuck at the same line every time. I run it on less epochs than in your file (1-2) just for testing purposes. My hyperparameters look like this:

    def __init__(self):

        self.batchsize = 64
        self.test_batchsize = 10
        self.epochs = **1**
        self.student_epochs = **1**
        self.lr = 0.01
        self.momentum = 0.5
        self.no_cuda = False
        self.seed = 1
        self.log_interval = 30
        self.n_teachers = **1**
        self.save_model = **True**

Last four lines of code look like this:

counts_lol = torch.stack(counts).contiguous().view(**1**, 10000)
predict_lol = torch.tensor(predict).view(10000)

data_dep_eps, data_ind_eps = teacher.analyze(counts_lol, predict_lol, moments=20)
print("Epsilon: ", teacher.analyze(counts_lol, predict_lol))

And it seems that my file is stuck at:

Traceback (most recent call last):
  File "main.py", line 91, in <module>
    #data_dep_eps, data_ind_eps = teacher.analyze(counts_lol, predict_lol, moments=20)
  File "C:\Users\Koen\PycharmProjects\pate-g\PATE\Teacher.py", line 176, in analyze
    preds, indices, noise_eps=0.1, delta=self.epsilon, moments=moments, beta=0.09
  File "C:\Users\Koen\Anaconda3\envs\pysyft\lib\site-packages\syft-0.2.0a2-py3.7.egg\syft\frameworks\torch\differential_privacy\pate.py", line 484, in perform_analysis_torch
  File "C:\Users\Koen\Anaconda3\envs\pysyft\lib\site-packages\syft-0.2.0a2-py3.7.egg\syft\frameworks\torch\differential_privacy\pate.py", line 484, in <listcomp>
  File "C:\Users\Koen\Anaconda3\envs\pysyft\lib\site-packages\syft-0.2.0a2-py3.7.egg\syft\frameworks\torch\differential_privacy\pate.py", line 379, in logmgf_from_counts_torch
  File "C:\Users\Koen\Anaconda3\envs\pysyft\lib\site-packages\syft-0.2.0a2-py3.7.egg\syft\frameworks\torch\differential_privacy\pate.py", line 354, in compute_q_noisy_max_torch
  File "C:\Users\Koen\Anaconda3\envs\pysyft\lib\site-packages\syft-0.2.0a2-py3.7.egg\syft\frameworks\torch\differential_privacy\pate.py", line 294, in tensors_to_literals
  File "C:\Users\Koen\Anaconda3\envs\pysyft\lib\site-packages\torch\tensor.py", line 427, in <lambda>
    return iter(imap(lambda i: self[i], range(self.size(0))))

Would you have any idea what I'm doing wrong here? I'm hesitant to run it for 1000 epochs as it would take a lot of time and I might still get stuck at the same point. I am very curious to what the epsilon is and want to play around to see what the actual loss of accuracy would be with my own datasets.

Thanks in advance!

opened by Koen-Vannisselroij 3

Problem When I run Main.py

the error msg is as follows:

Traceback (most recent call last):
  File "Main.py", line 3, in 
    from Teacher import Teacher
  File "E:\1-Research\0-DP+GCN\Code\(2020-09-20)PATE\Teacher.py", line 6, in 
    from syft.frameworks.torch.differential_privacy import pate
  File "D:\Anaconda\envs\torch\lib\site-packages\syft-0.2.9-py3.7.egg\syft\__init__.py", line 14, in 
    import syft.frameworks.torch.hook.hook_args
  File "D:\Anaconda\envs\torch\lib\site-packages\syft-0.2.9-py3.7.egg\syft\frameworks\torch\hook\hook_args.py", line 4, in 
    from syft.frameworks.torch.tensors.interpreters.native import TorchTensor
  File "D:\Anaconda\envs\torch\lib\site-packages\syft-0.2.9-py3.7.egg\syft\frameworks\torch\tensors\interpreters\native.py", line 11, in 
    from syft.messaging.message import TensorCommandMessage
  File "D:\Anaconda\envs\torch\lib\site-packages\syft-0.2.9-py3.7.egg\syft\messaging\message.py", line 17, in 
    from syft.execution.action import Action
  File "D:\Anaconda\envs\torch\lib\site-packages\syft-0.2.9-py3.7.egg\syft\execution\action.py", line 5, in 
    from syft.execution.placeholder import PlaceHolder
  File "D:\Anaconda\envs\torch\lib\site-packages\syft-0.2.9-py3.7.egg\syft\execution\placeholder.py", line 7, in 
    from syft_proto.execution.v1.placeholder_pb2 import Placeholder as PlaceholderPB
  File "D:\Anaconda\envs\torch\lib\site-packages\syft_proto-0.5.2-py3.7.egg\syft_proto\execution\v1\placeholder_pb2.py", line 14, in 
    from syft_proto.types.syft.v1 import id_pb2 as syft__proto_dot_types_dot_syft_dot_v1_dot_id__pb2
  File "D:\Anaconda\envs\torch\lib\site-packages\syft_proto-0.5.2-py3.7.egg\syft_proto\types\syft\v1\id_pb2.py", line 21, in 
    create_key=_descriptor._internal_create_key,
AttributeError: module 'google.protobuf.descriptor' has no attribute '_internal_create_key'

opened by Billy1900 2

Problem when I run /PATE/SHVN/Main.py

$ python Main.py Using downloaded and verified file: ../data/train_32x32.mat Using downloaded and verified file: ../data/test_32x32.mat TRAINING model_0 EPOCH: 1 Traceback (most recent call last): File "Main.py", line 35, in teacher.train(train_loader) File "/home/liujj/PATE/SHVN/Teacher.py", line 116, in train self.loop_body(split[index],model_name,1) File "/home/liujj/PATE/SHVN/Teacher.py", line 129, in loop_body output = model(data) File "/mnt/data/conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/liujj/PATE/SHVN/Model.py", line 18, in forward x = F.relu(self.conv1(x)) File "/mnt/data/conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/mnt/data/conda/envs/pytorch/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 338, in forward self.padding, self.dilation, self.groups) RuntimeError: Given groups=1, weight of size 20 1 5 5, expected input[64, 3, 32, 32] to have 1 channels, but got 3 channels instead

what is the RuntimeError mean? Can you help me solve?

opened by Huangpq2019 2

Pytorch implementation of paper Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

Related tags

Overview

PATE Example

Requirements:

You might also like...

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Source codes for the paper "Local Additivity Based Data Augmentation for Semi-supervised NER"

[CVPR 2021] MiVOS - Mask Propagation module. Reproduced STM (and better) with training code :star2:. Semi-supervised video object segmentation evaluation.

ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

Comments

Can't seem to finish running Main.py (not SHVN)

Problem When I run Main.py

Problem when I run /PATE/SHVN/Main.py

Owner

Hrishikesh Kamath

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.

The code for our paper Semi-Supervised Learning with Multi-Head Co-Training

[IJCAI-2021] A benchmark of data-free knowledge distillation from paper "Contrastive Model Inversion for Data-Free Knowledge Distillation"

🌈 PyTorch Implementation for EMNLP'21 Findings "Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer"

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

TF2 implementation of knowledge distillation using the "function matching" hypothesis from the paper Knowledge distillation: A good teacher is patient and consistent by Beyer et al.

Pytorch implementation of paper Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data

Related tags

Overview

PATE Example

Requirements:

You might also like...

Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Source codes for the paper "Local Additivity Based Data Augmentation for Semi-supervised NER"

[CVPR 2021] MiVOS - Mask Propagation module. Reproduced STM (and better) with training code :star2:. Semi-supervised video object segmentation evaluation.

ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation

Unified Pre-training for Self-Supervised Learning and Supervised Learning for ASR

PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

This is the repository for the AAAI 21 paper [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning].

Comments

Can't seem to finish running Main.py (not SHVN)

Problem When I run Main.py

Problem when I run /PATE/SHVN/Main.py

Owner

Hrishikesh Kamath

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

The code for our paper Semi-Supervised Learning with Multi-Head Co-Training

[IJCAI-2021] A benchmark of data-free knowledge distillation from paper "Contrastive Model Inversion for Data-Free Knowledge Distillation"

🌈 PyTorch Implementation for EMNLP'21 Findings "Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer"

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

PyTorch code for the paper: FeatMatch: Feature-Based Augmentation for Semi-Supervised Learning

CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

TF2 implementation of knowledge distillation using the "function matching" hypothesis from the paper Knowledge distillation: A good teacher is patient and consistent by Beyer et al.

code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren, Raymond A. Yeh, Alexander G. Schwing.