Improving Transferability of Representations via Augmentation-Aware Self-Supervision

hankook

Last update: Sep 16, 2022

Related tags

Deep Learning AugSelf

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Accepted to NeurIPS 2021

TL;DR: Learning augmentation-aware information by predicting the difference between two augmented samples improves the transferability of representations.

Dependencies

conda create -n AugSelf python=3.8 pytorch=1.7.1 torchvision=0.8.2 cudatoolkit=10.1 ignite -c pytorch
conda activate AugSelf
pip install scipy tensorboard kornia==0.4.1 sklearn

Checkpoints

We provide ImageNet100-pretrained models in this Dropbox link.

Pretraining

We here provide SimSiam+AugSelf pretraining scripts. For training the baseline (i.e., no AugSelf), remove --ss-crop and --ss-color options. For using other frameworks like SimCLR, use the --framework option.

STL-10

CUDA_VISIBLE_DEVICES=0 python pretrain.py \
    --logdir ./logs/stl10/simsiam/aug_self \
    --framework simsiam \
    --dataset stl10 \
    --datadir DATADIR \
    --model resnet18 \
    --batch-size 256 \
    --max-epochs 200 \
    --ss-color 1.0 --ss-crop 1.0

ImageNet100

python pretrain.py \
    --logdir ./logs/imagenet100/simsiam/aug_self \
    --framework simsiam \
    --dataset imagenet100 \
    --datadir DATADIR \
    --batch-size 256 \
    --max-epochs 500 \
    --model resnet50 \
    --base-lr 0.05 --wd 1e-4 \
    --ckpt-freq 50 --eval-freq 50 \
    --ss-crop 0.5 --ss-color 0.5 \
    --num-workers 16 --distributed

Evaluation

Our main evaluation setups are linear evaluation on fine-grained classification datasets (Table 1) and few-shot benchmarks (Table 2).

linear evaluation

CUDA_VISIBLE_DEVICES=0 python transfer_linear_eval.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cifar10 \
    --datadir DATADIR \
    --metric top1

few-shot

CUDA_VISIBLE_DEVICES=0 python transfer_few_shot.py \
    --pretrain-data imagenet100 \
    --ckpt CKPT \
    --model resnet50 \
    --dataset cub200 \
    --datadir DATADIR

Hierarchical Metadata-Aware Document Categorization under Weak Supervision (WSDM'21)

Hierarchical Metadata-Aware Document Categorization under Weak Supervision This project provides a weakly supervised framework for hierarchical metada

53 Sep 17, 2022

[EMNLP 2021] MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

MuVER This repo contains the code and pre-trained model for our EMNLP 2021 paper: MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity

24 May 30, 2022

Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Relation Prediction as an Auxiliary Training Objective for Knowledge Base Completion This repo provides the code for the paper Relation Prediction as

85 Jan 2, 2023

Improving Query Representations for DenseRetrieval with Pseudo Relevance Feedback:A Reproducibility Study.

APR The repo for the paper Improving Query Representations for DenseRetrieval with Pseudo Relevance Feedback:A Reproducibility Study. Environment setu

8 Nov 26, 2022

Self-training with Weak Supervision (NAACL 2021)

This repo holds the code for our weak supervision framework, ASTRA, described in our NAACL 2021 paper: "Self-Training with Weak Supervision"

148 Nov 20, 2022

Code release for SLIP Self-supervision meets Language-Image Pre-training

SLIP: Self-supervision meets Language-Image Pre-training What you can find in this repo: Pre-trained models (with ViT-Small, Base, Large) and code to

621 Dec 31, 2022

A library built upon PyTorch for building embeddings on discrete event sequences using self-supervision

pytorch-lifestream a library built upon PyTorch for building embeddings on discrete event sequences using self-supervision. It can process terabyte-si

103 Dec 17, 2022

[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision Kehong Gong*, Bingbing Li*, Jianfeng Zhang*, Ta

256 Dec 28, 2022

Codes for our IJCAI21 paper: Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization

DDAMS This is the pytorch code for our IJCAI 2021 paper Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization [Arxiv Pr

55 Dec 27, 2022

Comments

Linear evaluation of MoCo on CUB
Hi! Thank you for a great paper and for sharing the code!

I'm looking to reproduce your results on the MoCo model, especially for transferring it to the CUB dataset.

The command I'm running is:

CUDA_VISIBLE_DEVICES=0 python transfer_linear_eval.py \ --pretrain-data imagenet100 \ --ckpt $CHECKPOINT_PATH \ --model resnet50 \ --dataset cub200 \ --datadir $CUB_DIR \ --metric top1

However, the model achieves accuracy lower than the one reported in the paper (37.0, as reported in Tab. 3):

For MoCo baseline (checkpoint shared by you) I got test acc=0.2575 and for MoCo_augself (checkpoint shared by you) I got test acc=0.3224 For MoCo pretrained by myself, I got test acc=0.3309

Thus, I think the issue may be in linear evaluation.

Oddly enough, I roughly reproduced your results on the CIFAR-10 and CIFAR-100 dataset (with ~0.5% difference), so maybe the issue is with CUB only.

Could you kindly provide some guidance on whether I got the hyperparameters right? Alternatively, how did you set up the CUB files - did you use the default train / test / split?

Best regards!
opened by mprzewie 0
about AugSelf

hello,excuse me,different data augment methods have different forms of parameters, such as color perturbation and random cropping, like this different type of augment, how can Augself learn the difference between them?

opened by ZhaoYuTJPU 2

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Related tags

Overview

Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Dependencies

Checkpoints

Pretraining

STL-10

ImageNet100

Evaluation

linear evaluation

few-shot

You might also like...

Hierarchical Metadata-Aware Document Categorization under Weak Supervision (WSDM'21)

[EMNLP 2021] MuVER: Improving First-Stage Entity Retrieval with Multi-View Entity Representations

Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Improving Query Representations for DenseRetrieval with Pseudo Relevance Feedback:A Reproducibility Study.

Self-training with Weak Supervision (NAACL 2021)

Code release for SLIP Self-supervision meets Language-Image Pre-training

A library built upon PyTorch for building embeddings on discrete event sequences using self-supervision

[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)

Codes for our IJCAI21 paper: Dialogue Discourse-Aware Graph Model and Data Augmentation for Meeting Summarization

Comments

Linear evaluation of MoCo on CUB

about AugSelf

Owner

hankook

Learning trajectory representations using self-supervision and programmatic supervision.

Official repository for "On Improving Adversarial Transferability of Vision Transformers" (2021)

This repo includes our code for evaluating and improving transferability in domain generalization (NeurIPS 2021)

Mixup for Supervision, Semi- and Self-Supervision Learning Toolbox and Benchmark

Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Pytorch codes for "Self-supervised Multi-view Stereo via Effective Co-Segmentation and Data-Augmentation"

The source code for the Cutoff data augmentation approach proposed in this paper: "A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation".

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.