https://arxiv.org/abs/2102.11005

Related tags

Deep Learning LogME
Overview

LogME

LogME: Practical Assessment of Pre-trained Models for Transfer Learning

How to use

Just feed the features f and labels y to the function, and you can get a nice score which well correlates with the transfer learning performance.

from LogME import LogME
score = LogME(f, y)

Then you can use the score to quickly select a good pre-trained model. The larger the score is, the better transfer performance you get.

Experimental results

We extensively validate the generality and superior performance of LogME on 14 pre-trained models and 17 downstream tasks, covering various pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Check the paper for all the results.

Computer vision

9 datasets and 10 pre-trained models. LogME is a reasonably good indicator for transfer performance.

image-20210222204141915

NLP

7 tasks and 4 pre-trained models. LogME is a good indicator for transfer performance.

image-20210222204350389

Speedup

LogME provides a dramatic speedup for assessing pre-trained models. The speedup comes from two aspects:

  • LogME does not need hyper-parameter tuning whereas vanilla fine-tuning requires extensive hyper-parameter tuning.
  • We designed a fast algorithm to further speedup the computation of LogME.

image-20210222204712553

Citation

If you find it useful, please cite the following paper:

@article{you_logme:_2021,
	title = {LogME: Practical Assessment of Pre-trained Models for Transfer Learning},
	author = {You, Kaichao and Liu, Yong and Long, Mingsheng and Wang, Jianmin},
	journal = {arxiv},
	volume = {abs/2102.11005},
	year = {2021},
	url = {https://arxiv.org/abs/2102.11005},
}

Contact

If you have any question or want to use the code, please contact [email protected] .

Comments
  • A question about the meaning of the LogME score

    A question about the meaning of the LogME score

    Hello, thanks for your this work. I have a question about the meaning of the score after I read the paper.

    For example, I have a pre-trained model M, and a labeled target dataset {x,y_truth}. And I pass x to the model M to get the features X and y_pred.

    I noticed that the logme(X,y_pred) > logme(X,y_truth).

    I am confused that if the meaning of LogME score is the benefits of fine tuning the model using the target dataset, the logme(X,y_truth) should be greater than logme(X,y_pred) because it's ground_truth. Because I think the result using a "True" dataset to fine-tune must be better than using a "Noisy" dataset.

    opened by zxC0der 7
  • Sec. 5.1中的实验,ResNet系列用的哪一层的特征?

    Sec. 5.1中的实验,ResNet系列用的哪一层的特征?

    您好,我现在使用torchvision内置的resent在FC之前的2048维的特征,在CIFAR10和CIFAR100上求出的logME score和论文中有较大区别,并且线性关系也不强,所以向请问一下论文中的数据是使用的哪一层的feature?以及后续有开源论文中数据集的LogME评测的code吗,这样使用起来比较方便?

    opened by Minwellcym 4
  • Question about comparing different models

    Question about comparing different models

    Hi, thanks for your nice work. I have a question regarding comparing transferability of different models: since different models produce features of different shapes in the penultimate layer, is it advised that one perform some form of dimensionality reduction on the features to get a consistent feature dimension (e.g. 64) on all the models before performing transferability estimate? What is your thoughts on this?

    opened by MoeinSorkhei 3
  • If LogME is conducted on downstream training split or testing split?

    If LogME is conducted on downstream training split or testing split?

    Hi @youkaichao, Thanks for the nice work and code, I would like to learn if the model transferability is measured on the downstream training split or testing split?

    opened by Kyfafyd 3
  • A question about reproducing the NCE and LEEP results

    A question about reproducing the NCE and LEEP results

    Hi, thanks for your excellent work and friendly code style again. I encountered some difficulties in reproducing the NCE / LEEP results. Sorry, I am confused again.

    I guess the calculation of NCE and LEEP scores in the paper may be wrong. Actually, I used the opposite of pseudo_source_label and target_label to obtain the results of the paper, as shown in the code below:

    score = NCE(source_label=targets.numpy(), target_label=torch.argmax(predictions, dim=1).numpy())
    score = LEEP(pseudo_source_label=np.squeeze(np.eye((torch.max(targets) + 1).item())[targets.numpy().reshape(-1)]), target_label=torch.argmax(predictions, dim=1).numpy())
    

    where the predictions variable is calculated from output, i.e.,

    features, outputs, targets = forward_pass(self.data_loader_train, model, fc_layer)
    predictions = F.softmax(outputs, dim=1)
    

    The above wrong code successfully reproduces the results of NCE and LEEP in the paper, i.e., Section C. Original Results in Figures - Table 5. Original results in Figure 4.

    From the NCE and LEEP code, the correct way to pass the pseudo_source_label and target_label would be:

    score = NCE(source_label=torch.argmax(predictions, dim=1).numpy(), target_label=targets.numpy())
    score = LEEP(pseudo_source_label=predictions.numpy(), target_label=targets.numpy())
    

    But this does not yield the results in the paper. This will impact the calculation of the final weighted tau result.

    Thanks again for your great work and open source spirit. I'm really sorry to come back to you for advice. Thank you very much.

    opened by ZhangYikaii 3
  • How to apply LogME on the multi-label classification tasks?

    How to apply LogME on the multi-label classification tasks?

    Hi, thanks your excellent work and nice code style.

    I notice that LogME can be applied on multi-label dataset as the paper says. But the y is not the one-hot format, I want to know how to apply it on multi-label classification tasks.

    Because the label is not one-hot format, so one possible solution is to construct every feature-label pairs for each sample. Such as:

    The original label is [0, 1, 1, 0, 1] as for the feature f

    f: [0,1,1,0,1]
    

    Now i need to constrcut three feature-label pairs:

    f : 1
    f : 2
    f : 4
    

    So I want to know if it is the right way to use it on the multi-label setting. If this is not correct, can you give me an example.

    Looking forward to your reply. Thanks!

    Best Regards! Yun

    opened by Muyun99 3
  • 运行结果与论文部分不一致

    运行结果与论文部分不一致

    image image 你好!我认为这是一个很棒的工作,但请问下为啥LogME计算结果和论文中不一致(比如inception_v3结果是0.9397而论文中的是0.953差距那么大)。若把inceptionV3这一项不看,算加权kendall-tau只有0.2(加权kendall-tau给了inceptionV3很高的权重,因为acc是排第一)。我后面又试验了CIFAR-10数据集,结论类似,除了inceptionV3差别很大,其余几乎一致。请问下是哪里出了问题,谢谢!

    opened by zhanglei1172 3
  • can I use LogME to guide my model finetuning

    can I use LogME to guide my model finetuning

    Does the LogME score of each epoch checkpoint in the fine-tuning process of downstream tasks have a special meaning? In other words, I mean, can I use LogME to guide my model to be optimized in the right direction?

    Thanks.

    opened by zxC0der 2
  • A question about reproducing the results

    A question about reproducing the results

    Hi, thanks for your excellent work and friendly code style. I encountered some difficulties in reproducing the results.

    I directly passed the path where the downloaded dataset was located, but the result was quite different.

    Next, I use the dataloader implemented here for the FGVCAircraft dataset. Which class type of this dataset should I use? Different parts of this dataset have a significant impact on the results. So I can not reproduce the results in the paper.

    Thanks again. I look forward to hearing from you!

    image

    image

    opened by ZhangYikaii 2
  • Would you please provide detailed code for finetuning different datasets?

    Would you please provide detailed code for finetuning different datasets?

    Hello, I am very interested in your work. But I find that the code only contains the aircraft dataset, would you please provide detailed code for finetuning different datasets (Figure 4)?

    opened by yaolu-zjut 1
  • How to reproduce the fine-tuning accuracies?

    How to reproduce the fine-tuning accuracies?

    Hi, I have seen details about the learning rate and weight decay in the paper. Still, I couldn't find more information about how you split train/validation, optimizer, schedulers, training epochs, etc. Could you provide the exact methodology to reproduce the fine-tuning results?

    Thank you.

    opened by VirtualSpaceman 0
  • May I ask which transfer accuracy used in LogME paper?

    May I ask which transfer accuracy used in LogME paper?

    Hi,

    In LEEP paper, there is two transfer accuracy depending on which transfer learning method is used, re-train head or fine-tune (whole model). Is LogME correlated well with the transfer accuracy of fine-tuning whole model?

    If we have a large target-domain dataset and finetune for long enough epoch, the knowledge in pretrained model will be forgot. All pretrained models will have similar accuracy. So we should expect the logME for different models also be similar. Will this similar logME scores happen?

    Thank you!

    Best, Luzai

    opened by luzai 2
  • Why the LogME is implemented not in torch but in numpy?

    Why the LogME is implemented not in torch but in numpy?

    RT. Maybe with CUDA, LogME costs less time? Or if this is the problem setting assuming the transferability evaluation can not access GPU? (but the feature extraction part needs GPU?)

    opened by Kyfafyd 3
Owner
THUML: Machine Learning Group @ THSS
Machine Learning Group, School of Software, Tsinghua University
THUML: Machine Learning Group @ THSS
This repository contains the code used for Predicting Patient Outcomes with Graph Representation Learning (https://arxiv.org/abs/2101.03940).

Predicting Patient Outcomes with Graph Representation Learning This repository contains the code used for Predicting Patient Outcomes with Graph Repre

Emma Rocheteau 76 Dec 22, 2022
Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow 201 Dec 21, 2022
Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

[TCSVT] Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization LPN [Paper] NEWs Prerequisites Python 3.6 GPU Memory >= 8G Numpy > 1.

null 46 Dec 14, 2022
Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Meta-Solver for Neural Ordinary Differential Equations Towards robust neural ODEs using parametrized solvers. Main idea Each Runge-Kutta (RK) solver w

Julia Gusak 25 Aug 12, 2021
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 153 Dec 14, 2022
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

null 967 Jan 4, 2023
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

Representation Robustness Evaluations Our implementation is based on code from MadryLab's robustness package and Devon Hjelm's Deep InfoMax. For all t

Sicheng 19 Dec 7, 2022
ISTR: End-to-End Instance Segmentation with Transformers (https://arxiv.org/abs/2105.00637)

This is the project page for the paper: ISTR: End-to-End Instance Segmentation via Transformers, Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wa

Jie Hu 182 Dec 19, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

PAWS-TF ?? Implementation of Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples (PAWS)

Sayak Paul 43 Jan 8, 2023
YOLO5Face: Why Reinventing a Face Detector (https://arxiv.org/abs/2105.12931)

Introduction Yolov5-face is a real-time,high accuracy face detection. Performance Single Scale Inference on VGA resolution(max side is equal to 640 an

DeepCam Shenzhen 1.4k Jan 7, 2023
A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Spiking Neural Network training with EventProp This is an unofficial PyTorch implemenation of EventProp, a method to compute exact gradients for Spiki

Pedro Savarese 35 Jul 29, 2022
Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Swin-Transformer-Tensorflow A direct translation of the official PyTorch implementation of "Swin Transformer: Hierarchical Vision Transformer using Sh

null 52 Dec 29, 2022
Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

null 458 Jan 2, 2023
Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

alias-free-gan-pytorch Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) This implementation

Kim Seonghyeon 502 Jan 3, 2023
source code for https://arxiv.org/abs/2005.11248 "Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics"

Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics This work will be published in Nature Biomedical

International Business Machines 71 Nov 15, 2022
Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

Hurdles to Progress in Long-form Question Answering This repository contains the official scripts and datasets accompanying our NAACL 2021 paper, "Hur

Kalpesh Krishna 41 Nov 8, 2022