https://arxiv.org/abs/2102.11005

THUML: Machine Learning Group @ THSS

Last update: Dec 19, 2022

Related tags

Deep Learning LogME

Overview

LogME

LogME: Practical Assessment of Pre-trained Models for Transfer Learning

How to use

Just feed the features f and labels y to the function, and you can get a nice score which well correlates with the transfer learning performance.

from LogME import LogME
score = LogME(f, y)

Then you can use the score to quickly select a good pre-trained model. The larger the score is, the better transfer performance you get.

Experimental results

We extensively validate the generality and superior performance of LogME on 14 pre-trained models and 17 downstream tasks, covering various pre-trained models (supervised pre-trained and unsupervised pre-trained), downstream tasks (classification and regression), and modalities (vision and language). Check the paper for all the results.

Computer vision

9 datasets and 10 pre-trained models. LogME is a reasonably good indicator for transfer performance.

NLP

7 tasks and 4 pre-trained models. LogME is a good indicator for transfer performance.

Speedup

LogME provides a dramatic speedup for assessing pre-trained models. The speedup comes from two aspects:

LogME does not need hyper-parameter tuning whereas vanilla fine-tuning requires extensive hyper-parameter tuning.
We designed a fast algorithm to further speedup the computation of LogME.

Citation

If you find it useful, please cite the following paper:

@article{you_logme:_2021,
	title = {LogME: Practical Assessment of Pre-trained Models for Transfer Learning},
	author = {You, Kaichao and Liu, Yong and Long, Mingsheng and Wang, Jianmin},
	journal = {arxiv},
	volume = {abs/2102.11005},
	year = {2021},
	url = {https://arxiv.org/abs/2102.11005},
}

Contact

If you have any question or want to use the code, please contact [email protected] .

Comments

A question about the meaning of the LogME score

Hello, thanks for your this work. I have a question about the meaning of the score after I read the paper.

For example, I have a pre-trained model M, and a labeled target dataset {x,y_truth}. And I pass x to the model M to get the features X and y_pred.

I noticed that the logme(X,y_pred) > logme(X,y_truth).

I am confused that if the meaning of LogME score is the benefits of fine tuning the model using the target dataset, the logme(X,y_truth) should be greater than logme(X,y_pred) because it's ground_truth. Because I think the result using a "True" dataset to fine-tune must be better than using a "Noisy" dataset.

opened by zxC0der 7
Sec. 5.1中的实验，ResNet系列用的哪一层的特征？

您好，我现在使用torchvision内置的resent在FC之前的2048维的特征，在CIFAR10和CIFAR100上求出的logME score和论文中有较大区别，并且线性关系也不强，所以向请问一下论文中的数据是使用的哪一层的feature？以及后续有开源论文中数据集的LogME评测的code吗，这样使用起来比较方便？

opened by Minwellcym 4
Question about comparing different models

Hi, thanks for your nice work. I have a question regarding comparing transferability of different models: since different models produce features of different shapes in the penultimate layer, is it advised that one perform some form of dimensionality reduction on the features to get a consistent feature dimension (e.g. 64) on all the models before performing transferability estimate? What is your thoughts on this?

opened by MoeinSorkhei 3
If LogME is conducted on downstream training split or testing split?

Hi @youkaichao, Thanks for the nice work and code, I would like to learn if the model transferability is measured on the downstream training split or testing split?

opened by Kyfafyd 3
A question about reproducing the NCE and LEEP results
Hi, thanks for your excellent work and friendly code style again. I encountered some difficulties in reproducing the NCE / LEEP results. Sorry, I am confused again.

I guess the calculation of NCE and LEEP scores in the paper may be wrong. Actually, I used the opposite of pseudo_source_label and target_label to obtain the results of the paper, as shown in the code below:

score = NCE(source_label=targets.numpy(), target_label=torch.argmax(predictions, dim=1).numpy()) score = LEEP(pseudo_source_label=np.squeeze(np.eye((torch.max(targets) + 1).item())[targets.numpy().reshape(-1)]), target_label=torch.argmax(predictions, dim=1).numpy())

where the predictions variable is calculated from output, i.e.,

features, outputs, targets = forward_pass(self.data_loader_train, model, fc_layer) predictions = F.softmax(outputs, dim=1)

The above wrong code successfully reproduces the results of NCE and LEEP in the paper, i.e., Section C. Original Results in Figures - Table 5. Original results in Figure 4.

From the NCE and LEEP code, the correct way to pass the pseudo_source_label and target_label would be:

score = NCE(source_label=torch.argmax(predictions, dim=1).numpy(), target_label=targets.numpy()) score = LEEP(pseudo_source_label=predictions.numpy(), target_label=targets.numpy())

But this does not yield the results in the paper. This will impact the calculation of the final weighted tau result.

Thanks again for your great work and open source spirit. I'm really sorry to come back to you for advice. Thank you very much.
opened by ZhangYikaii 3
How to apply LogME on the multi-label classification tasks?
Hi, thanks your excellent work and nice code style.

I notice that LogME can be applied on multi-label dataset as the paper says. But the y is not the one-hot format, I want to know how to apply it on multi-label classification tasks.

Because the label is not one-hot format, so one possible solution is to construct every feature-label pairs for each sample. Such as:

The original label is [0, 1, 1, 0, 1] as for the feature f

f: [0,1,1,0,1]

Now i need to constrcut three feature-label pairs:

f : 1 f : 2 f : 4

So I want to know if it is the right way to use it on the multi-label setting. If this is not correct, can you give me an example.

Looking forward to your reply. Thanks!

Best Regards! Yun
opened by Muyun99 3
运行结果与论文部分不一致

你好！我认为这是一个很棒的工作，但请问下为啥LogME计算结果和论文中不一致（比如inception_v3结果是0.9397而论文中的是0.953差距那么大）。若把inceptionV3这一项不看，算加权kendall-tau只有0.2（加权kendall-tau给了inceptionV3很高的权重，因为acc是排第一）。我后面又试验了CIFAR-10数据集，结论类似，除了inceptionV3差别很大，其余几乎一致。请问下是哪里出了问题，谢谢！

opened by zhanglei1172 3
can I use LogME to guide my model finetuning

Does the LogME score of each epoch checkpoint in the fine-tuning process of downstream tasks have a special meaning? In other words, I mean, can I use LogME to guide my model to be optimized in the right direction?

Thanks.

opened by zxC0der 2
A question about reproducing the results

Hi, thanks for your excellent work and friendly code style. I encountered some difficulties in reproducing the results.

I directly passed the path where the downloaded dataset was located, but the result was quite different.

Next, I use the dataloader implemented here for the FGVCAircraft dataset. Which class type of this dataset should I use? Different parts of this dataset have a significant impact on the results. So I can not reproduce the results in the paper.

Thanks again. I look forward to hearing from you!

opened by ZhangYikaii 2
Would you please provide detailed code for finetuning different datasets?

Hello, I am very interested in your work. But I find that the code only contains the aircraft dataset, would you please provide detailed code for finetuning different datasets (Figure 4)?

opened by yaolu-zjut 1
How to reproduce the fine-tuning accuracies?

Hi, I have seen details about the learning rate and weight decay in the paper. Still, I couldn't find more information about how you split train/validation, optimizer, schedulers, training epochs, etc. Could you provide the exact methodology to reproduce the fine-tuning results?

Thank you.

opened by VirtualSpaceman 0
May I ask which transfer accuracy used in LogME paper?

Hi,

In LEEP paper, there is two transfer accuracy depending on which transfer learning method is used, re-train head or fine-tune (whole model). Is LogME correlated well with the transfer accuracy of fine-tuning whole model?

If we have a large target-domain dataset and finetune for long enough epoch, the knowledge in pretrained model will be forgot. All pretrained models will have similar accuracy. So we should expect the logME for different models also be similar. Will this similar logME scores happen?

Thank you!

Best, Luzai

opened by luzai 2
Why the LogME is implemented not in torch but in numpy?

RT. Maybe with CUDA, LogME costs less time? Or if this is the problem setting assuming the transferability evaluation can not access GPU? (but the feature extraction part needs GPU?)

opened by Kyfafyd 3

https://arxiv.org/abs/2102.11005

Related tags

Overview

LogME

How to use

Experimental results

Computer vision

NLP

Speedup

Citation

Contact

Comments

Owner

THUML: Machine Learning Group @ THSS

This repository contains the code used for Predicting Patient Outcomes with Graph Representation Learning (https://arxiv.org/abs/2101.03940).

Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Code for the paper: Learning Adversarially Robust Representations via Worst-Case Mutual Information Maximization (https://arxiv.org/abs/2002.11798)

ISTR: End-to-End Instance Segmentation with Transformers (https://arxiv.org/abs/2105.00637)

Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Minimal implementation of PAWS (https://arxiv.org/abs/2104.13963) in TensorFlow.

YOLO5Face: Why Reinventing a Face Detector (https://arxiv.org/abs/2105.12931)

A PyTorch implementation of EventProp [https://arxiv.org/abs/2009.08378], a method to train Spiking Neural Networks

Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

source code for https://arxiv.org/abs/2005.11248 "Accelerating Antimicrobial Discovery with Controllable Deep Generative Models and Molecular Dynamics"

Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).