ImageNet-CoG is a benchmark for concept generalization. It provides a full evaluation framework for pre-trained visual representations which measure how well they generalize to unseen concepts.

Overview

The ImageNet-CoG Benchmark

Project Website Paper (arXiv)

Code repository for the ImageNet-CoG Benchmark introduced in the paper "Concept Generalization in Visual Representation Learning" (ICCV 2021). It contains code for reproducing all the experiments reported in the paper, as well as instructions on how to evaluate any custom model on the ImageNet-CoG Benchmark.

@InProceedings{sariyildiz2021conceptgeneralization,
    title={Concept Generalization in Visual Representation Learning},
    author={Sariyildiz, Mert Bulent and Kalantidis, Yannis and Larlus, Diane and Alahari, Karteek},
    booktitle={International Conference on Computer Vision},
    year={2021}
}

Contents of the Readme file:

Prerequisites: Benchmark data and code

Installation

We developed the benchmark using:

  • GCC 7.5.0
  • Python 3.7.9
  • PyTorch 1.7.1
  • Torchvision 0.8.2
  • CUDA 10.2.89
  • Optuna 2.3.0
  • YACS 0.1.8
  • termcolor 1.1.0

We recommend creating a separate conda environment for the benchmark:

conda create -n cog python=3.7.9
conda activate cog
conda install -c pytorch pytorch==1.7.1 torchvision==0.8.2 cudatoolkit=10.2
conda install -c conda-forge optuna=2.3
conda install termcolor
pip install yacs

Note: To reproduce the results for SimCLR-v2 and BYOL, you will also need TensorFlow, as well as the repos that contain code for converting the SimCLR-v2 and BYOL pre-trained models into PyTorch. More info can be found in the corresponding scripts under prepare_models/.

Data

To evaluate a model on the ImageNet-CoG benchmark you will need the following data:

The full ImageNet dataset (IN-21K)

The levels of ImageNet-CoG consist of a selection of 5K synsets in the full ImageNet. To download the full ImageNet, you need to create an account on the ImageNet website. Then, under the "Download" section, you will find the "Winter 2021" release (a tar file of size 1.1TB). Once you download it, extract it to folder <imagenet_images_root> (you will need this path later when extracting features). After extracting you should see the following folder structure, i.e., a separate folder of images per synset:

<imagenet_images_root>
...
|--- n07696728
|   |--- *.JPEG
|--- n11944954
|   |--- *.JPEG
...

The ILSVRC-2012 dataset (IN-1K)

To evaluate models on the pretraining dataset, you will need the ubiquitous ILSVRC-2012 subset of ImageNet, which we also refer to as IN-1K. It is also available on the ImageNet website. You will download two tar files for training (138GB) and validation (6.3GB) images, and extract them under <in1k_images_root> (again, you will need this path later). We expect the following folder structure for IN-1K:

<in1k_images_root>
...
|--- train
|   |--- n11939491
|       |--- *.JPEG
|   |--- n07836838
|       |--- *.JPEG
|--- val
|   |--- n11939491
|       |--- *.JPEG
|   |--- n07836838
|       |--- *.JPEG
...

The CoG level files

These are files that contain the concepts and data splits for ImageNet-CoG, and can be directly downloaded from the links below:

(Note: If clicking on the file names does not open a pop-up window for download, try 1) entering the file URLs directly on the address bar of your browser, or 2) using wget by giving the file URLs as arguments.)

Evaluating a model on the ImageNet-CoG benchmark

After installing the required packages and downloading the ImageNet data, you can follow the steps below:

  1. CoG step-1: Prepare the model you want to evaluate.
  2. CoG step-2: Extract image features for IN-1K and the CoG levels using the frozen backbone of the model.
  3. CoG step-3: Train linear classifiers on the pre-extracted features and measure accuracy.

CoG step-1: Model preparation

The ImageNet-CoG benchmark is designed to evaluate models that are pre-trained on IN-1K (the ILSVRC-2012 dataset).

Models evaluated in the paper

To reproduce the results for any of the models evaluated in our paper you can follow the instructions for downloading the their checkpoints. You will end up with all the checkpoints under the folder <models_root_dir>. Note that every model has a unique name that should be passed with the --model argument in all the scripts below. You can find the model names we used for all the models evaluated in our paper in this table.

Custom models

Follow these three steps to prepare your model for evaluation on the CoG benchmark.

  1. Give a name to your model: Every model has a unique name that is passed with the --model=<model_name> argument in all the scripts below. The model name consists of two parts <model_name>="<model_title>_<architecture_name>". You need to give such a name to your model, e.g., <myModel_myArchitecture>.

  2. Write a model loader function: To be able to extract features from your custom model, the load_pretrained_backbone() function must be able to load your pretrained model correctly. To do so, you will need to add a custom model loader function in model_loaders.py, that takes as input an arguments dict init_args and returns the backbone module and its forward function, e.g., backbone, forward = load_my_model(init_args)

  3. Register your model: Add a new element in MODEL_LOADER_DICT in the model_loader_dict.py script specifying the name and loader function for your model, e.g.:

[...]
"myModel_myArchitecture": load_my_model,
[...]

CoG step-2: Feature extraction

In this step, you will extract image features for the 6 datasets (IN-1K and our CoG levels L1, L2, L3, L4, L5). We provide for you a bash script to automatize feature extraction ./bash-scripts/extract_features.sh for a given model. As you notice, there are variables in this script that you need to set properly; Moreover, this script calls extract_features.py with the provided arguments for each of 6 datasets. In our GPU cluster, each feature extraction experiment takes ~60min using a V100 GPU and 8 CPU cores.

Please see below for the documentation of the arguments and several examples for extract_features.py.

(Click to see the details) Documentation of the arguments for extract_features.py:
  • Model arguments:

    • --model=<model_name>: The name of the model using which you would like to extract features. It can be one of the models we report in the paper (see the table with model names for those) or your custom model. This argument will point the code to the proper checkpoint loader function for your model.
    • For loading checkpoint files you have two options (it is enough to set one of them):
      • --models_root_dir=<models_root_dir>: If set, the script will look for a model checkpoint at path <models_root_dir>/<model_title>/<architecture_name>/model.ckpt,
      • --ckpt_file=<ckpt_file>: The full path of any valid checkpoint of the model <model_name>.
  • Dataset and split arguments: You can specify the dataset and split by using the following arguments:

    • --dataset=<dataset>: The dataset to extract features from ("in1k", "cog_l1", "cog_l2", "cog_l3", "cog_l4", "cog_l5")
    • --split=<split>: The split ("train" or "test"). Note that for the IN-1K dataset, "test" will extract features from the official validation set.
  • Arguments for the dataset (IN-1K and IN-21K) folders, and the CoG level files:

    • --in1k_images_root=<in1k_images_root>: The full path to the folder you downloaded IN-1K. This argument is required only when extracting features from IN-1K.
    • For the CoG levels (required only when extracting features from any of the CoG levels):
      • --imagenet_images_root=<imagenet_images_root>: The full path to the folder with IN-21K.
      • --cog_levels_mapping_file=cog_levels_mapping_file.pkl: The full path to the CoG concept-to-level mapping file that you can download from the section above.
      • --cog_concepts_split_file=cog_concepts_split_file.pkl: The full path to the CoG concepts split file that you can download from the section above.
  • Arguments for an output folder for features: For specifying where to save features you have two options (it is enough to set one of them):

    • --models_root_dir=<models_root_dir>: If set, features will be saved to the path <models_root_dir>/<model_title>/<architecture_name>/<dataset>/features_<split>/X_Y.pth.
    • --output_dir=<output_dir>: The full path to the folder where features will be stored (i.e., <output_dir>/X_Y.pth)
(Click to see the details) Examples for running extract_features.py:

To extract features for the training set of IN-1K from the "ResNet-50" model (which is the supervised baseline we use in the paper), run this command:

python extract_features.py \
    --model="sup_resnet50" \
    --dataset="in1k" \
    --split="train" \
    --models_root_dir=<models_root_dir> \
    --in1k_images_root=<in1k_images_root>

Note that --models_root_dir is set. Therefore, this command will load the pretrained weights of "ResNet50" from <models_root_dir>/sup/resnet50/model.ckpt and save the features into the file <models_root_dir>/sup/resnet50/in1k/features_train/X_Y.pth.

To extract the features for the test set of L5 from the "SwAV" model, run this command:

python extract_features.py \
    --model="swav_resnet50" \
    --dataset="cog_l5" \
    --split="test" \
    --models_root_dir=<models_root_dir> \
    --imagenet_images_root=<imagenet_images_root> \
    --cog_levels_mapping_file=cog_levels_mapping_file.pkl \
    --cog_concepts_split_file=cog_concepts_split_file.pkl

This script will load the pretrained weights of "SwAV" from <models_root_dir>/swav/resnet50/model.ckpt and save the features into the file <models_root_dir>/swav/resnet50/cog_l5/features_test/X_Y.pth

Also don't forget to check the bash script ./bash-scripts/extract_features.sh.

CoG step-3: Learning linear classifiers and testing

After extracting features for the 6 datasets (for their both training and test sets), we train linear classifiers on them. We divide the classification experiments into two settings:

  1. Learning with all available images: Training classifiers with all available training data for the concepts. (This setting is to reproduce the scores we report in Section 4.2.1 - Generalization to unseen concepts of our paper.) We train 5 classifiers with different seeds on each CoG level and IN-1K separately, then report their average score. So, for this setting, you will need to train 30 (6 datasets, 5 seeds) logistic regression classifiers using all available training data for the concepts. We provide for you a bash script ./bash-scripts/logreg.sh to automatize running 5 classifiers on a specific dataset (or on a specific training and test set features). There are variables in this script that you need to set properly. Then, this script calls logreg.py with the provided arguments for 5 seeds. Finally, it prints the average score of these 5 experiments. In our GPU cluster, each experiment takes ~75min using a V100 GPU and 8 CPU cores.

  2. Few-shot: Training classifiers with N in {1, 2, 4, 8, 16, 32, 64 or 128} training samples per concept. (This setting is to reproduce the scores we report in Section 4.2.2 - How fast can models adapt to unseen concepts? of our paper.) We again train 5 classifiers with different seeds on each dataset separately, then report their average score. But this time, we also change the number of training samples per concept. So, for this setting, you will need to train 240 (8 x 6 x 5) logistic regression classifiers. We provide for you a bash script ./bash-scripts/fewshot.sh to automatize running 5 classifiers on a specific dataset for N in {1, 2, 4, 8, 16, 32, 64 or 128}. There are variables in this script that you need to set properly. Then this script calls logreg.py with the provided arguments for 5 seeds and each N value. Finally, it prints the average score for each N.

Note on hyper-parameter tuning. To minimize performance differences due to sub-optimal hyper-parameters, we use the Optuna hyperparameter optimization framework to tune the learning rate and weight decay hyper-parameters when training a classifier. We sample 30 learning rate and weight decay pairs and perform hyper-parameter tuning by partitioning the training set into two temporary subsets used for hyper-parameter training and validation. Once the optimal hyper-parameters are found, then we train the final classifier with these hyper-parameters on the complete training set and report the accuracy on the test set. We combine hyper-parameter tuning, classifier training and testing in a single script; one just needs to run the logreg.py script.

Please see below for the documentation of the arguments and several examples for logreg.py.

(Click to see the details) Documentation of the arguments for logreg.py:
  • Arguments for training and test set features: To train a classifier, you need to load training and test set features extracted for a particular dataset (i.e., "X_Y.pth" files you obtained in the feature extraction step above). You have two options for specifying from where to load pre-extracted features (it is enough to set the arguments for one option):

    • Option-1: Loading pre-extracted features for a particular dataset extracted from a particular model located under <models_root_dir>. By setting the three arguments below, the script will look for features files <models_root_dir>/<model_name>/<architecture_name>/<dataset>/features_{train and test}/X_Y.pth. Please refer to the feature extraction part above to see how to set these arguments correctly.
      • --model=<model_name>: The name of the model.
      • --dataset=<dataset>: The name of the dataset.
      • --models_root_dir=<models_root_dir>: The full path to the models directory.
    • Option-2: Loading pre-extracted features from arbitrary paths. With these two arguments set, the script will look for training and test set features files <train_set_features_dir>/X_Y.pth and <test_set_features_dir>/X_Y.pth, respectively.
      • --train_features_dir=<train_features_dir>: The full path to the folder containing training set features.
      • --test_features_dir=<test_features_dir>: The full path to the folder containing test set features.
  • Arguments for an output folder for logs: logreg.py produces output logs that can be read by the print_results.py script to print the average of the classification results obtained with multiple seeds. For specifying where to save logs you have two options (it is enough to set one of them):

    • --models_root_dir=<models_root_dir>: If set, the output logs will be saved under the folder <models_root_dir>/<model_name>/<architecture_name>/<dataset>/eval_logreg/seed-<seed>/.
    • --output_dir=<output_dir>: The full path to the output folder where logs will be saved.
  • Seed for reproducibility:

    • EVAL.SEED <seed_number>: Seed for Python's random module, NumPy and PyTorch. Its default value is 22. Note that this is not an argument but a config entry. So you need to append EVAL.SEED <seed_number> to the arguments list to overwrite the default config value (see the example below). We run all classification experiments 5 times, and report the mean and variance of these runs and strongly recommend you to do the same.
  • Number of training sample per concept (for few-shot learning):

    • CLF.N_SHOT <img_per_class>: The number of training samples per concept to use for training a few-shot classifier. Note that this is not an argument but a config entry. So you need to append CLF.N_SHOT <img_per_class> to the arguments list to overwrite the default config value (see the example below).

Averaging results over multiple seeds. The logreg.py script stores top-1 and top-5 accuracies per epoch in a Python list and saves them into a pickle file named final-*/logs.pkl under the output logs folder. For getting the final mean and variance over multiple runs, i.e., after running the logreg.py script with multiple seeds, one can run the print_results.py script by providing the output logs directory with the --logreg_root_dir argument.

(Click to see the details) Examples for running logreg.py:

To train a linear classifier (including its hyper-parameter tuning phase) on top of L5 features extracted by SwAV, run:

python logreg.py \
    --model="swav_resnet50" \
    --dataset="cog_l5" \
    --models_root_dir=<models_root_dir> \
    EVAL.SEED 55  # --> training the classifier with seed 55

This script will load pre-extracted features from <models_root_dir>/swav/resnet50/cog_l5/features_{train and test}/X_Y.pth and save output logs under <models_root_dir>/swav/resnet50/cog_l5/eval_logreg/seed-55. It will also print the final top1/top5 accuracies for this run. To print the averaged scores over 5 seeds for SwAV for L5, and assuming one used the default output paths, the command is:

python print_results.py \
    --logreg_root_dir="<models_root_dir>/swav/resnet50/cog_l5/eval_logreg"

To simulate few-shot learning scenarios, you can overwrite the config entry:

python logreg.py \
    --model="swav_resnet50" \
    --dataset="cog_l5" \
    --models_root_dir=<models_root_dir> \
    CLF.N_SHOT 8 # --> training the classifier with 8 random training samples per concept

This script will use 8 random training samples per concept to train linear classifiers and save output logs under <models_root_dir>/swav/resnet50/cog_l5/eval_logreg_N8/seed-22.

Also don't forget to check the bash scripts ./bash-scripts/logreg.sh and ./bash-scripts/fewshot.sh.

You might also like...
 PyTorch evaluation code for Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.
PyTorch evaluation code for Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.

Out-of-distribution Generalization Investigation on Vision Transformers This repository contains PyTorch evaluation code for Delving Deep into the Gen

The first machine learning framework that encourages learning ML concepts instead of memorizing class functions.
The first machine learning framework that encourages learning ML concepts instead of memorizing class functions.

SeaLion is designed to teach today's aspiring ml-engineers the popular machine learning concepts of today in a way that gives both intuition and ways of application. We do this through concise algorithms that do the job in the least jargon possible and examples to guide you through every step of the way.

Code of PVTv2 is released! PVTv2 largely improves PVTv1 and works better than Swin Transformer with ImageNet-1K pre-training.
Code of PVTv2 is released! PVTv2 largely improves PVTv1 and works better than Swin Transformer with ImageNet-1K pre-training.

Updates (2020/06/21) Code of PVTv2 is released! PVTv2 largely improves PVTv1 and works better than Swin Transformer with ImageNet-1K pre-training. Pyr

Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation
Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation

Unseen Object Clustering: Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentation Introduction In this work, we propose a new method

The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

DS3L This is the code for paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020. Setups The code is implem

[MedIA2021]MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning
[MedIA2021]MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning

MIDeepSeg: Minimally Interactive Segmentation of Unseen Objects from Medical Images Using Deep Learning [MedIA or Arxiv] and [Demo] This repository pr

Codes of paper
Codes of paper "Unseen Object Amodal Instance Segmentation via Hierarchical Occlusion Modeling"

Unseen Object Amodal Instance Segmentation (UOAIS) Seunghyeok Back, Joosoon Lee, Taewon Kim, Sangjun Noh, Raeyoung Kang, Seongho Bak, Kyoobin Lee This

Code for Parameter Prediction for Unseen Deep Architectures (NeurIPS 2021)
Code for Parameter Prediction for Unseen Deep Architectures (NeurIPS 2021)

Parameter Prediction for Unseen Deep Architectures (NeurIPS 2021) authors: Boris Knyazev, Michal Drozdzal, Graham Taylor, Adriana Romero-Soriano Overv

Benchmark spaces - Benchmarks of how well different two dimensional spaces work for clustering algorithms

benchmark_spaces Benchmarks of how well different two dimensional spaces work fo

Code used to generate the results appearing in "Train longer, generalize better: closing the generalization gap in large batch training of neural networks"

Train longer, generalize better - Big batch training This is a code repository used to generate the results appearing in "Train longer, generalize bet

Elad Hoffer 145 Sep 16, 2022
The Hailo Model Zoo includes pre-trained models and a full building and evaluation environment

Hailo Model Zoo The Hailo Model Zoo provides pre-trained models for high-performance deep learning applications. Using the Hailo Model Zoo you can mea

Hailo 50 Dec 7, 2022
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 11 Jul 28, 2022
pybaum provides tools to work with pytrees which is a concept burrowed from JAX.

pybaum provides tools to work with pytrees which is a concept burrowed from JAX.

Open Source Economics 9 May 11, 2022
[CVPR 2021] A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts

Visual-Reasoning-eXplanation [CVPR 2021 A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts] Project Page | Vid

Andy_Ge 54 Dec 21, 2022
A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

Justin 1.1k Dec 24, 2022
《Train in Germany, Test in The USA: Making 3D Object Detectors Generalize》(CVPR 2020)

Train in Germany, Test in The USA: Making 3D Object Detectors Generalize This paper has been accpeted by Conference on Computer Vision and Pattern Rec

Xiangyu Chen 101 Jan 2, 2023
TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

TorchGeo is a PyTorch domain library, similar to torchvision, that provides datasets, transforms, samplers, and pre-trained models specific to geospatial data.

Microsoft 1.3k Dec 30, 2022
Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts

t5-japanese Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts. The following is a list of models that

Kimio Kuramitsu 1 Dec 13, 2021