This project hosts the code for implementing the ISAL algorithm for object detection and image classification

Related tags

Deep Learning ISAL
Overview

Influence Selection for Active Learning (ISAL)

This project hosts the code for implementing the ISAL algorithm for object detection and image classification, as presented in our paper:

Influence Selection for Active Learning;
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, Conghui He;
In: Proc. Int. Conf. Computer Vision (ICCV), 2021.
arXiv preprint arXiv:2108.09331

The full paper is available at: https://arxiv.org/abs/2108.09331.

Implementation based on MMDetection is included in MMDetection.

Highlights

  • Task agnostic: We evaluate ISAL in both object detection and image classification. Compared with previous methods, ISAL decreases the annotation cost at least by 12%, 12%, 3%, 13% and 16% on CIFAR10, SVHN, CIFAR100, VOC2012 and COCO, respectively.

  • Model agnostic: We evaluate ISAL with different model in object detection. On COCO dataset, with one-stage anchor-free detector FCOS, ISAL decreases the annotation cost at least by 16%. With two-stage anchor-based detector Faster R-CNN, ISAL decreases the annotation cost at least by 10%.

ISAL just needs to use the model gradients, which can be easily obtained in a neural network no matter what task is and how complex the model structure is, our proposed ISAL is task-agnostic and model-agnostic.

Required hardware

We use 4 NVIDIA V100 GPUs for object detection. We use 1 NVIDIA TITAN Xp GPUs for image classification.

Installation

Our ISAL implementation for object detection is based on mmdetection v2.4.0 with mmcv v1.1.1. Their need Pytorch version = 1.5, CUDA version = 10.1, CUDNN version = 7. We provide a docker file (./detection/Dockerfile) to prepare the environment. Once the environment is prepared, please copy all the files under the folder ./detection into the directory /mmdetection in the docker.

Our ISAL implementation for image classification is based on pycls v0.1. It need Pytorch version = 1.6, CUDA version = 10.1, CUDNN version = 7.

Training

The following command line will perform the ISAL algorithm with FCOS detector on COCO dataset, the active learning algorithm will iterate 20 steps with 4 GPUS:

bash dist_run_isal.sh /workdir /datadir \
    /mmdetection/configs/mining_experiments/ \
    fcos/fcos_r50_caffe_fpn_1x_coco_influence_function.py \
    --mining-method=influence --seed=42 --deterministic \
    --noised-score-thresh=0.1

Note that:

  1. If you want to use fewer GPUs, please change GPUS in shell script. In addition, you may need to change the samples_per_gpu in the config file to mantain the total batch size is equal to 8.
  2. The models and all inference results will be saved into /workdir.
  3. The data should be place in /datadir.
  4. If you want to run our code on VOC or your own dataset, we suggest that you should change the data format into COCO format.
  5. If you want to change the active learning iteration steps, please change the TRAIN_STEP in shell script. If you want to change the image selected by step_0 or the following steps, please change the INIT_IMG_NUM or IMG_NUM in shell script, respectively.
  6. The shell script will delete all the trained models after all the active learning steps. If you want to maintain the models please change the DELETE_MODEL in shell script.

The following command line will perform the ISAL algorithm with ResNet-18 on CIFAR10 dataset, the active learning algorithm will iterate 10 steps with 1 GPU:

bash run_isal.sh /workdir /datadir \
    pycls/configs/archive/cifar/resnet/R-18_nds_1gpu_cifar10.yaml \
    --mining-method=influence --random-seed=0

Note that:

  1. The models and all inference results will be saved into /workdir.
  2. The data should be place in /datadir.
  3. If you want to train SHVN or your own dataset, we suggest that you should change the data format into CIFAR10 format.
  4. The STEP in shell script indicates that in each active learning step the algorithm will add (1/STEP)% of the whole dataset into labeled dataset. The TRAIN_STEP indicates the total steps of active learning algorithm.

Citations

Please consider citing our paper in your publications if the project helps your research. BibTeX reference is as follows.

@inproceedings{liu2021influence,
  title={Influence selection for active learning},
  author={Liu, Zhuoming and Ding, Hao and Zhong, Huaping and Li, Weijia and Dai, Jifeng and He, Conghui},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={9274--9283},
  year={2021}
}

Acknowledgments

We thank Zheng Zhu for implementing the classification pipeline. We thank Bin Wang and Xizhou Zhu for discussion and helping with the experiments. We thank Yuan Tian and Jiamin He for discussing the mathematic derivation.

License

For academic use only. For commercial use, please contact the authors.

Issues
  • Why ISAL picks images with less boxes

    Why ISAL picks images with less boxes

    As the experiment results showed, ISAL decreases the annotation cost by picking images with less boxes compared to the random baseline. However, the influence calculation formula has no direct relationship with the boxes number of each image. I didn't find any explanation in the paper about why it tends to pick those images with less boxes. In other words, if the objective is decreasing the annotation cost, why would you design the influence formula in this form? Thanks a lot.

    opened by Artificial-Inability 5
  • KeyError: 'Non-existent config key: MODEL.OUTPUT_MID_LAYER'

    KeyError: 'Non-existent config key: MODEL.OUTPUT_MID_LAYER'

    Hi, I am currently trying to reproduce your work based on the code uploaded on github. I wanted to do a quick training run on classification task based on the below command line. bash run_isal.sh /workdir /datadir
    pycls/configs/archive/cifar/resnet/R-18_nds_1gpu.yaml
    --mining-method=influence --random-seed=0 However, I kept getting the error mentioned above, saying there is non-existent config key. I tried googling but was out of luck. Hope you can advise me on how to appropriately reproduce your work, especially on classification task. Thank you for your inspiring work.

    opened by jokc28 3
  • grad is None when upgrading mmdet to a higher version (2.16.0)

    grad is None when upgrading mmdet to a higher version (2.16.0)

    Hey, dragon. Thank you for your work!

    I am trying to make ISAL compatible to a higher version of mmdet (e.g., 2.16.0). While other methods work fine, ISAL throws an error in src/test_calc_s_test.py, and the error log is

    [                                                  ] 0/1784, elapsed: 0s, ETA:Traceback (most recent call last):                                                                                                     
      File "src/test_calc_s_test.py", line 146, in <module>
    Traceback (most recent call last):
      File "src/test_calc_s_test.py", line 146, in <module>
        main()
      File "src/test_calc_s_test.py", line 131, in main
        loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
      File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
        main()
      File "src/test_calc_s_test.py", line 131, in main
        hv = hessian_vec_prod(all_loss, params, h_estimate)
      File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
        return_grads = [ele.detach() for ele in return_grads]
      File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
        loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
      File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
        return_grads = [ele.detach() for ele in return_grads]
    AttributeError: 'NoneType' object has no attribute 'detach'
    Traceback (most recent call last):
      File "src/test_calc_s_test.py", line 146, in <module>
        hv = hessian_vec_prod(all_loss, params, h_estimate)
      File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
        return_grads = [ele.detach() for ele in return_grads]
      File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
        return_grads = [ele.detach() for ele in return_grads]
        main()
      File "src/test_calc_s_test.py", line 131, in main
    AttributeError: 'NoneType' object has no attribute 'detach'
        loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
      File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
        hv = hessian_vec_prod(all_loss, params, h_estimate)
      File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
        return_grads = [ele.detach() for ele in return_grads]
      File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
        return_grads = [ele.detach() for ele in return_grads]
    AttributeError: 'NoneType' object has no attribute 'detach'
    Traceback (most recent call last):
      File "src/test_calc_s_test.py", line 146, in <module>
        main()
      File "src/test_calc_s_test.py", line 131, in main
        loss_val_mul_hessian_reverse = multi_gpu_s_test(val_sum_grad, model, params, train_data_loader, img_num=args.img_num)
      File "/home/username/Detection/isal/src/utils.py", line 146, in multi_gpu_s_test
        hv = hessian_vec_prod(all_loss, params, h_estimate)
      File "/home/username/Detection/isal/src/utils.py", line 203, in hessian_vec_prod
        return_grads = [ele.detach() for ele in return_grads]
      File "/home/username/Detection/isal/src/utils.py", line 203, in <listcomp>
        return_grads = [ele.detach() for ele in return_grads]
    AttributeError: 'NoneType' object has no attribute 'detach'
    

    It seems that the 2nd derivative of conv_cls is None. Could you help with it? Thanks in advance!

    opened by siiibyl 2
  • Stops after step 1 for classification

    Stops after step 1 for classification

    I've tried reproducing your work on classification task. Running the code went great, but it stopped working after finishing step 1. It doesn't seem to be my GPU's problem since I'm using 24G NVIDIA RTX A5000. There aren't any error messages showing up, not clearly telling me what the problem is. However, I did get a UserWarning message while running the code shown as below. I really want to successfully re-implement your work and hope you can give me some insight in how to deal with the situation.

    UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. pseudo_label = torch.nn.functional.softmax(preds).argmax().unsqueeze(dim=0)

    P.S. Would it be possible to print influence value of the selected unlabeled sample layer-wise? It would be interesting if I could see how the influence value changes after each layer, but it doesn't seem like it is coded in such a way.

    opened by jokc28 2
  • Calculation of s_test

    Calculation of s_test

    I had doubt in the calculation of s_test in line number 55 of utils.pyinside classification/src. I see it is different from the description given in the paper Algorithm 3. I see there is no damping factor or scaling factor in algorithm 3 whereas they are present in the code. Could you please tell me how to interpret these terms?

    opened by vidit98 1
  • CIFAR10 test_calc_val_grad.py ,grad_result = grad(loss, params),cuda out of memory

    CIFAR10 test_calc_val_grad.py ,grad_result = grad(loss, params),cuda out of memory

    I have a GPU NVIDIA RTX3060 12G,When the code runs to grad_result = grad(loss, params) in classification test_calc_val_grad.py, error: cuda out of memory.but the memory of NIVIDIA TITAN Xp in your paper is also 12G, I've been debugging code on windows and ubuntu 18.04 for a long time,but I can't sovle it.

    opened by LxL1996318 1
  • no such file plot_results.py

    no such file plot_results.py

    run_isal_cifar100.sh

    #plot all step results python $(dirname "$0")/src/tools/plot_results.py
    --work-dir=${WORK_DIR}/${TIMESTAMP} --train-step=${TRAIN_STEP}

    opened by LxL1996318 1
  • Implementation of coreset method

    Implementation of coreset method

    Thanks for your open source code, I have reproduced the experiments and get similar results using ISAL. However, I want to compare the results with coreset method but I didn't find any open source code about coreset on object detection. I have read the coreset paper and its experiments are based on classification task. The key problem is how to calculate the distance between two different images in object detection. Did you implement coreset on object detection by yourself? If so, would you please share the code or is there any open source link? I'll be really appreciate it.

    opened by jacksonxu 1
Real Time Object Detection and Classification using Yolo Algorithm.

Real time Object detection & Classification using YOLO algorithm. Real Time Object Detection and Classification using Yolo Algorithm. What is Object D

Ketan Chawla 1 Apr 17, 2022
Simple-Image-Classification - Simple Image Classification Code (PyTorch)

Simple-Image-Classification Simple Image Classification Code (PyTorch) Yechan Kim This repository contains: Python3 / Pytorch code for multi-class ima

Yechan Kim 6 Jun 2, 2022
Library for implementing reservoir computing models (echo state networks) for multivariate time series classification and clustering.

Framework overview This library allows to quickly implement different architectures based on Reservoir Computing (the family of approaches popularized

Filippo Bianchi 231 Aug 6, 2022
Image Classification - A research on image classification and auto insurance claim prediction, a systematic experiments on modeling techniques and approaches

A research on image classification and auto insurance claim prediction, a systematic experiments on modeling techniques and approaches

null 0 Jan 23, 2022
Final project code: Implementing MAE with downscaled encoders and datasets, for ESE546 FA21 at University of Pennsylvania

546 Final Project: Masked Autoencoder Haoran Tang, Qirui Wu 1. Training To train the network, please run mae_pretraining.py. Please modify folder path

Haoran Tang 0 Apr 22, 2022
Hl classification bc - A Network-Based High-Level Data Classification Algorithm Using Betweenness Centrality

A Network-Based High-Level Data Classification Algorithm Using Betweenness Centr

Esteban Vilca 2 Mar 13, 2022
Final project code: Implementing BicycleGAN, for CIS680 FA21 at University of Pennsylvania

680 Final Project: BicycleGAN Haoran Tang Instructions 1. Training To train the network, please run train.py. Change hyper-parameters and folder paths

Haoran Tang 0 Apr 22, 2022
This project uses Template Matching technique for object detecting by detection of template image over base image.

Object Detection Project Using OpenCV This project uses Template Matching technique for object detecting by detection the template image over base ima

Pratham Bhatnagar 7 May 29, 2022
This project uses Template Matching technique for object detecting by detection of template image over base image

Object Detection Project Using OpenCV This project uses Template Matching technique for object detecting by detection the template image over base ima

Pratham Bhatnagar 4 Nov 16, 2021
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 4 Mar 21, 2022
Yolo object detection - Yolo object detection with python

How to run download required files make build_image make download Docker versio

null 3 Jan 26, 2022
Implementing yolov4 target detection and tracking based on nao robot

Implementing yolov4 target detection and tracking based on nao robot

null 6 Apr 19, 2022
Auto-Lama combines object detection and image inpainting to automate object removals

Auto-Lama Auto-Lama combines object detection and image inpainting to automate object removals. It is build on top of DE:TR from Facebook Research and

null 37 Aug 2, 2022
Implement face detection, and age and gender classification, and emotion classification.

YOLO Keras Face Detection Implement Face detection, and Age and Gender Classification, and Emotion Classification. (image from wider face dataset) Ove

Chloe 11 Jan 19, 2022
This repo is about implementing different approaches of pose estimation and also is a sub-task of the smart hospital bed project :smile:

Pose-Estimation This repo is a sub-task of the smart hospital bed project which is about implementing the task of pose estimation ?? Many thanks to th

Max 11 Jul 24, 2022
BisQue is a web-based platform designed to provide researchers with organizational and quantitative analysis tools for 5D image data. Users can extend BisQue by implementing containerized ML workflows.

Overview BisQue is a web-based platform specifically designed to provide researchers with organizational and quantitative analysis tools for up to 5D

Vision Research Lab @ UCSB 23 Aug 2, 2022
This project deals with the detection of skin lesions within the ISICs dataset using YOLOv3 Object Detection with Darknet.

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Skin Lesion detection using YOLO This project deal

Lalith Veerabhadrappa Badiger 1 Nov 22, 2021