This code is for our paper "VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers"

Overview

ICCV Workshop 2021 VTGAN

PWC

This code is for our paper "VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers" which is part of the supplementary materials for ICCV 2021 Workshop on Computer Vision for Automated Medical Diagnosis. The paper has since been accpeted and presented at ICCV 2021 Workshop.

Arxiv Pre-print

https://arxiv.org/abs/2104.06757

CVF ICCVW 2021

https://openaccess.thecvf.com/content/ICCV2021W/CVAMD/html/Kamran_VTGAN_Semi-Supervised_Retinal_Image_Synthesis_and_Disease_Prediction_Using_Vision_ICCVW_2021_paper.html

IEE Xplore ICCVW 2021

https://ieeexplore.ieee.org/document/9607858

Citation

@INPROCEEDINGS{9607858,
  author={Kamran, Sharif Amit and Hossain, Khondker Fariha and Tavakkoli, Alireza and Zuckerbrod, Stewart Lee and Baker, Salah A.},
  booktitle={2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)}, 
  title={VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers}, 
  year={2021},
  volume={},
  number={},
  pages={3228-3238},
  doi={10.1109/ICCVW54120.2021.00362}
}

Pre-requisite

  • Ubuntu 18.04 / Windows 7 or later
  • NVIDIA Graphics card

Installation Instruction for Ubuntu

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt install python3.7
  • Install Tensorflow-Gpu version-2.5.0 and Keras version-2.5.0
sudo pip3 install tensorflow-gpu
sudo pip3 install keras
  • Install packages from requirements.txt
sudo pip3 install -r requirements.txt

Dataset download link for Hajeb et al.

https://sites.google.com/site/hosseinrabbanikhorasgani/datasets-1/fundus-fluorescein-angiogram-photographs--colour-fundus-images-of-diabetic-patients
  • Please cite the paper if you use their data
@article{hajeb2012diabetic,
  title={Diabetic retinopathy grading by digital curvelet transform},
  author={Hajeb Mohammad Alipour, Shirin and Rabbani, Hossein and Akhlaghi, Mohammad Reza},
  journal={Computational and mathematical methods in medicine},
  volume={2012},
  year={2012},
  publisher={Hindawi}
}
  • Folder structure for data-preprocessing given below. Please make sure it matches with your local repository.
├── Dataset
|   ├──ABNORMAL
|   ├──NORMAL

Dataset Pre-processing

  • Type this in terminal to run the random_crop.py file
python3 random_crop.py --output_dir=data --input_dim=512 --datadir=Dataset
  • There are different flags to choose from. Not all of them are mandatory.
    '--input_dim', type=int, default=512
    '--n_crops', type=int, default=50
    '--datadir', type=str, required=True, help='path/to/data_directory',default='Dataset'
    '--output_dir', type=str, default='data'   

NPZ file conversion

  • Convert all the images to npz format
python3 convert_npz.py --outfile_name=vtgan --input_dim=512 --datadir=data --n_crops=50
  • There are different flags to choose from. Not all of them are mandatory.
    '--input_dim', type=int, default=512
    '--n_crops', type=int, default=50
    '--datadir', type=str, required=True, help='path/to/data_directory',default='data'
    '--outfile_name', type=str, default='vtgan'
    '--n_images', type=int, default=17

Training

  • Type this in terminal to run the train.py file
python3 train.py --npz_file=vtgan --batch=2 --epochs=100 --savedir=VTGAN
  • There are different flags to choose from. Not all of them are mandatory
    '--epochs', type=int, default=100
    '--batch_size', type=int, default=2
    '--npz_file', type=str, default='vtgan', help='path/to/npz/file'
    '--input_dim', type=int, default=512
    '--n_patch', type=int, default=64
    '--savedir', type=str, required=False, help='path/to/save_directory',default='VTGAN'
    '--resume_training', type=str, required=False,  default='no', choices=['yes','no']

License

The code is released under the BSD 3-Clause License, you can read the license file included in the repository for details.

Comments
  • dataloader.py

    dataloader.py

    I think there are some problems in the dataloader.py file, as an example: In the generate_real_data function, the function returns the y2 value, but nothing is defined as y2 inside it. may you please update this .py file to the latest version?

    bug 
    opened by niloofarAzari 16
  • Questions about the label

    Questions about the label

    I have notice that you use the -1*np.ones, in other word, all -1 patch, to represent the label 1. You have marked this in your code in the function "generate_real_data":

    # generate 'real' class labels (1)
    y1 = -np.ones((random_samples, patch_shape[0], patch_shape[0], 1))
    

    why you use the -1 to represent "Yes", is it because of the activation you use is the "tanh"? Since the use of tanh in discriminator is rare(it seems that sigmoid is more normal?) , I couldn't understand using -1 patches to represent "Yes", but using 1 patches to represent "No". Could you make a little bit explaination?

    opened by Tinysqua 5
  • Convert data problem

    Convert data problem

    Hi, thanks for your contribution! There is a problem when i run convert_npz.py

    Loaded:  (850, 512, 512, 3) (850, 512, 512, 1)
    Traceback (most recent call last):
      File "convert_npz.py", line 53, in <module>
        savez_compressed(filename, src_images, tar_images.label)
    AttributeError: 'numpy.ndarray' object has no attribute 'label'
    

    it will solved when i revome label savez_compressed(filename, src_images, tar_images.label) but there is another problem when i run train.py

    File "train.py", line 179, in <module>
        dataset = load_real_data(args.npz_file+'.npz')
      File "/home/VTGAN-main/src/dataloader.py", line 9, in load_real_data
        X1, X2, y = data['arr_0'], data['arr_1'], data['arr_2']
      File "/opt/conda/lib/python3.8/site-packages/numpy/lib/npyio.py", line 260, in __getitem__
        raise KeyError("%s is not a file in the archive" % key)
    KeyError: 'arr_2 is not a file in the archive'
    

    So i think the main problem is in convert_npz.py, by the way, my Tensorflow==2.41,keras==2.4.3. Thanks for your time, appreciate it!

    opened by zhangbaijin 4
  • Ask For Details

    Ask For Details

        Hello, dear professor! Glad to receive your reply! It is a great honor to have this opportunity to communicate with you! I've thought about your reply and have some new questions to chat with you!
    

    The first point is that I am currently reproducing the Pytorch version with reference to your Tensorflow version of Fundus2Angio. My current reproduction effect is 130. It should be that there is a problem with my implementation. I will carefully modify it according to your suggestion. I am also going to implement the Tensorflow version of Fundus2Angio to compare the effects. In addition, I am running the Pytorch version of VTGAN that you taught on the homepage. I still have questions about the model selection of these two jobs. I would like to ask how you choose the best What about the best model? Specifically, how many epoch models are used for testing? 

    The second point is that the dataset link you provided contains a total of 68 pairs of images. I would like to ask whether these images are all from the 850 pairs of images obtained after the random crop operation of the original dataset? Or are they derived from datasets elsewhere? 

    The third point is that I want to communicate with you about the Gan image generation task of converting Fundus to Angio image. Do you think there is still room for improvement and research? Why I ask this is because I want to take this direction as my future research direction! 

    If I can get your patient reply one by one, I really appreciate it! Thanks for taking the time to respond to me! I wish you success in your work, Professor!

    opened by WZRJohn 1
  • change input size from 512 to 256

    change input size from 512 to 256

    Thanks for your nice coding! I am trying to run your code on the dataset with a bit lower resolution and I want to crop images to 256x256 instead of 512x512. I have modified the input_dim for preprocessing files but when I start training, some errors occur. Therefore, I want to ask what should I change if the input size is 256x256. (i.e., input_dim, n_patch, patch_size, or any network structures...?)

    opened by ustlsh 1
  • Article reproduction effect

    Article reproduction effect

    Hello, dear professor! I've been working on your Fundus2Angio and VTGAN work recently, I'm very interested in your work and trying to reproduce the results in the paper.

    However, I have encountered some difficulties at present, and I would like to ask you a question to solve it.

    What I want to ask is which pictures did you use for testing and how many pictures did you use to calculate the FID score?

    In addition, there is another problem that I set the parameters batchsize to 4 and epoch to 100, and select the model of the last epoch from the trained model to test 850 pictures (the training set is also 850 pictures), so the calculation My FID score is as high as 130, and I really encountered difficulties, so I would like to ask you for help.

    If you can give me a reply in your busy schedule, I will be very grateful!

    opened by WZRJohn 3
  • need some help

    need some help

    hi,i love this application. I rewirtte it in pytorch and upload to github, i will appreciate that if you can have a look at my github account. By the way, my code don't work well even though it can run. Maybe you can give me some advise for me to improve it. i have no idea about why the network doesn't learn anything about blood vessels, but learn the style and optic disc area. Thank you for your time. 0

    opened by superstarNathan 4
  • Pretrained model

    Pretrained model

    Thanks a lot for your help!I would like to ask if I can get the pretrained model, because I feel that it takes a long time to train from 0. and how many time you spend on train? Best wishes

    opened by zhangbaijin 1
  • train problem: Incompatible shapes: [2,64,64,1] vs. [2,32,32,1]

    train problem: Incompatible shapes: [2,64,64,1] vs. [2,32,32,1]

    I am sorry to ask many questions, when i run train.py,there is a problem, and my tf==2.5.0 , keras==2.5.0, appreciate it, best wishes :D Traceback (most recent call last): File "train.py", line 220, in <module> train(d_model1, d_model2, g_model_coarse, g_model_fine, vt_model, dataset, n_epochs=args.epochs, File "train.py", line 73, in train d_loss4 = d_model2.train_on_batch([X_realA_half,X_fakeB_half],[y1_coarse,y2,d_feat2_fake[2]])[0] # [,X_fakeB_half] File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 1800, in train_on_batch logs = self.train_function(iterator) File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 830, in train_function return step_function(self, iterator) File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 820, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 1285, in run return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 2833, in call_for_each_replica return self._call_for_each_replica(fn, args, kwargs) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/distribute/distribute_lib.py", line 3608, in _call_for_each_replica return fn(*args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 597, in wrapper return func(*args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 813, in run_step outputs = model.train_step(data) File "/opt/conda/lib/python3.8/site-packages/keras/engine/training.py", line 771, in train_step loss = self.compiled_loss( File "/opt/conda/lib/python3.8/site-packages/keras/engine/compile_utils.py", line 201, in __call__ loss_value = loss_obj(y_t, y_p, sample_weight=sw) File "/opt/conda/lib/python3.8/site-packages/keras/losses.py", line 142, in __call__ losses = call_fn(y_true, y_pred) File "/opt/conda/lib/python3.8/site-packages/keras/losses.py", line 246, in call return ag_fn(y_true, y_pred, **self._fn_kwargs) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper return target(*args, **kwargs) File "/opt/conda/lib/python3.8/site-packages/keras/losses.py", line 1202, in mean_squared_error return backend.mean(tf.math.squared_difference(y_pred, y_true), axis=-1) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/ops/gen_math_ops.py", line 10405, in squared_difference _ops.raise_from_not_ok_status(e, name) File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 6897, in raise_from_not_ok_status six.raise_from(core._status_to_exception(e.code, message), None) File "<string>", line 3, in raise_from tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [2,64,64,1] vs. [2,32,32,1] [Op:SquaredDifference]

    opened by zhangbaijin 2
Owner
Sharif Amit Kamran
Interested in Deep learning for Medical Imaging and Computer Vision. Designing robust generative architectures for Ophthalmology and Calcium Imaging.
Sharif Amit Kamran
Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Zhensu Sun 1 Oct 26, 2021
Code for our method RePRI for Few-Shot Segmentation. Paper at http://arxiv.org/abs/2012.06166

Region Proportion Regularized Inference (RePRI) for Few-Shot Segmentation In this repo, we provide the code for our paper : "Few-Shot Segmentation Wit

Malik Boudiaf 138 Dec 12, 2022
Code for our ICASSP 2021 paper: SA-Net: Shuffle Attention for Deep Convolutional Neural Networks

SA-Net: Shuffle Attention for Deep Convolutional Neural Networks (paper) By Qing-Long Zhang and Yu-Bin Yang [State Key Laboratory for Novel Software T

Qing-Long Zhang 199 Jan 8, 2023
Code for our CVPR 2021 paper "MetaCam+DSCE"

Joint Noise-Tolerant Learning and Meta Camera Shift Adaptation for Unsupervised Person Re-Identification (CVPR'21) Introduction Code for our CVPR 2021

FlyingRoastDuck 59 Oct 31, 2022
Code for our CVPR2021 paper coordinate attention

Coordinate Attention for Efficient Mobile Network Design (preprint) This repository is a PyTorch implementation of our coordinate attention (will appe

Qibin (Andrew) Hou 726 Jan 5, 2023
[CVPR2021] The source code for our paper 《Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Learning》.

TBE The source code for our paper "Removing the Background by Adding the Background: Towards Background Robust Self-supervised Video Representation Le

Jinpeng Wang 150 Dec 28, 2022
Code for our paper at ECCV 2020: Post-Training Piecewise Linear Quantization for Deep Neural Networks

PWLQ Updates 2020/07/16 - We are working on getting permission from our institution to release our source code. We will release it once we are granted

null 54 Dec 15, 2022
Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

CorDA Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation Prerequisite Please create and activate the follo

Qin Wang 60 Nov 30, 2022
the code for our CVPR 2021 paper Bilateral Grid Learning for Stereo Matching Network [BGNet]

BGNet This repository contains the code for our CVPR 2021 paper Bilateral Grid Learning for Stereo Matching Network [BGNet] Environment Python 3.6.* C

3DCV developer 87 Nov 29, 2022
PyTorch code for our paper "Attention in Attention Network for Image Super-Resolution"

Under construction... Attention in Attention Network for Image Super-Resolution (A2N) This repository is an PyTorch implementation of the paper "Atten

Haoyu Chen 71 Dec 30, 2022
Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Image Crop Analysis This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post. If you plan to use this

Twitter Research 239 Jan 2, 2023
Code for our ACL 2021 paper - ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

ConSERT Code for our ACL 2021 paper - ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer Requirements torch==1.6.0

Yan Yuanmeng 478 Dec 25, 2022
code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

SHOT++ Code for our TPAMI submission "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer" that is ext

null 75 Dec 16, 2022
Code for our paper "SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization", ACL 2021

SimCLS Code for our paper: "SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization", ACL 2021 1. How to Install Requirements

Yixin Liu 150 Dec 12, 2022
Code for our paper "Sematic Representation for Dialogue Modeling" in ACL2021

AMR-Dialogue An implementation for paper "Semantic Representation for Dialogue Modeling". You may find our paper here. Requirements python 3.6 pytorch

xfbai 45 Dec 26, 2022
Code for our ACL 2021 paper "One2Set: Generating Diverse Keyphrases as a Set"

One2Set This repository contains the code for our ACL 2021 paper “One2Set: Generating Diverse Keyphrases as a Set”. Our implementation is built on the

Jiacheng Ye 63 Jan 5, 2023
Code for our TKDE paper "Understanding WeChat User Preferences and “Wow” Diffusion"

wechat-wow-analysis Understanding WeChat User Preferences and “Wow” Diffusion. Fanjin Zhang, Jie Tang, Xueyi Liu, Zhenyu Hou, Yuxiao Dong, Jing Zhang,

null 18 Sep 16, 2022
This is the source code for our ICLR2021 paper: Adaptive Universal Generalized PageRank Graph Neural Network.

GPRGNN This is the source code for our ICLR2021 paper: Adaptive Universal Generalized PageRank Graph Neural Network. Hidden state feature extraction i

Jianhao 92 Jan 3, 2023
PyTorch code for our paper "Image Super-Resolution with Non-Local Sparse Attention" (CVPR2021).

Image Super-Resolution with Non-Local Sparse Attention This repository is for NLSN introduced in the following paper "Image Super-Resolution with Non-

null 143 Dec 28, 2022