This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

Overview

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer

This repo is the official implementation for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer. The multimodal brain tumor dataset (BraTS 2019) could be acquired from here.

TransBTS

TransBTS Architecture of 3D TransBTS.

Requirements

  • python 3.7
  • pytorch 1.6.0
  • torchvision 0.7.0
  • pickle
  • nibabel

Data preprocess

After downloading the dataset from here, data preprocessing is needed which is to convert the .nii files as .pkl files and realize date normalization.

python3 preprocess.py

Training

Run the training script on BraTS dataset. Distributed training is available for training the proposed TransBTS, where --nproc_per_node decides the numer of gpus and --master_port implys the port number.

python3 -m torch.distributed.launch --nproc_per_node=4 --master_port 20003 train.py

Testing

If you want to test the model which has been trained on the BraTS dataset, run the testing script as following.

python3 test.py

After the testing process stops, you can upload the submission file to here for the final Dice_scores.

Quantitive comparison of performance

Quantitive comparison of performance on BraTS2019 validation set between our proposed TransBTS with other SOTA methods.

quantitive_comparison

Visual comparison

Here are some samples from BraTS 2019 dataset for visual comparison between our proposed TransBTS with other SOTA methods.

visual_comparison

Citation

@article{wang2021transbts,
  title={TransBTS: Multimodal Brain Tumor Segmentation Using Transformer},  
  author={Wang, Wenxuan and Chen, Chen and Ding, Meng and Li, Jiangyun and Yu, Hong and Zha, Sen},
  journal={arXiv preprint arXiv:2103.04430},
  year={2021}
}

Reference

1.setr-pytorch

2.BraTS2017

Comments
  • Submission Stats results are not giving the required csv file

    Submission Stats results are not giving the required csv file

    I have used your code and tested it to predict the segmentation results for validation set. Then I submitted the generated segmentation files on cbica evaluation portal for BraTS 2020 validation segmentation task. My job is completed but in zip file I am not receiving any csv file with segmentation scores. Results-733562074827703492.zip

    Please help.

    opened by gjkaur 21
  • why are there  no label 0 in my predict ?

    why are there no label 0 in my predict ?

    My dear friend, when I run your code, there are only label 1,2,4 in my result . All the background are label 1, I was puzzled. I just only download your code here , then change the path to my environment, and edit the parallel code into single GPU.
    My platform is Ubuntu20.04 + RXT 3090.

    Have you ever encountered the same problem ?

    Would you please give me some suggestions?

    Many thanks in advance!

    opened by cqlouis 3
  • 关于num_gpu的疑问

    关于num_gpu的疑问

    Wenxuan您好, 在train.py中,有: train_loader = DataLoader(dataset=train_set, sampler=train_sampler, batch_size=args.batch_size // num_gpu, drop_last=True, num_workers=args.num_workers, pin_memory=True) 此处,我不是太明白batch_size这里为什么要用args.batch_size除以num_gpu? args.batch_size 为8, 再除以gpu数量4, 等于2, 相当于每个gpu得到的数据batch_size为2? 难道不是每个gpu得到的数据batch_size就是args.batch_size 吗? 谢谢,并祝好!

    opened by cqlouis 2
  • would you please explain tailor_and_concat(x, model) for me?

    would you please explain tailor_and_concat(x, model) for me?

    My dear friend, sorry to bother you.

    I'm a greenhand in medical image segmentation, maybe this function seems very easy for you , but I can not understand it clearly.

    Both 中文 and English are OK, ^__^

    Many thanks in advance

    opened by cqlouis 2
  • Some question about the code

    Some question about the code

    Hi, Great work. But there are few small problems that puzzle me. 1) The image shape changes from 240240155 to 240240160,what are the considerations here? 2) I can't understand the intmd_encoder_outputs of the encoder which is the output intmd_x of transformer, what's the difference between xand intmd_x? Besides, the encoder_output seems to be not used in Decoder.

    x, intmd_x = self.transformer(x)

    def forward(self, x, auxillary_output_layers=[1, 2, 3, 4]):
    
         x1_1, x2_1, x3_1, encoder_output, intmd_encoder_outputs = self.encode(x)
    
         decoder_output = self.decode(
             x1_1, x2_1, x3_1, encoder_output, intmd_encoder_outputs, auxillary_output_layers
         )
    
         if auxillary_output_layers is not None:
             auxillary_outputs = {}
             for i in auxillary_output_layers:
                 val = str(2 * i - 1)
                 _key = 'Z' + str(i)
                 auxillary_outputs[_key] = intmd_encoder_outputs[val]
    
             return decoder_output
    
         return decoder_output
    
    
    opened by chengjianhong 2
  • RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 10.76 GiB total capacity; 9.65 GiB already allocated; 124.19 MiB free; 9.68 GiB reserved in total by PyTorch)

    RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 10.76 GiB total capacity; 9.65 GiB already allocated; 124.19 MiB free; 9.68 GiB reserved in total by PyTorch)

    Hi, I had meet "RuntimeError: CUDA out of memory" when I reccurent, with "python -m torch.distributed.launch --nproc_per_node=2 --master_port 20011 train.py --gpu 0,1", and my env is RTX2080ti. could you tell me what is nvidia device you used? thanks you very much!

    opened by caixh39 2
  • Can u plz give some idea about resuming training if interrupted in between

    Can u plz give some idea about resuming training if interrupted in between

    Hello wenxuan. I hope you recognize me. I had raised an issue earlier about predictions on brats 2020 data. I need your help in resuming the training. I am training model on google colab which shuts after 12 hours. So I want to resume it from the same point where it gets stopped.

    opened by gjkaur 1
  • Log error

    Log error

    Traceback (most recent call last): File "train.py", line 290, in main_worker() File "train.py", line 114, in main_worker log_args(log_file) File "train.py", line 271, in log_args fh = logging.FileHandler(log_file) File "/usr/lib/python3.7/logging/init.py", line 1087, in init StreamHandler.init(self, self._open()) File "/usr/lib/python3.7/logging/init.py", line 1116, in _open return open(self.baseFilename, self.mode, encoding=self.encoding) FileNotFoundError: [Errno 2] No such file or directory: '/content/drive/MyDrive/Colab Notebooks/Wenxuan_segmentation/log/TransBTS2021-06-26.txt'

    opened by kiransrdr 1
  • in the transformer part, is there any Pytorch  API to  Transformer Encoder?

    in the transformer part, is there any Pytorch API to Transformer Encoder?

    Hi Dude , good afternoon. I'm reading your code. To my understanding , Transformer.py is for define the transformer encoder.
    I'm wondering whether there is any API that we can use directly instead of tedious coding.

    I just found torch.nn.TransformerEncoder exists. Is it usable in this project? Have you ever thought about this problem before?

    Many thanks!

    opened by cqlouis 1
  • About 3D transformer

    About 3D transformer

    Hello, I run the tranBTS code for 3D medical images, the shape of the generated attention weight is (Batch_size,num_heads, n_patches,n_patches ), is this not applicable to 3D images? Thank you in advance!

    opened by vv123-star 1
  • About training settings

    About training settings

    First of all, thank you for your paper and code. Actually, I am a bit puzzled about the training settings. As mentioned in the paper, the model was trained for 6000 epochs, with batch size 8 on 4 RTX GPU. Indeed I am not quite familiar with this dataset and wonder the scale of it? how many iterations/ steps in each epoch? if the dataset is small, why it takes so many epochs to converge ? Thanks again!

    opened by cvbird 1
  • About the kindey tumor dataset KiTS mentioned in Data Acquisition

    About the kindey tumor dataset KiTS mentioned in Data Acquisition

    I noticed that you used the BraTS data set in preprocess.py to process the 4 dimensions of the MRI image. I would like to ask that there is only one image.nii and one segmentation.nii in KiTS, and there is no 4-dimensional .nii , May I ask how to do preprocessing for this piece? my solution: Is it possible to just remove this part of the code? Or copy 4 copies of image.nii and use this code? Hope to get your reply, best wishes! preprocess

    opened by Li-jingfeng 0
  • Why setting num_classes+1 as the output classes?

    Why setting num_classes+1 as the output classes?

    I notice that the output of the model contains 4 classes, but there's only 3 classes of tumor. I'm wondering what is the purpose of the dimension zero?

    opened by johnson111788 0
  • About validation set and test set

    About validation set and test set

    I divided my data into training sets, validation sets, and test sets. When training, I use "mode=train", but when I learn the code, I find that the validation set is not in training, and there is no parameter adjustment. In test.py, although there is an option to opt out of validation at training time, the setup of a single dataset file does not satisfy both validation and testing. 我将我的数据分为训练集、验证集和测试集。训练时,使用"mode=train",但我学习代码时发现验证集并没有进入训练,没有进行参数调整.在test.py中,尽管有选择在训练时未进行验证的选项,但只有一个数据集文件的设置, 并不能同时满足验证和测试。

    opened by Lnan1615 0
Owner
Raymond
CV rookie
Raymond
Official repository with code and data accompanying the NAACL 2021 paper "Hurdles to Progress in Long-form Question Answering" (https://arxiv.org/abs/2103.06332).

Hurdles to Progress in Long-form Question Answering This repository contains the official scripts and datasets accompanying our NAACL 2021 paper, "Hur

Kalpesh Krishna 41 Nov 8, 2022
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 153 Dec 14, 2022
Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Meta-Solver for Neural Ordinary Differential Equations Towards robust neural ODEs using parametrized solvers. Main idea Each Runge-Kutta (RK) solver w

Julia Gusak 25 Aug 12, 2021
The official implementation of NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021]. https://arxiv.org/pdf/2101.12378.pdf

NeMo: Neural Mesh Models of Contrastive Features for Robust 3D Pose Estimation [ICLR-2021] Release Notes The offical PyTorch implementation of NeMo, p

Angtian Wang 76 Nov 23, 2022
Source Code for DialogBERT: Discourse-Aware Response Generation via Learning to Recover and Rank Utterances (https://arxiv.org/pdf/2012.01775.pdf)

DialogBERT This is a PyTorch implementation of the DialogBERT model described in DialogBERT: Neural Response Generation via Hierarchical BERT with Dis

Xiaodong Gu 67 Jan 6, 2023
PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

This is the original implementation of our paper, A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem (arXiv:1706.1

Zhengyao Jiang 1.5k Dec 29, 2022
Cancer-and-Tumor-Detection-Using-Inception-model - In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks, specifically here the Inception model by google.

Cancer-and-Tumor-Detection-Using-Inception-model In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks

Deepak Nandwani 1 Jan 1, 2022
Repo for "Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions" https://arxiv.org/abs/2201.12296

Benchmarking Robustness of 3D Point Cloud Recognition against Common Corruptions This repo contains the dataset and code for the paper Benchmarking Ro

Jiachen Sun 168 Dec 29, 2022
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
Self-supervised Multi-modal Hybrid Fusion Network for Brain Tumor Segmentation

JBHI-Pytorch This repository contains a reference implementation of the algorithms described in our paper "Self-supervised Multi-modal Hybrid Fusion N

FeiyiFANG 5 Dec 13, 2021
ISTR: End-to-End Instance Segmentation with Transformers (https://arxiv.org/abs/2105.00637)

This is the project page for the paper: ISTR: End-to-End Instance Segmentation via Transformers, Jie Hu, Liujuan Cao, Yao Lu, ShengChuan Zhang, Yan Wa

Jie Hu 182 Dec 19, 2022
This repository contains various models targetting multimodal representation learning, multimodal fusion for downstream tasks such as multimodal sentiment analysis.

Multimodal Deep Learning ?? ?? ?? Announcing the multimodal deep learning repository that contains implementation of various deep learning-based model

Deep Cognition and Language Research (DeCLaRe) Lab 398 Dec 30, 2022
Official implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis https://arxiv.org/abs/2011.13775

CIPS -- Official Pytorch Implementation of the paper Image Generators with Conditionally-Independent Pixel Synthesis Requirements pip install -r requi

Multimodal Lab @ Samsung AI Center Moscow 201 Dec 21, 2022
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

null 967 Jan 4, 2023
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
TumorInsight is a Brain Tumor Detection and Classification model built using RESNET50 architecture.

A Brain Tumor Detection and Classification Model built using RESNET50 architecture. The model is also deployed as a web application using Flask framework.

Pranav Khurana 0 Aug 17, 2021
This repository contains the code used for Predicting Patient Outcomes with Graph Representation Learning (https://arxiv.org/abs/2101.03940).

Predicting Patient Outcomes with Graph Representation Learning This repository contains the code used for Predicting Patient Outcomes with Graph Repre

Emma Rocheteau 76 Dec 22, 2022
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022