Code for "Multi-Compound Transformer for Accurate Biomedical Image Segmentation"

Last update: Jan 5, 2023

Related tags

Overview

News

The code of MCTrans has been released. if you are interested in contributing to the standardization of the medical image analysis community, please feel free to contact me.

Introduction

This repository provides code for "Multi-Compound Transformer for Accurate Biomedical Image Segmentation" [paper].
The MCTrans repository heavily references and uses the packages of MMSegmentation, MMCV, and MONAI. We thank them for their selfless contributions

Highlights

A comprehensive toolbox for medical image segmentation, including flexible data loading, processing, modular network construction, and more.
Supports representative and popular medical image segmentation methods, e.g. UNet, UNet++, CENet, AttentionUNet, etc.

Changelog

The first version was released on 2021.7.16.

Model Zoo

Supported backbones:

VGG
ResNet

Supported methods:

Installation and Usage

Please see the guidance.md.

Citation

If you find this project useful in your research, please consider cite:

@article{ji2021multi,
  title={Multi-Compound Transformer for Accurate Biomedical Image Segmentation},
  author={Ji, Yuanfeng and Zhang, Ruimao and Wang, Huijie and Li, Zhen and Wu, Lingyun and Zhang, Shaoting and Luo, Ping},
  journal={arXiv preprint arXiv:2106.14385},
  year={2021}
}

Contribution

I don't have a lot of time to improve the code base at this stage, so if you have some free time and are interested in contributing to the standardization of the medical image analysis community, please feel free to contact me ([email protected]).

License

This project is released under the Apache 2.0 license.

Comments

训练这个模型在其他数据集

你好，我将这个模型用于别的数据集（19类），并更改了配置文件里的类别个数，但似乎并不能直接运行起来，出现了下面的错误：

Traceback (most recent call last):
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/train.py", line 131, in <module>
    main()
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/train.py", line 127, in main
    SegPipline(cfg, distributed, not args.no_validate).run()
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/pipline/segpipline.py", line 99, in run
    self.runner.run(self.data_loaders, self.cfg.workflow, self.cfg.max_epochs)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter
    **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 75, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/segmentors/base.py", line 152, in train_step
    losses = self(**data_batch)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 109, in new_func
    return old_func(*args, **kwargs)
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/segmentors/base.py", line 122, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/segmentors/encoder_decoder.py", line 92, in forward_train
    loss_aux = self._auxiliary_head_forward(x, seg_label, return_loss=True)
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/segmentors/encoder_decoder.py", line 81, in _auxiliary_head_forward
    return self.aux_head.forward_train(x, seg_label)
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/heads/mctrans_aux_head.py", line 49, in forward_train
    outputs = self.ca(inputs_flatten)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/trans/transformer.py", line 145, in forward
    query = layer(query, src)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/csip-102/CaiJiaXin/MCTrans-master/mctrans/models/trans/transformer.py", line 122, in forward
    tgt2 = self.cross_attn(tgt, src, src)[0]
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 1038, in forward
    attn_mask=attn_mask)
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py", line 5025, in multi_head_attention_forward
    k = k.contiguous().view(-1, bsz * num_heads, head_dim).transpose(0, 1)
RuntimeError: shape '[-1, 152, 16]' is invalid for input of size 2752512

我看到关闭的问题中也有类似问题，但我按照他的修改方式也并不能成功，期待你的解答

opened by imcjx 8

i had run this project and found an RuntimeError: shape '[32, 48, 16]' is invalid for input of size 22020096.

I use the pannuke dataset. But I found an error. Traceback (most recent call last): File "tools/train.py", line 138, in <module> main() File "tools/train.py", line 135, in main SegPipline(cfg, distributed, not args.no_validate).run() File "/home/aMI_projects/MCTrans-master/mctrans/pipline/segpipline.py", line 99, in run self.runner.run(self.data_loaders, self.cfg.workflow, self.cfg.max_epochs) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run epoch_runner(data_loaders[i], **kwargs) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train self.run_iter(data_batch, train_mode=True, **kwargs) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 30, in run_iter **kwargs) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 75, in train_step return self.module.train_step(*inputs[0], **kwargs[0]) File "/home/aMI_projects/MCTrans-master/mctrans/models/segmentors/base.py", line 152, in train_step losses = self(**data_batch) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 109, in new_func return old_func(*args, **kwargs) File "/home/aMI_projects/MCTrans-master/mctrans/models/segmentors/base.py", line 122, in forward return self.forward_train(img, img_metas, **kwargs) File "/home/aMI_projects/MCTrans-master/mctrans/models/segmentors/encoder_decoder.py", line 92, in forward_train loss_aux = self._auxiliary_head_forward(x, seg_label, return_loss=True) File "/home/aMI_projects/MCTrans-master/mctrans/models/segmentors/encoder_decoder.py", line 81, in _auxili ary_head_forward return self.aux_head.forward_train(x, seg_label) File "/home/aMI_projects/MCTrans-master/mctrans/models/heads/mctrans_aux_head.py", line 51, in forward_train outputs = self.ca(inputs_flatten) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/aMI_projects/MCTrans-master/mctrans/models/trans/transformer.py", line 149, in forward query = layer(query, src) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/aMI_projects/MCTrans-master/mctrans/models/trans/transformer.py", line 124, in forward tgt2 = self.cross_attn(tgt, src, src)[0] File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/torch/nn/modules/activation.py", line 1045, in forward attn_mask=attn_mask, average_attn_weights=average_attn_weights) File "/home/anaconda3/envs/mctrans/lib/python3.7/site-packages/torch/nn/functional.py", line 5301, in multi_head_attention_forward k = k.contiguous().view(k.shape[0], bsz * num_heads, head_dim).transpose(0, 1) RuntimeError: shape '[32, 48, 16]' is invalid for input of size 22020096

opened by LuluQin766 8
Some questions about your code
Hi, thank you for your work. But I've noticed some inconsistencies with your paper:

It seems like the features extracted by the self-attention module are fed into the UNet decoder directly instead of the cross-attention module.

Proxy embedding is added to each input of the cross-attention layer, which seems not mentioned in the paper.

But maybe I misunderstood something? Look forward to your reply.
opened by hetranger 8
'Single-channel predictions not supported'

Hello, i have a"Single channel image training error" problems。How can I solve that? Could I get your some suggestions？ Thank you very much！！！

opened by jscodecode 5
The implement of NonLocalUNet is quiet different from the official implement

The implement of NonLocalUNet in this repo is VGG encoder + NonLocal center + UNet decoder, but the official implement is: replacing upsample and downsample with Global Aggregation Block, reduce the layers of UNet and replace the Concatenation with Add. Refer to https://github.com/divelab/Non-local-U-Nets、https://arxiv.org/abs/1812.04103

opened by Lloyd-Pottiger 3
gpu memory

Hi, Thanks for your great job, i like your algorithm very much. i have been trying to reproduce your result, may i ask how big is your gpu memory, i have 16GB, but always "CUDA out of memory."

opened by jjjcs 2
实验结果与论文不同

+---------------------------+--------+-------+-------+--------+-----------+--------+ | Class | Dice | Acc | IoU | Fscore | Precision | Recall | +---------------------------+--------+-------+-------+--------+-----------+--------+ | Background | 142.16 | 600.0 | 245.8 | 142.16 | 80.63 | 600.0 | | Neoplastic | 0.0 | 0.0 | 0.0 | nan | nan | 0.0 | | Inflammatory | 0.0 | 0.0 | 0.0 | nan | nan | 0.0 | | Connective | 0.0 | 0.0 | 0.0 | nan | nan | 0.0 | | Dead | 0.0 | 0.0 | 0.0 | nan | nan | 0.0 | | Non-Neoplastic Epithelial | 0.0 | 0.0 | 0.0 | nan | nan | 0.0 | +---------------------------+--------+-------+-------+--------+-----------+--------+ 这是我跑完模型它在log中统计的结果，为什么所有结果都是0？而且我没有发现生成的预测图。

opened by yimuu 2
DLL load failed

How can I solve this problems？ Thanks you very much！！！

Traceback (most recent call last): File "Z:\py_item\7_MCTrans-master\MCTrans-master\tools\train.py", line 14, in <module> from mctrans.pipline import SegPipline File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\pipline\__init__.py", line 1, in <module> from .segpipline import SegPipline File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\pipline\segpipline.py", line 7, in <module> from mctrans.models import build_model File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\__init__.py", line 1, in <module> from .centers import * File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\centers\__init__.py", line 1, in <module> from .mctrans import MCTrans File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\centers\mctrans.py", line 8, in <module> from ..ops.modules import MSDeformAttn File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\ops\modules\__init__.py", line 9, in <module> from .ms_deform_attn import MSDeformAttn File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\ops\modules\ms_deform_attn.py", line 21, in <module> from ..functions import MSDeformAttnFunction File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\ops\functions\__init__.py", line 9, in <module> from .ms_deform_attn_func import MSDeformAttnFunction File "Z:\py_item\7_MCTrans-master\MCTrans-master\mctrans\models\ops\functions\ms_deform_attn_func.py", line 18, in <module> import MultiScaleDeformableAttention as MSDA ImportError: DLL load failed: 找不到指定的模块。

opened by jscodecode 1
请问使用.npy格式的数据和使用.png格式的数据有区别吗

首先感谢作者的工作！您提供的PanNuke数据集里面有split-images(masks)-npy和split-images(masks)，前者是.npy格式，后者是.png格式，请问使用这两种不同格式的数据集进行训练和测试是等价的吗？因为我load了npy格式的，发现其中的数据也是0~255的，和png格式的数据是一样的，是不是代表着使用.png格式的数据可以获得和.npy格式数据的同样效果？感谢！

opened by BerenChou 1
使用其他数据集出现的问题 AttributeError: 'ConfigDict' object has no attribute 'log_level'等等

我是用自己的数据集，使用模型mctrans_vgg32_d5.py，显示：Traceback (most recent call last): File "G:/codelearn/papercode329/3MCTrans-master/tools/train.py", line 128, in main() File "G:/codelearn/papercode329/3MCTrans-master/tools/train.py", line 103, in main print(cfg.log_level) File "g:\pythonproject\swin-zhenghe\mmcv-1.3.1\mmcv\utils\config.py", line 425, in getattr return getattr(self._cfg_dict, name) File "g:\pythonproject\swin-zhenghe\mmcv-1.3.1\mmcv\utils\config.py", line 44, in getattr raise ex AttributeError: 'ConfigDict' object has no attribute 'log_level' 请问有人知道什么问题吗，感谢

手动将log level设置为0

出现错误： AttributeError: 'ConfigDict' object has no attribute 'data'

我怀疑是配置文件没弄好，有大哥知道嘛

opened by haomayang1126 1
unet_vgg32_d5_256x256_400ep_pannuke.py

你好，我在运行程序时，出现了如下错误： FileNotFoundError: file "/home/yyang17/projects/MCTrans/configs/debug/unet_vgg32_d5_256x256_400ep_pannuke.py" does not exist 在项目里面搜索 unet_vgg32_d5_256x256_400ep_pannuke.py 也没有发现，请问是否有部分模型文件没有上传呢？

opened by yangyang117 1
Where can I find the Appendix mentioned in the paper?

I want to know more about the folding operation in the model, it is said that details can be found in the Appendix, but I cannot find the Appendix. Any idea? Thanks.

opened by rongzhao-zhang 0

Owner

GitHub

TensorFlow code for the neural network presented in the paper: "Structural Language Models of Code" (ICML'2020)

SLM: Structural Language Models of Code This is an official implementation of the model described in: "Structural Language Models of Code" [PDF] To ap

73 Nov 6, 2022

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Visual Understanding Lab @ Samsung AI Center Moscow

18 Oct 6, 2022

A code generator from ONNX to PyTorch code

onnx-pytorch Generating pytorch code from ONNX. Currently support onnx==1.9.0 and torch==1.8.1. Installation From PyPI pip install onnx-pytorch From

94 Jan 6, 2023

This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code for training a DPR model then continuing training with RAG.

KGI (Knowledge Graph Induction) for slot filling This is the code for our KILT leaderboard submission to the T-REx and zsRE tasks. It includes code fo

72 Jan 6, 2023

Convert Python 3 code to CUDA code.

Py2CUDA Convert python code to CUDA. Usage To convert a python file say named py_file.py to CUDA, run python generate_cuda.py --file py_file.py --arch

3 Jul 14, 2021

Empirical Study of Transformers for Source Code & A Simple Approach for Handling Out-of-Vocabulary Identifiers in Deep Learning for Source Code

Transformers for variable misuse, function naming and code completion tasks The official PyTorch implementation of: Empirical Study of Transformers fo

56 Nov 15, 2022

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.

This repository is a toolkit to do machine learning for programming languages. It implements tokenization, dataset preprocessing, model training and m

408 Jan 1, 2023

Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

1 Oct 26, 2021

Low-code/No-code approach for deep learning inference on devices

EzEdgeAI A concept project that uses a low-code/no-code approach to implement deep learning inference on devices. It provides a componentized framewor

7 Apr 5, 2022

Code for all the Advent of Code'21 challenges mostly written in python

Advent of Code 21 Code for all the Advent of Code'21 challenges mostly written in python. They are not necessarily the best or fastest solutions but j

4 May 26, 2022

Code to use Augmented Shapiro Wilks Stopping, as well as code for the paper "Statistically Signifigant Stopping of Neural Network Training"

This codebase is being actively maintained, please create and issue if you have issues using it Basics All data files are included under losses and ea

32 Nov 9, 2021

Opinionated code formatter, just like Python's black code formatter but for Beancount

beancount-black Opinionated code formatter, just like Python's black code formatter but for Beancount Try it out online here Features MIT licensed - b

16 Oct 11, 2022

a delightful machine learning tool that allows you to train, test and use models without writing code

igel A delightful machine learning tool that allows you to train/fit, test and use models without writing code Note I'm also working on a GUI desktop

3k Jan 5, 2023

Pytorch Lightning code guideline for conferences

Deep learning project seed Use this seed to start new deep learning / ML projects. Built in setup.py Built in requirements Examples with MNIST Badges

1k Jan 2, 2023

Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

397 Dec 30, 2022

Applications using the GTN library and code to reproduce experiments in "Differentiable Weighted Finite-State Transducers"

gtn_applications An applications library using GTN. Current examples include: Offline handwriting recognition Automatic speech recognition Installing

68 Dec 29, 2022