Vision Transformer for 3D medical image registration (Pytorch).

Last update: Dec 20, 2022

Overview

ViT-V-Net: Vision Transformer for Volumetric Medical Image Registration

keywords: vision transformer, convolutional neural networks, image registration

This is a PyTorch implementation of my short paper:

Chen, Junyu, et al. "ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration. " arXiv, 2021.

train.py is the training script. models.py contains ViT-V-Net model.

Pretrained ViT-V-Net: pretrained model

Dataset: Due to restrictions, we cannot distribute our brain MRI data. However, several brain MRI datasets are publicly available online: IXI, ADNI, OASIS, ABIDE, etc. Note that those datasets may not contain labels (segmentation). To generate labels, you can use FreeSurfer, which is an open-source software for normalizing brain MRI images. Here are some useful commands in FreeSurfer: Brain MRI preprocessing and subcortical segmentation using FreeSurfer.

Model Architecture:

Vision Transformer Achitecture:

Example Results:

Quantitative Results:

Reference:

TransUnet

ViT-pytorch

VoxelMorph

If you find this code is useful in your research, please consider to cite:

@misc{chen2021vitvnet,
title={ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration}, 
author={Junyu Chen and Yufan He and Eric C. Frey and Ye Li and Yong Du},
year={2021},
eprint={2104.06468},
archivePrefix={arXiv},
primaryClass={eess.IV}
}

About Me

Comments

The process of making and loading datasets
Hi,

I'm sorry to bother you. I want to run this architecture on my own datasets, but I ran into some difficulties. Prior to this, I had read the issue and code, but could not solve the problem independently, so I would like to ask the following questions:

What structure does ".pkl" use to encapsulate datasets?

Further, how to wrap the "nii/niI.gz" file into the above ".pkl" according to the desired structure? I have the preprocessed datasets of LPBA40(".niI.gz" format, a total of 40 MRI images and their corresponding Ground truth), but I do not know how to load these data into correct "pkl" format files. I tried to find the relevant information, but did not understand and implement it. I hope to get your instructions, thank you!

In view of these two questions, I made the following reflections combined with issues opened by others, but failed to solve these two problems independently, I will describe my ideas so that you can quickly clarify my questions, My partial understanding and guess are as follows:

For data encapsulation of medical images, a pre-processed MRI image and its corresponding segmentation result (label/ground truth) are encapsulated into an independent "***.pkl" file for data reading. PKL = pkload("***.pkl"); "data = pkload("***.pkl")"; Where 2 means 1(preprocessed MRI image)+1(segmentation result, i.e., ground truth), that is, data[0] is preprocessed MRI image and data[1] is its corresponding ground truth; But according to the code of datasets.py, this does not seem to be the case, because in "class JHUBrainDataset(Dataset)" getitem is "x, y = pkload(path)", In "class JHUBrainInferDataset(Dataset)", the getitem mode is "x, y, x_seg, y_seg = pkload(path)", It looks like a "***.pkl" file containing two MRI images and their corresponding two ground truths.

According to the code part of reading the dataset in the training stage and validation stage of "datasets.py", I speculated as follows: In the training stage, only preprocessed MRI images were used, without ground truth, because there are no "x_seg" and "y_seg" in "__getitem__" of "class JHUBrainDataset" ; In the validation stage, pre-processed MRI images and their corresponding segmentation results (label/ground truth) were used at the same time.

Thanks
opened by tzayuan 17
The process of making data

Hi！ I'm sorry to bother you. I followed your suggestion to convert the file to pkl format, but it still doesn't work properly. Could you please help me see where I went wrong? Train Val Very grateful for your help

opened by wenqian11 2
How do you setting your data in order to run python train.py?

Please help me setting the data in order to train. Or help me have the simply data to train ( I also download your datas that mentioned but I can not use it) I assumer use the BraTS_2018 dataset. It looks like:

Val:

When I set the above path, the following error occurs:

Thank you, very much.

opened by tphankr 2
How was the visualization results visualized?

Hi, I really wonder how the registration results were visualized? Especially for the deformed grid, many papers talked about registration but no one even mentioned how the results were visualized.

opened by chengjiawang 1
How to use the pretrained model?

I'm glad to find that you have open-sourced your pretrained model, I got it from google drive, but how can we just apply it to infer.py? What do the files in the dicterory mean in your pretrained model? Thanks for your time and reply. $4O} X_L4V34S{D11_WD1HJQ$

opened by Dududu233 1

Owner

Junyu Chen

Ph.D. student in the Department of Electrical and Computer Engineering & the Department of Radiology and Radiological Science @ Johns Hopkins University

GitHub

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

12.6k Jan 9, 2023

Registration Loss Learning for Deep Probabilistic Point Set Registration

RLLReg This repository contains a Pytorch implementation of the point set registration method RLLReg. Details about the method can be found in the 3DV

35 Nov 2, 2022

This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

vision-transformer-from-scratch This repository includes several kinds of vision transformers from scratch so that one beginner can understand the the

1 Dec 24, 2021

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Vision Longformer This project provides the source code for the vision longformer paper. Multi-Scale Vision Longformer: A New Vision Transformer for H

209 Dec 30, 2022

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

1.2k Jan 4, 2023

Build a medical knowledge graph based on Unified Language Medical System (UMLS)

UMLS-Graph Build a medical knowledge graph based on Unified Language Medical System (UMLS) Requisite Install MySQL Server 5.6 and import UMLS data int

6 Dec 25, 2022

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

61 Jan 1, 2023

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

CoTr: Efficient 3D Medical Image Segmentation by bridging CNN and Transformer This is the official pytorch implementation of the CoTr: Paper: CoTr: Ef

218 Dec 25, 2022

The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"

Swin-Unet The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"(https://arxiv.org/abs/2105.05537). A validatio

869 Jan 7, 2023

MISSFormer: An Effective Medical Image Segmentation Transformer

MISSFormer Code for paper "MISSFormer: An Effective Medical Image Segmentation Transformer". Please read our preprint at the following link: paper_add

22 Dec 24, 2022

Deep learning image registration library for PyTorch

TorchIR: Pytorch Image Registration TorchIR is a image registration library for deep learning image registration (DLIR). I have integrated several ide

40 Dec 16, 2022

GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Geometric Transformer for Fast and Robust Point Cloud Registration PyTorch imple

220 Jan 5, 2023

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

1.4k Dec 30, 2022

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

62 Dec 24, 2022

The implementation of "Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer"

Shuffle Transformer The implementation of "Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer" Introduction Very recently, window-

87 Nov 29, 2022

Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Swin-Transformer-Tensorflow A direct translation of the official PyTorch implementation of "Swin Transformer: Hierarchical Vision Transformer using Sh

52 Dec 29, 2022

Vision Transformer for 3D medical image registration (Pytorch).

Related tags

Overview

ViT-V-Net: Vision Transformer for Volumetric Medical Image Registration

Model Architecture:

Vision Transformer Achitecture:

Example Results:

Quantitative Results:

Reference:

About Me

Comments

The process of making and loading datasets

The process of making data

How do you setting your data in order to run python train.py?

How was the visualization results visualized?

How to use the pretrained model?

Owner

Junyu Chen

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Registration Loss Learning for Deep Probabilistic Point Set Registration

This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

Build a medical knowledge graph based on Unified Language Medical System (UMLS)

Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"

MISSFormer: An Effective Medical Image Segmentation Transformer

Deep learning image registration library for PyTorch

GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

The implementation of "Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer"

Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped

Breaking the Dilemma of Medical Image-to-image Translation

Copy Paste positive polyp using poisson image blending for medical image segmentation