Vision Transformer for 3D medical image registration (Pytorch).

Related tags

Deep Learning convolutional-neural-networks image-registration pytorch-implementation vision-transformer
Overview

ViT-V-Net: Vision Transformer for Volumetric Medical Image Registration

keywords: vision transformer, convolutional neural networks, image registration

This is a PyTorch implementation of my short paper:

Chen, Junyu, et al. "ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration. " arXiv, 2021.

train.py is the training script. models.py contains ViT-V-Net model.

Pretrained ViT-V-Net: pretrained model

Dataset: Due to restrictions, we cannot distribute our brain MRI data. However, several brain MRI datasets are publicly available online: IXI, ADNI, OASIS, ABIDE, etc. Note that those datasets may not contain labels (segmentation). To generate labels, you can use FreeSurfer, which is an open-source software for normalizing brain MRI images. Here are some useful commands in FreeSurfer: Brain MRI preprocessing and subcortical segmentation using FreeSurfer.

Model Architecture:

Vision Transformer Achitecture:

Example Results:

Quantitative Results:

Reference:

TransUnet

ViT-pytorch

VoxelMorph

If you find this code is useful in your research, please consider to cite:

@misc{chen2021vitvnet,
title={ViT-V-Net: Vision Transformer for Unsupervised Volumetric Medical Image Registration}, 
author={Junyu Chen and Yufan He and Eric C. Frey and Ye Li and Yong Du},
year={2021},
eprint={2104.06468},
archivePrefix={arXiv},
primaryClass={eess.IV}
}

About Me

Comments
  • The process of making and loading datasets

    The process of making and loading datasets

    Hi,

    I'm sorry to bother you. I want to run this architecture on my own datasets, but I ran into some difficulties. Prior to this, I had read the issue and code, but could not solve the problem independently, so I would like to ask the following questions:

    • What structure does ".pkl" use to encapsulate datasets?

    • Further, how to wrap the "nii/niI.gz" file into the above ".pkl" according to the desired structure? I have the preprocessed datasets of LPBA40(".niI.gz" format, a total of 40 MRI images and their corresponding Ground truth), but I do not know how to load these data into correct "pkl" format files. I tried to find the relevant information, but did not understand and implement it. I hope to get your instructions, thank you!

    image image

    In view of these two questions, I made the following reflections combined with issues opened by others, but failed to solve these two problems independently, I will describe my ideas so that you can quickly clarify my questions, My partial understanding and guess are as follows:

    • For data encapsulation of medical images, a pre-processed MRI image and its corresponding segmentation result (label/ground truth) are encapsulated into an independent "***.pkl" file for data reading. PKL = pkload("***.pkl"); "data = pkload("***.pkl")"; Where 2 means 1(preprocessed MRI image)+1(segmentation result, i.e., ground truth), that is, data[0] is preprocessed MRI image and data[1] is its corresponding ground truth; But according to the code of datasets.py, this does not seem to be the case, because in "class JHUBrainDataset(Dataset)" getitem is "x, y = pkload(path)", In "class JHUBrainInferDataset(Dataset)", the getitem mode is "x, y, x_seg, y_seg = pkload(path)", It looks like a "***.pkl" file containing two MRI images and their corresponding two ground truths.

    • According to the code part of reading the dataset in the training stage and validation stage of "datasets.py", I speculated as follows: In the training stage, only preprocessed MRI images were used, without ground truth, because there are no "x_seg" and "y_seg" in "__getitem__" of "class JHUBrainDataset" ; In the validation stage, pre-processed MRI images and their corresponding segmentation results (label/ground truth) were used at the same time.

    image image image image

    Thanks

    opened by tzayuan 17
  • The process of making data

    The process of making data

    Hi！ I'm sorry to bother you. I followed your suggestion to convert the file to pkl format, but it still doesn't work properly. Could you please help me see where I went wrong? image image Train image Val image Very grateful for your help

    opened by wenqian11 2
  • How do you setting your data in order to run python train.py?

    How do you setting your data in order to run python train.py?

    Please help me setting the data in order to train. Or help me have the simply data to train ( I also download your datas that mentioned but I can not use it) image I assumer use the BraTS_2018 dataset. It looks like: image image

    Val: image image

    When I set the above path, the following error occurs: image

    Thank you, very much.

    opened by tphankr 2
  • How was the visualization results visualized?

    How was the visualization results visualized?

    Hi, I really wonder how the registration results were visualized? Especially for the deformed grid, many papers talked about registration but no one even mentioned how the results were visualized.

    opened by chengjiawang 1
  • How to use the pretrained model?

    How to use the pretrained model?

    I'm glad to find that you have open-sourced your pretrained model, I got it from google drive, but how can we just apply it to infer.py? What do the files in the dicterory mean in your pretrained model? Thanks for your time and reply. 4O} X_L4V34S{D11_WD1HJQ

    opened by Dududu233 1
Owner
Junyu Chen
Ph.D. student in the Department of Electrical and Computer Engineering & the Department of Radiology and Radiological Science @ Johns Hopkins University
Junyu Chen
GitHub
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Phil Wang 12.6k Jan 9, 2023
Registration Loss Learning for Deep Probabilistic Point Set Registration

RLLReg This repository contains a Pytorch implementation of the point set registration method RLLReg. Details about the method can be found in the 3DV

Felix Järemo Lawin 35 Nov 2, 2022
This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

vision-transformer-from-scratch This repository includes several kinds of vision transformers from scratch so that one beginner can understand the the

null 1 Dec 24, 2021
Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding

Vision Longformer This project provides the source code for the vision longformer paper. Multi-Scale Vision Longformer: A New Vision Transformer for H

Microsoft 209 Dec 30, 2022
The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

MIC-DKFZ 1.2k Jan 4, 2023
Build a medical knowledge graph based on Unified Language Medical System (UMLS)

UMLS-Graph Build a medical knowledge graph based on Unified Language Medical System (UMLS) Requisite Install MySQL Server 5.6 and import UMLS data int

Donghua Chen 6 Dec 25, 2022
Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

ImageProcessingTransformer Third party Pytorch implement of Image Processing Transformer (Pre-Trained Image Processing Transformer arXiv:2012.00364v2)

null 61 Jan 1, 2023
CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

CoTr: Efficient 3D Medical Image Segmentation by bridging CNN and Transformer This is the official pytorch implementation of the CoTr: Paper: CoTr: Ef

null 218 Dec 25, 2022
The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"

Swin-Unet The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"(https://arxiv.org/abs/2105.05537). A validatio

null 869 Jan 7, 2023
MISSFormer: An Effective Medical Image Segmentation Transformer

MISSFormer Code for paper "MISSFormer: An Effective Medical Image Segmentation Transformer". Please read our preprint at the following link: paper_add

Fong 22 Dec 24, 2022
Deep learning image registration library for PyTorch

TorchIR: Pytorch Image Registration TorchIR is a image registration library for deep learning image registration (DLIR). I have integrated several ide

Bob de Vos 40 Dec 16, 2022
GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Geometric Transformer for Fast and Robust Point Cloud Registration PyTorch imple

Zheng Qin 220 Jan 5, 2023
This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

Swin Transformer 1.4k Dec 30, 2022
Episodic Transformer (E.T.) is a novel attention-based architecture for vision-and-language navigation. E.T. is based on a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions.

Episodic Transformers (E.T.) Episodic Transformer for Vision-and-Language Navigation Alexander Pashevich, Cordelia Schmid, Chen Sun Episodic Transform

Alex Pashevich 62 Dec 24, 2022
The implementation of "Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer"

Shuffle Transformer The implementation of "Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer" Introduction Very recently, window-

null 87 Nov 29, 2022
Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Swin-Transformer-Tensorflow A direct translation of the official PyTorch implementation of "Swin Transformer: Hierarchical Vision Transformer using Sh

null 52 Dec 29, 2022
CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped

CSWin-Transformer This repo is the official implementation of "CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows". Th

Microsoft 409 Jan 6, 2023
Breaking the Dilemma of Medical Image-to-image Translation

Breaking the Dilemma of Medical Image-to-image Translation Supervised Pix2Pix and unsupervised Cycle-consistency are two modes that dominate the field

Kid Liet 86 Dec 21, 2022
Copy Paste positive polyp using poisson image blending for medical image segmentation

Copy Paste positive polyp using poisson image blending for medical image segmentation According poisson image blending I've completely used it for bio

Phạm Vũ Hùng 2 Oct 19, 2021