Unsupervised Feature Loss (UFLoss) for High Fidelity Deep learning (DL)-based reconstruction

Related tags

Deep Learning UFLoss
Overview

Unsupervised Feature Loss (UFLoss) for High Fidelity Deep learning (DL)-based reconstruction

Official github repository for the paper High Fidelity Deep Learning-based MRI Reconstruction with Instance-wise Discriminative Feature Matching Loss. In this work, a novel patch-based Unsupervised Feature loss (UFLoss) is proposed and incorporated into the training of DL-based reconstruction frameworks in order to preserve perceptual similarity and high-order statistics. In-vivo experiments indicate that adding the UFLoss encourages sharper edges with higher overall image quality under DL-based reconstruction framework. Our implementations are in PyTorch

Installation

To use this package, install the required python packages (tested with python 3.8 on Ubuntu 20.04 LTS):

pip install -r requirements.txt

Dataset

We used a subset of FastMRI knee dataset for the training and evaluation. We used E-SPIRiT to pre-compute sensitivity maps using BART. Post-processed data (including Sens Maps, Coil combined images) and pre-trained model can be requested by emailing [email protected].

Update We provide our data-preprocessing code at UFloss_training/data_preprocessing.py. This script computes the sensitivity maps and performs data normalization and coil combination. BART toolbox is required for computing the sensitivity maps. Follow the installation instructions on the website and add the following lines to your .bashrc file.

/python/" export PATH=" :$PATH"">
export PYTHONPATH="${PYTHONPATH}:
    
     /python/
     "
    
export PATH="
    
     :
     $PATH
     "
    

To run the data-preprocessing code, download and unzip the fastMRI Multi-coil knee dataset. Simplu run

python data_preprocessing.py -l <path to your fastMRI multi-coil dataset> -t <target directory> -c <size for your E-SPIRiT calibration region>

Step 0: Patch Extraction

To extract patches from the fully-smapled training data, go to the UFloss_training/ folder and run patch_extraction.py to extract patches. Please specify the directories of the training dataset and the target folder. Instructions are avaible by runing:

python patch_extraction.py -h

Step 1: Train the UFLoss feature mapping network

To train the UFLoss feature mapping network, go to the UFloss_training/ folder and run patch_learning.py. We provide a demo training script to perform the training on fully-sampled patches:

bash launch_training_patch_learning.sh

Visualiztion (Patch retrival results, shown below) script will be available soon.

Step 2: Train the DL-based reconstruction with UFLoss

To train the DL-based reconstruction with UFLoss, we provide our source code here at DL_Recon_UFLoss/. We adoped MoDL as our DL-based reconstruction network. We provide training scripts for MoDL with and without UFLoss at DL_Recon_UFLoss/models/unrolled2D/scripts:

bash launch_training_MoDL_traditional_UFLoss_256_demo.sh

You can easily paly around with the parameters by editing the training script. One representative reconstruction results is shown as below.

Perform inference with the trained model

To perform the inference reconstruction on the testing set, we provide an inference script at DL_Recon_UFLoss/models/unrolled2D/inference_ufloss.py. run the following command for inference:

python inference_ufloss.py --data-path <Path to the dataset> 
                        --device-num <Which device to train on>
                        --exp-dir <Path where the results should be saved>
                        --checkpoint <Path to an existing checkpoint>

Acknoledgements

Reconstruction code borrows heavily from fastMRI Github repo and DL-ESPIRiT by Christopher Sandino. This work is a colaboration between UC Berkeley and GE Healthcare. Please contact [email protected] if you have any questions.

Citation

If you find this code useful for your research, please consider citing our paper High Fidelity Deep Learning-based MRI Reconstruction with Instance-wise Discriminative Feature Matching Loss:

@article{wang2021high,
  title={High Fidelity Deep Learning-based MRI Reconstruction with Instance-wise Discriminative Feature Matching Loss},
  author={Wang, Ke and Tamir, Jonathan I and De Goyeneche, Alfredo and Wollner, Uri and Brada, Rafi and Yu, Stella and Lustig, Michael},
  journal={arXiv preprint arXiv:2108.12460},
  year={2021}
}
Comments
  • Pre-processed Data

    Pre-processed Data

    I have got some difficulties in pre-calculating the sensitive map, when runnning the code Get-patch. And I want to know how you deal with the raw data(Download from fastmri.org) , using ESPIRiT and BART. It would be better if you can email me the pre-trained model or processed data. There is my email [email protected]

    opened by s77coolest 2
  • Training time and GPU

    Training time and GPU

    I got a question about the training time is too long and the GPU memory is too high. I use GTX 3090 24G, but the batch size I can set just 1, if setting 2 I'll get cuda out of memory. So I wanna ask to how I can speed up my training time. About the GPU memory, whether the input image can be blocked and then entered to increase the batch size. Can you share some details about training? Thank you so much.

    opened by Aristot1e 1
  • Training problem

    Training problem

    when running the scripts "launch_training_MoDL_traditional_UFLoss_256_demo.sh" , which type of the dir_data should be uesd? it is the data had been processed (1_0.h5) or the raw data I downloaded from "fastMRI.org" (file100001.h5)?

    opened by s77coolest 0
  • Training problem

    Training problem

    when running the scripts "launch_training_MoDL_traditional_UFLoss_256_demo.sh" , which type of the dir_data should be uesd? it is the data had been processed (1_0.h5) or the raw data I downloaded from "fastMRI.org" (file100001.h5)?

    opened by s77coolest 0
  • Train the DL-based reconstruction with UFLoss

    Train the DL-based reconstruction with UFLoss

    Traceback (most recent call last): File "../train_ufloss.py", line 803, in main(args) File "../train_ufloss.py", line 562, in main model_re.load_state_dict( File "/home/img/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1223, in load_state_dict raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for Model: size mismatch for memory_bank: copying a param with shape torch.Size([256, 1]) from checkpoint, the shape in current model is torch.Size([256, 2457600]).

    This is the error when running launch_training_MoDL_traditional_UFLoss_256_demo.sh. The model shape is not corresponding, so why? I can’t deal with it. And the other problem is in the file train_ufloss.py in line 193/194. if args.loss_normalized == False: output = output * std + mean target = target * std + mean Both the std and mean are not defined. What should I do?

    opened by Aristot1e 9
  • preprocessing single coil data

    preprocessing single coil data

    There are some problems when I run data_preprocessing.py using single-coil data. The program will report an error : (Calibration region not found!)

    /home/img/anaconda3/bin/python /home/img/Desktop/lff/UFLoss-main/UFloss_training/data_preprocessing.py fatal: Needed a single revision cuda:0 the last dimension 372, coil dimensional: 640: 0%| | 0/199 [00:00<?, ?it/s]ERROR: Calibration region not found! Aborted (core dumped) the last dimension 372, coil dimensional: 640: 0%| | 0/199 [06:34<?, ?it/s] Traceback (most recent call last): File "/home/img/Desktop/lff/UFLoss-main/UFloss_training/data_preprocessing.py", line 82, in sens = bart.bart( File "/home/img/Desktop/lff/UFLoss-main/UFloss_training/bart_/bart.py", line 94, in bart raise Exception("Command exited with an error.") Exception: Command exited with an error.

    Process finished with exit code 1

    I don’t know why ERROR:Calibration region not found and how to deal with it, so, please. And I have another request if it's possible, may I get the code that you have trained. This will be of great help to my study. This is my email [email protected] Thanks for the help.

    opened by Aristot1e 1
Owner
null
Deep generative modeling for time-stamped heterogeneous data, enabling high-fidelity models for a large variety of spatio-temporal domains.

Neural Spatio-Temporal Point Processes [arxiv] Ricky T. Q. Chen, Brandon Amos, Maximilian Nickel Abstract. We propose a new class of parameterizations

Facebook Research 75 Dec 19, 2022
pytorch implementation of "Contrastive Multiview Coding", "Momentum Contrast for Unsupervised Visual Representation Learning", and "Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination"

Unofficial implementation: MoCo: Momentum Contrast for Unsupervised Visual Representation Learning (Paper) InsDis: Unsupervised Feature Learning via N

Zhiqiang Shen 16 Nov 4, 2020
Tensorflow python implementation of "Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos"

Learning High Fidelity Depths of Dressed Humans by Watching Social Media Dance Videos This repository is the official tensorflow python implementation

Yasamin Jafarian 287 Jan 6, 2023
Implement of "Training deep neural networks via direct loss minimization" in PyTorch for 0-1 loss

This is the implementation of "Training deep neural networks via direct loss minimization" published at ICML 2016 in PyTorch. The implementation targe

Cuong Nguyen 1 Jan 18, 2022
Deep Unsupervised 3D SfM Face Reconstruction Based on Massive Landmark Bundle Adjustment.

(ACMMM 2021 Oral) SfM Face Reconstruction Based on Massive Landmark Bundle Adjustment This repository shows two tasks: Face landmark detection and Fac

BoomStar 51 Dec 13, 2022
Deep Unsupervised 3D SfM Face Reconstruction Based on Massive Landmark Bundle Adjustment.

(ACMMM 2021 Oral) SfM Face Reconstruction Based on Massive Landmark Bundle Adjustment This repository shows two tasks: Face landmark detection and Fac

BoomStar 51 Dec 13, 2022
《Towards High Fidelity Face Relighting with Realistic Shadows》(CVPR 2021)

Towards High Fidelity Face-Relighting with Realistic Shadows Andrew Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu. In CVPR, 2021. T

null 114 Dec 10, 2022
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis Jungil Kong, Jaehyeon Kim, Jaekyoung Bae In our paper, we p

Rishikesh (ऋषिकेश) 31 Dec 8, 2022
UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation. Training python train.py --c

Rishikesh (ऋषिकेश) 55 Dec 26, 2022
Unofficial PyTorch Implementation of UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation This is an unofficial PyTorch

MINDs Lab 170 Jan 4, 2023
Unofficial PyTorch Implementation of UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

UnivNet UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation This is an unofficial PyTorch

MINDs Lab 54 Aug 30, 2021
A two-stage U-Net for high-fidelity denoising of historical recordings

A two-stage U-Net for high-fidelity denoising of historical recordings Official repository of the paper (not submitted yet): E. Moliner and V. Välimäk

Eloi Moliner Juanpere 57 Jan 5, 2023
Implementation for HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

HFGI: High-Fidelity GAN Inversion for Image Attribute Editing High-Fidelity GAN Inversion for Image Attribute Editing Update: We released the inferenc

Tengfei Wang 371 Dec 30, 2022
SCI-AIDE : High-fidelity Few-shot Histopathology Image Synthesis for Rare Cancer Diagnosis

SCI-AIDE : High-fidelity Few-shot Histopathology Image Synthesis for Rare Cancer Diagnosis Pretrained Models In this work, we created synthetic tissue

Emirhan Kurtuluş 1 Feb 7, 2022
PyTorch Implementation of DiffGAN-TTS: High-Fidelity and Efficient Text-to-Speech with Denoising Diffusion GANs

DiffGAN-TTS - PyTorch Implementation PyTorch implementation of DiffGAN-TTS: High

Keon Lee 157 Jan 1, 2023
Parallel and High-Fidelity Text-to-Lip Generation; AAAI 2022 ; Official code

Parallel and High-Fidelity Text-to-Lip Generation This repository is the official PyTorch implementation of our AAAI-2022 paper, in which we propose P

Zhying 77 Dec 21, 2022
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

null 117 Dec 28, 2022
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
An implementation for the loss function proposed in Decoupled Contrastive Loss paper.

Decoupled-Contrastive-Learning This repository is an implementation for the loss function proposed in Decoupled Contrastive Loss paper. Requirements P

Ramin Nakhli 71 Dec 4, 2022