Deep Learning to Improve Breast Cancer Detection on Screening Mammography

Li Shen

Last update: Jan 3, 2023

Related tags

Overview

Shield:

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

Deep Learning to Improve Breast Cancer Detection on Screening Mammography (End-to-end Training for Whole Image Breast Cancer Screening using An All Convolutional Design)

Li Shen, Ph.D. CS

Icahn School of Medicine at Mount Sinai

New York, New York, USA

Introduction

This is the companion site for our paper that was originally titled "End-to-end Training for Whole Image Breast Cancer Diagnosis using An All Convolutional Design" and was retitled as "Deep Learning to Improve Breast Cancer Detection on Screening Mammography". The paper has been published here. You may also find the arXiv version here. This work was initially presented at the NIPS17 workshop on machine learning for health. Access the 4-page short paper here. Download the poster.

For our entry in the DREAM2016 Digital Mammography challenge, see this write-up. This work is much improved from our method used in the challenge.

Whole image model downloads

A few best whole image models are available for downloading at this Google Drive folder. YaroslavNet is the DM challenge top-performing team's method. Here is a table for individual downloads:

Database	Patch Classifier	Top Layers (two blocks)	Single AUC	Augmented AUC	Link
DDSM	Resnet50	[512-512-1024]x2	0.86	0.88	download
DDSM	VGG16	512x1	0.83	0.86	download
DDSM	VGG16	[512-512-1024]x2	0.85	0.88	download
DDSM	YaroslavNet	heatmap + max pooling + FC16-8 + shortcut	0.83	0.86	download
INbreast	VGG16	512x1	0.92	0.94	download
INbreast	VGG16	[512-512-1024]x2	0.95	0.96	download

Inference level augmentation is obtained by horizontal and vertical flips to generate 4 predictions.
The listed scores are single model AUC and prediction averaged AUC.
3 Model averaging on DDSM gives AUC of 0.91
2 Model averaging on INbreast gives AUC of 0.96.

Patch classifier model downloads

Several patch classifier models (i.e. patch state) are also available for downloading at this Google Drive folder. Here is a table for individual download:

Model	Train Set	Accuracy	Link
Resnet50	S10	0.89	download
VGG16	S10	0.84	download
VGG19	S10	0.79	download
YaroslavNet (Final)	S10	0.89	download
Resnet50	S30	0.91	download
VGG16	S30	0.86	download
VGG19	S30	0.89	download

With patch classifier models, you can convert them into any whole image classifier by adding convolutional, FC and heatmap layers on top and see for yourself.

A bit explanation of this repository's file structure

The .py files under the root directory are Python modules to be imported.
You shall set the PYTHONPATH variable like this: export PYTHONPATH=$PYTHONPATH:your_path_to_repos/end2end-all-conv so that the Python modules can be imported.
The code for patch sampling, patch classifier and whole image training are under the ddsm_train folder.
sample_patches_combined.py is used to sample patches from images and masks.
patch_clf_train.py is used to train a patch classifier.
image_clf_train.py is used to train a whole image classifier, either on top of a patch classifier or from another already trained whole image classifier (i.e. finetuning).
There are multiple shell scripts under the ddsm_train folder to serve as examples.

Some input files' format

I've got a lot of requests asking about the format of some input files. Here I provide the first few lines and hope they can be helpful:

roi_mask_path.csv

patient_id,side,view,abn_num,pathology,type
P_00005,RIGHT,CC,1,MALIGNANT,calc
P_00005,RIGHT,MLO,1,MALIGNANT,calc
P_00007,LEFT,CC,1,BENIGN,calc
P_00007,LEFT,MLO,1,BENIGN,calc
P_00008,LEFT,CC,1,BENIGN_WITHOUT_CALLBACK,calc

pat_train.txt

P_00601
P_00413
P_01163
P_00101
P_01122

Transfer learning is as easy as 1-2-3

In order to transfer a model to your own data, follow these easy steps.

Determine the rescale factor

The rescale factor is used to rescale the pixel intensities so that the max value is 255. For PNG format, the max value is 65535, so the rescale factor is 255/65535 = 0.003891. If your images are already in the 255 scale, set rescale factor to 1.

Calculate the pixel-wise mean

This is simply the mean pixel intensity of your train set images.

Image size

This is currently fixed at 1152x896 for the models in this study. However, you can change the image size when converting from a patch classifier to a whole image classifier.

Finetune

Now you can finetune a model on your own data for cancer predictions! You may check out this shell script. Alternatively, copy & paste from here:

TRAIN_DIR="INbreast/train"
VAL_DIR="INbreast/val"
TEST_DIR="INbreast/test"
RESUME_FROM="ddsm_vgg16_s10_[512-512-1024]x2_hybrid.h5"
BEST_MODEL="INbreast/transferred_inbreast_best_model.h5"
FINAL_MODEL="NOSAVE"
export NUM_CPU_CORES=4

python image_clf_train.py \
    --no-patch-model-state \
    --resume-from $RESUME_FROM \
    --img-size 1152 896 \
    --no-img-scale \
    --rescale-factor 0.003891 \
    --featurewise-center \
    --featurewise-mean 44.33 \
    --no-equalize-hist \
    --batch-size 4 \
    --train-bs-multiplier 0.5 \
    --augmentation \
    --class-list neg pos \
    --nb-epoch 0 \
    --all-layer-epochs 50 \
    --load-val-ram \
    --load-train-ram \
    --optimizer adam \
    --weight-decay 0.001 \
    --hidden-dropout 0.0 \
    --weight-decay2 0.01 \
    --hidden-dropout2 0.0 \
    --init-learningrate 0.0001 \
    --all-layer-multiplier 0.01 \
    --es-patience 10 \
    --auto-batch-balance \
    --best-model $BEST_MODEL \
    --final-model $FINAL_MODEL \
    $TRAIN_DIR $VAL_DIR $TEST_DIR

Some explanations of the arguments:

The batch size for training is the product of --batch-size and --train-bs-multiplier. Because training uses roughtly twice (both forward and back props) the GPU memory of testing, --train-bs-multiplier is set to 0.5 here.
For model finetuning, only the second stage of the two-stage training is used here. So --nb-epoch is set to 0.
--load-val-ram and --load-train-ram will load the image data from the validation and train sets into memory. You may want to turn off these options if you don't have sufficient memory. When turned off, out-of-core training will be used.
--weight-decay and --hidden-dropout are for stage 1. --weight-decay2 and --hidden-dropout2 are for stage 2.
The learning rate for stage 1 is --init-learningrate. The learning rate for stage 2 is the product of --init-learningrate and --all-layer-multiplier.

Computational environment

The research in this study is carried out on a Linux workstation with 8 CPU cores and a single NVIDIA Quadro M4000 GPU with 8GB memory. The deep learning framework is Keras 2 with Tensorflow as the backend.

About Keras version

It is known that Keras >= 2.1.0 can give errors due an API change. See issue #7. Use Keras with version < 2.1.0. For example, Keras=2.0.8 is known to work.

TERMS OF USE

All data is free to use for non-commercial purposes. For commercial use please contact MSIP.

Comments

How do I find ".csv" file?

Hello~lishen, I have download the CBIS-DDSM dataset from The Cancer Imaging Archive, it's show in the picture: But it havn't the ".csv" file，does this file need to be created by ourselves? But we just know the information about the patient, side, CC or MLO, there's no more information. As shown in the above picture, the test and train sets are scattered throughout the folder, and they contain many subfolders, I don't know how to set the parameter of "train_dir", "test_dir", "val_dir".

So~I have two requests: 1:Would you please provide the ".csv" file? 2:Would you tell us, does we need to sort out the data by ourselves and integrate all the images into the "test_set", "train_set", "val_set" three folders?

Thank you for your time, best wishes for you!

opened by wurenzhong 16
Too many values to unpack in calculation step

Hey, thank you for this work. I have been trying to reproduce the results with Calc_Test and Mass_Test but at the calculation step I keep getting the following error. Can you suggest what I might be doing wrong here.

opened by nerdykamil 6
Our miccai 2017 paper also works for the whole mammogram classification.

Hi Shen,

I got your email for your paper. Our MICCAI 2017 paper proposes several schemes for whole mammogram classification.

Zhu, Wentao, Qi Lou, Yeeleng Scott Vang, and Xiaohui Xie. "Deep multi-instance networks with sparse label assignment for whole mammogram classification." MICCAI (2017).

Thanks! Wentao

opened by wentaozhu 5
What's the order of the patch classifier predictions?

Is it noted in the paper, that the patch classifiers are all trained to predict the following classes: background, malignant mass, benign mass, malignant calcification and benign calcification. It seem consistent with the model prediction outputs of the patch classifiers in the Google Drive, which has a size of 5 output classes.

But how do we check which class each of the patch model outputs are associated with? I.e. what is the ordering of the outputs.

opened by frfa1 4
ValueError when model testing

I encountered the following error ValueError: not enough values to unpack (expected 3, got 2) when accessing
index_array, current_index, current_batch_size = next(self.index_generator) in line 1203 in dm_image.py

This happens in both fine tuning phase and model testing phase (as executing in "Example_model_test.ipynb" notebook inside ddsm_train folder)

Is there anyone also encountered this issue and resolved it?

opened by genuinwix 1
Good comment

Hi Li,

Well done on your Ph.D. thesis! I skimmed through your paper and saw it is very similar to a friend of mine's (@Adamouization) Master's dissertation.

Just a quick comment to say good luck for the future and well done!

opened by Anri-Lombard 0
predict result means?

i use the ddsm_vgg16_s10_[512-512-1024]x2_hybrid.h5 model to predict my own data,the result just like this,i want to know which one is the benign probability? 0.2 or 0.8?

[[0.2 0.8]]

opened by picEmily 0
Images used?

This is a great repo! I'm wondering if you have the images you used. I loaded one of the models and during inference I scale my new images to be 0-255 then shift by -44.33 (mean during training according to https://git.io/JeO7x). But i'm not sure its working as well as it should, so it would be easier if I can probe it with images used during training. Thanks!

opened by aliabd 0
Unclear Pretrained steps

In the publication https://www.nature.com/articles/s41598-019-48995-4 you mentioned the 3-stage training strategy ment top layer number is set to "46" for Resnet50.

I saw not documentation of this here. By 46, are you reffering to 'res3a_branch1' (deduced from net.layers[46].name)? What criteria was used to settle on this layer?

tnx.

opened by GNovich 0
Image preprocessing - convert to PNG, downscale

Hi, would it be possible to receive some code that performs the image pre-processing? I'm aware some of it is using ImageMagick via the Linux command line, but I cannot correctly sort this at present. edit: I am trying to use the CBIS-DDSM dataset in the same way, using the patch classifier. Not necessarily the same splits; just the pre-processing. Kind regards.

opened by priley95 1
Too Large Dataset

Hi all,

Is there anyone who is having only small set from the original dataset?? (Because original dataset is about 163 GB of size)

I just need small set of data from that dataset.

Thanks

opened by genuinwix 0
Preprocessed dataset

Hi, thanks to your great work. have you put your pre-processed dataset of DDSM in google drive ? I will be very happy if you help me to preprocess CBIS-DDSM dataset. thanks a lot.

opened by MahdiHeybati 1

Patch Classifier testing results in 0.25 AUC

Hello Shen, first congratulations on the great effort. I have tested two of your patch models on 400 jpg patches from CBIS-DDSM patches but I have gained an accuracy of 0.25 I have 100 images per class (calcification benign - calcification malignant - mass benign - mass malignant), so I was wondering what is the wrong that I am doing in the prediction?

test_generator = test_imgen.flow_from_directory(
                                                     directory=test_dir,
                                                     target_size=(500, 500),
                                                     color_mode="rgb",
                                                     batch_size=1,
                                                     class_mode=None,
                                                     shuffle=False)
test_generator.reset()
model = load_model("s30_resnet50.h5")
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])
pred=model.predict_generator(test_generator,verbose=1,steps=nb_test_samples)
predicted_class_indices=np.argmax(pred,axis=1)```

opened by MohammedAdelFahmi 0

Owner

Li Shen

I'm an academic researcher with many years of experience developing machine learning algorithms and bioinformatic software and analyzing genomic data.

GitHub

Patient-Survival - Using Python, I developed a Machine Learning model using classification techniques such as Random Forest and SVM classifiers to predict a patient's survival status that have undergone breast cancer surgery.

Patient-Survival - Using Python, I developed a Machine Learning model using classification techniques such as Random Forest and SVM classifiers to predict a patient's survival status that have undergone breast cancer surgery.

1 Dec 28, 2021

Predict Breast Cancer Wisconsin (Diagnostic) using Naive Bayes

Naive-Bayes Predict Breast Cancer Wisconsin (Diagnostic) using Naive Bayes Downloading Data Set Use our Breast Cancer Wisconsin Data Set Also you can

0 Apr 6, 2022

The official repository for our paper "The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers". We significantly improve the systematic generalization of transformer models on a variety of datasets using simple tricks and careful considerations.

Codebase for training transformers on systematic generalization datasets. The official repository for our EMNLP 2021 paper The Devil is in the Detail:

57 Nov 21, 2022

Fast and scalable uncertainty quantification for neural molecular property prediction, accelerated optimization, and guided virtual screening.

Evidential Deep Learning for Guided Molecular Property Prediction and Discovery Ava Soleimany*, Alexander Amini*, Samuel Goldman*, Daniela Rus, Sangee

75 Dec 15, 2022

Cancer metastasis detection with neural conditional random field (NCRF)

NCRF Prerequisites Data Whole slide images Annotations Patch images Model Training Testing Tissue mask Probability map Tumor localization FROC evaluat

731 Jan 1, 2023

Cancer Drug Response Prediction via a Hybrid Graph Convolutional Network

DeepCDR Cancer Drug Response Prediction via a Hybrid Graph Convolutional Network This work has been accepted to ECCB2020 and was also published in the

50 Dec 18, 2022

To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types

To Design and Implement Logistic Regression to Classify Between Benign and Malignant Cancer Types, from a Database Taken From Dr. Wolberg reports his Clinic Cases.

1 Jul 31, 2022

SCI-AIDE : High-fidelity Few-shot Histopathology Image Synthesis for Rare Cancer Diagnosis

SCI-AIDE : High-fidelity Few-shot Histopathology Image Synthesis for Rare Cancer Diagnosis Pretrained Models In this work, we created synthetic tissue

1 Feb 7, 2022

Keep CALM and Improve Visual Feature Attribution

Keep CALM and Improve Visual Feature Attribution Jae Myung Kim1*, Junsuk Choe1*, Zeynep Akata2, Seong Joon Oh1† * Equal contribution † Corresponding a

90 Dec 7, 2022

Official implementation of "Open-set Label Noise Can Improve Robustness Against Inherent Label Noise" (NeurIPS 2021)

Open-set Label Noise Can Improve Robustness Against Inherent Label Noise NeurIPS 2021: This repository is the official implementation of ODNL. Require

12 Dec 7, 2022

Use graph-based analysis to re-classify stocks and to improve Markowitz portfolio optimization

Dynamic Stock Industrial Classification Use graph-based analysis to re-classify stocks and experiment different re-classification methodologies to imp

10 Dec 5, 2022

SmallInitEmb - LayerNorm(SmallInit(Embedding)) in a Transformer to improve convergence

SmallInitEmb LayerNorm(SmallInit(Embedding)) in a Transformer I find that when t

11 Dec 25, 2022

Sub-tomogram-Detection - Deep learning based model for Cyro ET Sub-tomogram-Detection

Deep learning based model for Cyro ET Sub-tomogram-Detection High degree of stru

2 Feb 4, 2022

MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

MOT-Tracking-by-Detection-Pipeline Tracking-by-Detection形式のMOT(Multi Object Trac

41 Nov 23, 2022

Ivy is a templated deep learning framework which maximizes the portability of deep learning codebases.

Ivy is a templated deep learning framework which maximizes the portability of deep learning codebases. Ivy wraps the functional APIs of existing frameworks. Framework-agnostic functions, libraries and layers can then be written using Ivy, with simultaneous support for all frameworks. Ivy currently supports Jax, TensorFlow, PyTorch, MXNet and Numpy. Check out the docs for more info!

8.2k Jan 2, 2023

Deep learning (neural network) based remote photoplethysmography: how to extract pulse signal from video using deep learning tools

Deep-rPPG: Camera-based pulse estimation using deep learning tools Deep learning (neural network) based remote photoplethysmography: how to extract pu