ULMFiT for Genomic Sequence Data

Karl

Last update: Dec 12, 2022

Related tags

Deep Learning Genomic-ULMFiT

Overview

Genomic ULMFiT

This is an implementation of ULMFiT for genomics classification using Pytorch and Fastai. The model architecture used is based on the AWD-LSTM model, consisting of an embedding, three LSTM layers, and a final set of linear layers.

The ULMFiT approach uses three training phases to produce a classification model:

Train a language model on a large, unlabeled corpus
Fine tune the language model on the classification corpus
Use the fine tuned language model to initialize a classification model

This method is particularly advantageous for genomic data, where large amounts of unlabeled data is abundant and labeled data is scarce. The ULMFiT approach allows us to train a model on a large, unlabeled genomic corpus in an unsupervised fashion. The pre-trained language model serves as a feature extractor for parsing genomic data.

Typical deep learning approaches to genomics classification are highly restricted to whatever labeled data is available. Models are usually trained from scratch on small datasets, leading to problems with overfitting. When unsupervised pre-training is used, it is typically done only on the classification dataset or on synthetically generated data. The Genomic-ULMFiT approach uses genome scale corpuses for pre-training to produce better feature extractors than we would get by training only on the classification corpus.

For a deep dive into the ULMFiT approach, model architectures, regularization and training strategies, see the Methods Long Form document in the Methods section.

Results

Performance of Genomic-ULMFiT relative to other methods

Promoter Classification

E. coli promoters

The Genomic-ULMFiT method performs well at the task of classifying promoter sequences from random sections of the genome. The process of unsupervised pre-training and fine-tuning has a clear impact on the performance of the classification model

Model	Accuracy	Precision	Recall	Correlation Coefficient
Naive	0.834	0.847	0.816	0.670
E. coli Genome Pre-Training	0.919	0.941	0.893	0.839
Genomic Ensemble Pre-Training	0.973	0.980	0.966	0.947

Data generation described in notebook

Notebook Directory

Classification performance on human promoters is competitive with published results

Human Promoters (short)

For the short promoter sequences, using data from Recognition of Prokaryotic and Eukaryotic Promoters using Convolutional Deep Learning Neural Networks:

Model	DNA Size	kmer/stride	Accuracy	Precision	Recall	Correlation Coefficient	Specificity
Kh et al.	-200/50	-	-	-	0.9	0.89	0.98
Naive Model	-200/50	5/2	0.80	0.74	0.80	0.59	0.80
With Pre-Training	-200/50	5/2	0.922	0.963	0.849	0.844	0.976
With Pre-Training and Fine Tuning	-200/50	5/2	.977	.959	.989	.955	.969
With Pre-Training and Fine Tuning	-200/50	5/1	.990	.983	.995	.981	.987
With Pre-Training and Fine Tuning	-200/50	3/1	.995	.992	.996	.991	.994

Data Source

Notebook Directory

Human Promoters (long)

For the long promoter sequences, using data from PromID: Human Promoter Prediction by Deep Learning:

Model	DNA Size	Models	Accuracy	Precision	Recall	Correlation Coefficient
Umarov et al.	-1000/500	2 Model Ensemble	-	0.636	0.802	0.714
Umarov et al.	-200/400	2 Model Ensemble	-	0.769	0.755	0.762
Naive Model	-500/500	Single Model	0.858	0.877	0.772	0.708
With Pre-Training	-500/500	Single Model	0.888	0.90	0.824	0.770
With Pre-Training and Fine Tuning	-500/500	Single Model	0.892	0.877	0.865	0.778

Data generation described in notebook

Notebook Directory

Other Bacterial Promoters

This table shows results on data from Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. These results show how CNN based methods can sometimes perform better when training on small datasets.

Method	Organism	Training Examples	Accuracy	Precision	Recall	Correlation Coefficient	Specificity
Kh et al.	E. coli	2936	-	-	0.90	0.84	0.96
Genomic-ULMFiT	E. coli	2936	0.956	0.917	0.880	0.871	0.977
Kh et al.	B. subtilis	1050	-	-	0.91	0.86	0.95
Genomic-ULMFiT	B. subtilis	1050	0.905	0.857	0.789	0.759	0.95

Data Source

Notebook Directory

Metaganomics Classification

Genomic-ULMFiT shows improved performance on the metagenomics taxonomic dataset from Deep learning models for bacteria taxonomic classification of metagenomic data.

Method	Data Source	Accuracy	Precision	Recall	F1
Fiannaca et al.	Amplicon	.9137	.9162	.9137	.9126
Genomic-ULMFiT	Amplicon	.9239	.9402	.9332	.9306
Fiannaca et al.	Shotgun	.8550	.8570	.8520	.8511
Genomic-ULMFiT	Shotgun	.8797	.8824	.8769	.8758

Data Source

Notebook Directory

Enhancer Classification

When trained on a dataset of mammalian enhancer sequences from Enhancer Identification using Transfer and Adversarial Deep Learning of DNA Sequences, Genomic_ULMFiT improves on results from Cohn et al.

Model/ROC-AUC	Human	Mouse	Dog	Opossum
Cohn et al.	0.80	0.78	0.77	0.72
Genomic-ULMFiT 5-mer Stride 2	0.812	0.871	0.773	0.787
Genomic-ULMFiT 4-mer Stride 2	0.804	0.876	0.771	0.786
Genomic-ULMFiT 3-mer Stride 1	0.819	0.875	0.788	0.798

Data Source

Notebook Directory

mRNA/lncRNA Classification

This table shows results for training a classification model on a dataset of coding mRNA sequences and long noncoding RNA (lncRNA) sequences. The dataset comes from A deep recurrent neural network discovers complex biological rules to decipher RNA protein-coding potential by Hill et al. The dataset contains two test sets - a standard test set and a challenge test set.

Model	Test Set	Accuracy	Specificity	Sensitivity	Precision	MCC
GRU Ensemble (Hill et al.)*	Standard Test Set	0.96	0.97	0.95	0.97	0.92
Genomic ULMFiT (3mer stride 1)	Standard Test Set	0.963	0.952	0.974	0.953	0.926
GRU Ensemble (Hill et al.)*	Challenge Test Set	0.875	0.95	0.80	0.95	0.75
Genomic ULMFiT (3mer stride 1)	Challenge Test Set	0.90	0.944	0.871	0.939	0.817

(*) Hill et al. presented their results as a plot rather than as a data table. Values in the above table are estimated by reading off the plot

Data Source

Notebook Directory

Interpreting Results

One way to gain insight into how the classification model makes decisions is to perturb regions of a given input sequence to see how changing different regions of the sequence impact the classification result. This allows us to create plots like the one below, highlighting important sequence regions for classification. In the plot below, the red line corresponds to a true transcription start site. The plot shows how prediction results are sensitive to changes around that location. More detail on interpretations can be found in the Model Interpretations directory.

Long Sequence Inference

Inference on long, unlabeled sequences can be done by breaking the input sequence into chunks and plotting prediction results as a function of length. The image below shows a sample prediction of promoter locations on a 40,000 bp region of the E. coli genome. True promoter locations are shown in red. More detail can be found in this notebook

Relevant Literature

For a comparison to other published methods, see Section 6 of the Methods notebook. Here are some relevant papers in the deep genomics classification space.

DeepCRISPR: optimized CRISPR guide RNA design by deep learning

Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks

PromID: human promoter prediction by deep learning

Deep Learning for Genomics: A Concise Overview

Prediction of deleterious mutations in coding regions of mammals with transfer learning

Enhancer Identification using Transfer and Adversarial Deep Learning of DNA Sequences

PEDLA: predicting enhancers with a deep learning-based algorithmic framework

Predicting enhancers with deep convolutional neural networks

BiRen: predicting enhancers with a deep-learning-based model using the DNA sequence alone

Deep learning models for bacteria taxonomic classification of metagenomic data

Prediction of enhancer-promoter interactions via natural language processing

A deep recurrent neural network discovers complex biological rules to decipher RNA protein-coding potential

Recurrent Neural Network for Predicting Transcription Factor Binding Sites

Learning the Language of the Genome using RNNs

Comments

NameError: name 'Vocab' is not defined

I run link to re-produce Human Promoter Classification with Short Sequences.

I met the error when running Notebook 0: from utils import * File "Human-Short-Seq/utils.py", line 53, in class GenomicVocab(Vocab): NameError: name 'Vocab' is not defined

Please help me to solve it. Thank you very much.

opened by mainguyenanhvu 4
CUDA out of memory during classification.

I am using a Titan X GPU with 12GB memory. It handles language model training on a large protein dataset, and finetuning on a smaller one. But during classification on the small dataset, suddenly CUDA runs out of memory. Have you faced any such issue? How much GPU memory did you use during the training?

opened by nafizh 0

Getting an NameError: name 'BaseTokenizer' is not defined!

I can't running your script because the utils is giving this error in jupyter notebook! Any tip for make it work?

Thank you

NameError                                 Traceback (most recent call last)
<ipython-input-7-70b698022c71> in <module>
     19     return (df_t, df_v)
     20 
---> 21 class GenomicTokenizer(BaseTokenizer):
     22     def __init__(self, lang='en', ngram=5, stride=2):
     23         self.lang = lang

NameError: name 'BaseTokenizer' is not defined

opened by schlogl2017 6

Tips on using model for Regression
Hi, I got to know about your model thru fast.ai forum I am interested in using your genomic model for Kaggle competition: https://www.kaggle.com/c/stanford-covid-vaccine/overview

As the competition only last 3 weeks, need some tips

for RNA seqeunces about 100 sequence length, letters A, G, C, U --> How do I generate a vocab ?

There are some loop types need to predict: S: paired "Stem" M: Multiloop I: Internal loop B: Bulge H: Hairpin loop E: dangling End X: eXternal loop Not clear how to use model to predict loops ?

cheers
opened by sn88798 0
loss values

Hello, thanks a lot for the great repo!

I'm trying to train a language model with different k-mers with stride=1. As you already discussed, we should expect the loss for this LM should be around ln(4^1)~1.3.

However, when I train 6-mer with stride 1, loss is around 8.3 ~ ln(4^6), and if I train 4-mer with stride 1, loss is around 5.2 ~ ln(4^4). Do you have an idea why this happens?

Thanks!

opened by jsephine 0
State of the art?

Hi! Thanks for your great work. I was wondering if this pre-trained network is state of the art or close? I am interested in doing some pre-training with biological sequences, and could use some guidance. Thank you!

opened by lucidrains 2
NameError: name '_get_genomic_processor' is not defined

Function named '_get_genomic_processor' seems to be undefined in 'E. coli 1 Naive Model.ipynb'

NameError: name '_get_genomic_processor' is not defined

opened by nitinnairk 1

ULMFiT for Genomic Sequence Data

Related tags

Overview

Genomic ULMFiT

Results

Promoter Classification

E. coli promoters

Human Promoters (short)

Human Promoters (long)

Other Bacterial Promoters

Metaganomics Classification

Enhancer Classification

mRNA/lncRNA Classification

Interpreting Results

Long Sequence Inference

Relevant Literature

Comments

NameError: name 'Vocab' is not defined

CUDA out of memory during classification.

Getting an NameError: name 'BaseTokenizer' is not defined!

Tips on using model for Regression

loss values

State of the art?

NameError: name '_get_genomic_processor' is not defined

Owner

Karl

Sequence lineage information extracted from RKI sequence data repo

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence Learning (ICLR 2021)

[CVPR 2021] Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

Sequence to Sequence Models with PyTorch

Sequence-to-Sequence learning using PyTorch

Pervasive Attention: 2D Convolutional Networks for Sequence-to-Sequence Prediction

An implementation of a sequence to sequence neural network using an encoder-decoder

Official repository of OFA. Paper: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

DeepConsensus uses gap-aware sequence transformers to correct errors in Pacific Biosciences (PacBio) Circular Consensus Sequencing (CCS) data.

Selene is a Python library and command line interface for training deep neural networks from biological sequence data such as genomes.

A package to predict protein inter-residue geometries from sequence data

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

CLASP - Contrastive Language-Aminoacid Sequence Pretraining

CAUSE: Causality from AttribUtions on Sequence of Events

Code for the ACL2021 paper "Lexicon Enhanced Chinese Sequence Labelling Using BERT Adapter"

You Only 👀 One Sequence

Official codebase for Decision Transformer: Reinforcement Learning via Sequence Modeling.