Transformers-Tutorials
Hi there!
This repository contains demos I made with the Transformers library by
NOTE: if you are not familiar with HuggingFace and/or Transformers, I highly recommend to check out our free course, which introduces you to several Transformer architectures (such as BERT, GPT-2, T5, BART, etc.), as well as an overview of the HuggingFace libraries, including Transformers, Tokenizers, Datasets, Accelerate and the hub.
Currently, it contains the following demos:
- BERT (paper):
- fine-tuning
BertForTokenClassificationon a named entity recognition (NER) dataset. - fine-tuning
BertForSequenceClassificationfor multi-label text classification.
- fine-tuning
- CANINE (paper):
- fine-tuning
CanineForSequenceClassificationon IMDb
- fine-tuning
- DETR (paper):
- performing inference with
DetrForObjectDetection - fine-tuning
DetrForObjectDetectionon a custom object detection dataset - evaluating
DetrForObjectDetectionon the COCO detection 2017 validation set - performing inference with
DetrForSegmentation - fine-tuning
DetrForSegmentationon COCO panoptic 2017
- performing inference with
- GPT-J-6B (repository):
- performing inference with
GPTJForCausalLMto illustrate few-shot learning and code generation
- performing inference with
- ImageGPT (blog post):
- (un)conditional image generation with
ImageGPTForCausalLM - linear probing with ImageGPT
- (un)conditional image generation with
- LayoutLM (paper):
- LayoutLMv2 (paper):
- fine-tuning
LayoutLMv2ForSequenceClassificationon RVL-CDIP - fine-tuning
LayoutLMv2ForTokenClassificationon FUNSD - fine-tuning
LayoutLMv2ForTokenClassificationon FUNSD using the🤗 Trainer - performing inference with
LayoutLMv2ForTokenClassificationon FUNSD - true inference with
LayoutLMv2ForTokenClassification(when no labels are available) + Gradio demo - fine-tuning
LayoutLMv2ForTokenClassificationon CORD - fine-tuning
LayoutLMv2ForQuestionAnsweringon DOCVQA
- fine-tuning
- LUKE (paper):
- fine-tuning
LukeForEntityPairClassificationon a custom relation extraction dataset using PyTorch Lightning
- fine-tuning
- SegFormer (paper):
- performing inference with
SegformerForSemanticSegmentation - fine-tuning
SegformerForSemanticSegmentationon custom data using native PyTorch
- performing inference with
- Perceiver IO (paper):
- showcasing masked language modeling and image classification with the Perceiver
- fine-tuning the Perceiver for image classification
- fine-tuning the Perceiver for text classification
- predicting optical flow between a pair of images with
PerceiverForOpticalFlow - auto-encoding a video (images, audio, labels) with
PerceiverForMultimodalAutoencoding
- showcasing masked language modeling and image classification with the Perceiver
- T5 (paper):
- fine-tuning
T5ForConditionalGenerationon a Dutch summarization dataset on TPU using HuggingFace Accelerate - fine-tuning
T5ForConditionalGeneration(CodeT5) for Ruby code summarization using PyTorch Lightning
- fine-tuning
- TAPAS (paper):
- fine-tuning
TapasForQuestionAnsweringon the Microsoft Sequential Question Answering (SQA) dataset - evaluating
TapasForSequenceClassificationon the Table Fact Checking (TabFact) dataset
- fine-tuning
- TrOCR (paper):
- performing inference with
TrOCRto illustrate optical character recognition with Transformers, as well as making a Gradio demo - fine-tuning
TrOCRon the IAM dataset using the Seq2SeqTrainer - fine-tuning
TrOCRon the IAM dataset using native PyTorch - evaluating
TrOCRon the IAM test set
- performing inference with
- Vision Transformer (paper):
... more to come!
If you have any questions regarding these demos, feel free to open an issue on this repository.
Btw, I was also the main contributor to add the following algorithms to the library:
- TAbular PArSing (TAPAS) by Google AI
- Vision Transformer (ViT) by Google AI
- Data-efficient Image Transformers (DeiT) by Facebook AI
- LUKE by Studio Ousia
- DEtection TRansformers (DETR) by Facebook AI
- CANINE by Google AI
- BEiT by Microsoft Research
- LayoutLMv2 (and LayoutXLM) by Microsoft Research
- TrOCR by Microsoft Research
- SegFormer by NVIDIA
- ImageGPT by OpenAI
- Perceiver by Deepmind
All of them were an incredible learning experience. I can recommend anyone to contribute an AI algorithm to the library!
Data preprocessing
Regarding preparing your data for a PyTorch model, there are a few options:
- a native PyTorch dataset + dataloader. This is the standard way to prepare data for a PyTorch model, namely by subclassing
torch.utils.data.Dataset, and then a creating correspondingDataLoader(which is a Python generator that allows to loop over the items of a dataset). When subclassing theDatasetclass, one needs to implement 3 methods:__init__,__len__(which returns the number of examples of the dataset) and__getitem__(which returns an example of the dataset, given an integer index). Here's an example of creating a basic text classification dataset (assuming one has a CSV that contains 2 columns, namely "text" and "label"):
from torch.utils.data import Dataset
class CustomTrainDataset(Dataset):
def __init__(self, df, tokenizer):
self.df = df
self.tokenizer = tokenizer
def __len__(self):
return len(self.df)
def __getitem__(self, idx):
# get item
item = df.iloc[idx]
text = item['text']
label = item['label']
# encode text
encoding = self.tokenizer(text, padding="max_length", max_length=128, truncation=True, return_tensors="pt")
# remove batch dimension which the tokenizer automatically adds
encoding = {k:v.squeeze() for k,v in encoding.items()}
# add label
encoding["label"] = torch.tensor(label)
return encoding
Instantiating the dataset then happens as follows:
from transformers import BertTokenizer
import pandas as pd
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
df = pd.read_csv("path_to_your_csv")
train_dataset = CustomTrainDataset(df=df tokenizer=tokenizer)
Accessing the first example of the dataset can then be done as follows:
encoding = train_dataset[0]
In practice, one creates a corresponding DataLoader, that allows to get batches from the dataset:
from torch.utils.data import DataLoader
train_dataloader = DataLoader(train_dataset, batch_size=4, shuffle=True)
I often check whether the data is created correctly by fetching the first batch from the data loader, and then printing out the shapes of the tensors, decoding the input_ids back to text, etc.
batch = next(iter(train_dataloader))
for k,v in batch.items():
print(k, v.shape)
# decode the input_ids of the first example of the batch
print(tokenizer.decode(batch['input_ids'][0].tolist())
- HuggingFace Datasets. Datasets is a library by HuggingFace that allows to easily load and process data in a very fast and memory-efficient way. It is backed by Apache Arrow, and has cool features such as memory-mapping, which allow you to only load data into RAM when it is required. It only has deep interoperability with the HuggingFace hub, allowing to easily load well-known datasets as well as share your own with the community.
Loading a custom dataset as a Dataset object can be done as follows (you can install datasets using pip install datasets):
from datasets import load_dataset
dataset = load_dataset('csv', data_files={'train': ['my_train_file_1.csv', 'my_train_file_2.csv'] 'test': 'my_test_file.csv'})
Here I'm loading local CSV files, but there are other formats supported (including JSON, Parquet, txt) as well as loading data from a local Pandas dataframe or dictionary for instance. You can check out the docs for all details.
Training frameworks
Regarding fine-tuning Transformer models (or more generally, PyTorch models), there are a few options:
- using native PyTorch. This is the most basic way to train a model, and requires the user to manually write the training loop. The advantage is that this is very easy to debug. The disadvantage is that one needs to implement training him/herself, such as setting the model in the appropriate mode (
model.train()/model.eval()), handle device placement (model.to(device)), etc. A typical training loop in PyTorch looks as follows (inspired by this great PyTorch intro tutorial):
import torch
model = ...
# I almost always use a learning rate of 5e-5 when fine-tuning Transformer based models
optimizer = torch.optim.Adam(model.parameters(), lr=5-e5)
# put model on GPU, if available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
for epoch in range(epochs):
model.train()
train_loss = 0.0
for batch in train_dataloader:
# put batch on device
batch = {k:v.to(device) for k,v in batch.items()}
# forward pass
outputs = model(**batch)
loss = outputs.loss
train_loss += loss.item()
loss.backward()
optimizer.step()
optimizer.zero_grad()
print("Loss after epoch {epoch}:", train_loss/len(train_dataloader))
model.eval()
val_loss = 0.0
with torch.no_grad():
for batch in eval_dataloader:
# put batch on device
batch = {k:v.to(device) for k,v in batch.items()}
# forward pass
outputs = model(**batch)
loss = outputs.logits
val_loss += loss.item()
print("Validation loss after epoch {epoch}:", val_loss/len(eval_dataloader))
- PyTorch Lightning (PL). PyTorch Lightning is a framework that automates the training loop written above, by abstracting it away in a Trainer object. Users don't need to write the training loop themselves anymore, instead they can just do
trainer = Trainer()and thentrainer.fit(model). The advantage is that you can start training models very quickly (hence the name lightning), as all training-related code is handled by theTrainerobject. The disadvantage is that it may be more difficult to debug your model, as the training and evaluation is now abstracted away. - HuggingFace Trainer. The HuggingFace Trainer API can be seen as a framework similar to PyTorch Lightning in the sense that it also abstracts the training away using a Trainer object. However, contrary to PyTorch Lightning, it is not meant not be a general framework. Rather, it is made especially for fine-tuning Transformer-based models available in the HuggingFace Transformers library. The Trainer also has an extension called
Seq2SeqTrainerfor encoder-decoder models, such as BART, T5 and theEncoderDecoderModelclasses. Note that all PyTorch example scripts of the Transformers library make use of the Trainer. - HuggingFace Accelerate: Accelerate is a new project, that is made for people who still want to write their own training loop (as shown above), but would like to make it work automatically irregardless of the hardware (i.e. multiple GPUs, TPU pods, mixed precision, etc.).
3.4k Dec 31, 2022
117 Dec 7, 2022
23 Oct 19, 2022
3 Nov 30, 2021
103 Jan 5, 2023
440 Jan 2, 2023
76 Dec 22, 2022
44 Oct 20, 2022
87 Dec 9, 2022
80 Dec 30, 2022
186 Jan 4, 2023
40 Dec 5, 2022
551 Dec 29, 2022
11 Jun 11, 2021
190 Jan 3, 2023
40 Dec 13, 2022
197 Nov 29, 2022
728 Dec 28, 2022
281 Dec 22, 2022