An End-to-End Machine Learning Library to Optimize AUC (AUROC, AUPRC).

Optimization for AI

Last update: Jan 7, 2023

Related tags

Deep Learning deep-learning tensorflow pytorch medical-imaging gcn graph-neural-networks auprc auroc auc-optimization

Overview

Logo by Zhuoning Yuan

LibAUC: A Machine Learning Library for AUC Optimization

LibAUC aims to provide efficient solutions for optimizing AUC scores (AUROC, AUPRC). We will continuously update our library by fixing bugs and adding new features. If you use or like our library, please star ⭐ our repo. Thank you!

🔍 Why LibAUC?

Deep AUC Maximization (DAM) is a paradigm for learning a deep neural network by maximizing the AUC score of the model on a dataset. In practice, many real-world datasets are usually imbalanced and AUC score is a better metric for evaluating and comparing different methods. Directly maximizing AUC score can potentially lead to the largest improvement in the model’s performance since maximizing AUC aims to rank the prediction score of any positive data higher than any negative data. Our library can be used in many applications, such as medical image classification and drug discovery.

⭐ Key Features

Easy Installation - Integrate AUROC, AUPRC training code with your existing pipeline in just a few steps
Large-scale Learning - Handle large-scale optimization and make the training more smoothly
Distributed Training - Extend to distributed setting to accelerate training efficiency and enhance data privacy
ML Benchmarks - Provide easy-to-use input pipeline and benchmarks on various datasets

⚙️ Installation

$ pip install libauc

You can also download source code from here.

📔 Usage

Official Tutorials:

Constructing Imbalanced Datasets for CIFAR10, CIFAR100, CATvsDOG, STL10 [Notebook][Script]
Training with Pytorch Learning Rate Scheduling [Notebook][Script]
Optimizing AUROC loss with ResNet20 on Imbalanced CIFAR10 [Notebook][Script]
Optimizing AUPRC loss with ResNet18 on Imbalanced CIFAR10 [Notebook][Script]
Optimizing AUROC loss with DenseNet121 on CheXpert [Notebook][Script]
Optimizing AUROC loss with DenseNet121 on CIFAR100 for Federated Learning [Preliminary Release]
Optimizing AUROC loss with DenseNet121 on Melanoma [Notebook][Script]
Optimizing AUROC (Multi-Label) loss with DenseNet121 on CheXpert [Notebook][Script]

Quickstart for Beginners:

Optimizing AUROC (Area Under the Receiver Operating Characteristic)

>>> #import library
>>> from libauc.losses import AUCMLoss
>>> from libauc.optimizers import PESG
...
>>> #define loss
>>> Loss = AUCMLoss(imratio=[YOUR NUMBER])
>>> optimizer = PESG()
...
>>> #training
>>> model.train()    
>>> for data, targets in trainloader:
>>>	data, targets  = data.cuda(), targets.cuda()
        logits = model(data)
	preds = torch.sigmoid(logits)
        loss = Loss(preds, targets) 
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
...	
>>> #restart stage
>>> optimizer.update_regularizer()

Optimizing AUPRC (Area Under the Precision-Recall Curve)

>>> #import library
>>> from libauc.losses import APLoss_SH
>>> from libauc.optimizers import SOAP_SGD, SOAP_ADAM
...
>>> #define loss
>>> Loss = APLoss_SH()
>>> optimizer = SOAP_ADAM()
...
>>> #training
>>> model.train()    
>>> for index, data, targets in trainloader:
>>>	data, targets  = data.cuda(), targets.cuda()
        logits = model(data)
	preds = torch.sigmoid(logits)
        loss = Loss(preds, targets, index) 
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

⚡ Useful Tips

Your dataset should have 0,1 labels, e.g., 1 is the minority class and 0 is the majority class
Compute imratio=#pos/#total based on training set and pass it to AUCMLoss(imratio=xxx)
Adopt a proper initial learning rate, e.g., lr=[0.1, 0.05] usually works better
Choose libauc.optimizers.PESG to optimize AUCMLoss(imratio=xxx)
Use optimizer.update_regularizer(decay_factor=10) to update learning rate and regularizer in stagewise
Add activation layer, e.g., torch.sigmoid(logits), before passing model outputs to loss function
Reshape both variables y_preds and y_targets to (N, 1) before calling loss function

📃 Citation

If you find LibAUC useful in your work, please acknowledge our library and cite the following paper:

@inproceedings{yuan2021robust,
	title={Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification},
	author={Yuan, Zhuoning and Yan, Yan and Sonka, Milan and Yang, Tianbao},
	booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
	year={2021}
	}

📧 Contact

If you have any questions, please contact us @ Zhuoning Yuan [[email protected]] and Tianbao Yang [[email protected]] or please open a new issue in the Github .

Comments

LibAUC code

I was wondering whether you could also commit the core library code to this repository? I just see examples and scripts, and no code for the actual library.

Are there plans to integrate the optimizers with other large scale training frameworks such as DeepSpeed? Have you tested the optimizers on large models? (say >500M parameters)

opened by aashiqmuhamed 5

Support for pytorch-lightning

I'm using pytorch-lightning and am experiencing this issue TypeError: step() got an unexpected keyword argument 'closure' when using PESG. Would appreciate your assistance if you managed to use LibAUC and pytorch-lightning together.

Thanks for your time and amazing work!

Usage is as follows (some lines of code omitted to keep it concise):

import pytorch_lightning as pl
...
# for AUC margin loss
from libauc.losses import AUCMLoss
from libauc.optimizers import PESG

# implementation for the model used in self.classifier
...

class Module(pl.LightningModule):
# implementation of methods unrelated to the error message
...

  def configure_optimizers(self):
    opt = PESG(self.classifier, a=self.criterion.a, b=self.criterion.b, alpha=self.criterion.alpha,
               lr=self.learning_rate, gamma=self.gamma, margin=self.margin, weight_decay=self.weight_decay)
    return opt

enhancement

opened by RoyBenjamin 5

Discrepancy between train and val loss

I was testing AUC margin loss on NIH Chest X-rays dataset in a multi-label binary classification scenario. I wasn't able to improve my validation ROC AUC score at all.

I noticed there is huge difference between train and val loss. Train loss starts from 0.1 and decreases to 0.00x in couple epochs (1.2k iterations with batch size 64 per epoch), but val loss doesn't even move. I tried both training from ImageNet weights and from my own best checkpoint. When I start training from ImageNet weights, val loss doesn't decrease and val auc oscillates between 0.49 and 0.51. When I start training from my own best checkpoint, val loss doesn't decrease again and val auc changes on the scale of 1e-6. Basically, we can say that there is no learning.

My imbalance ratios are

[0.10288084, 0.02472351, 0.11868534, 0.17722083, 0.05124866,
 0.05639493, 0.01206743, 0.04725294, 0.04162504, 0.02054049,
 0.02244024, 0.01503746, 0.03019087, 0.00202462]

which are calculated by simply doing n_positive_samples / (n_positive_samples + n_negative_samples). I use AUCM_MultiLabel loss with imbalance ratios above.

I tried PESG optimizer with different learning rates 1e-1 to 1e-5 but the final result was always same. I use gamma=500, margin=1.0 and weight_decay=1e-5. Finally, I update learning rate and regularizer after first epoch.

The only thing that comes up to my mind is using different heads in models so I checked libauc.models but they are same as well. My model outputs (N, 14) shaped sigmoided logits.

Basically, I did everything except the last one in Useful Tips section.

Your data should have binary labels 0,1 and 1 is the minority class and 0 is the majority class
Compute the imbalance_ratio from your train set and pass it to AUCMLoss(imratio=xxx)
Adopt a proper initial learning rate, e.g., lr=[0.1, 0.05] usually works better
Choose libauc.optimizers.PESG to optimize AUCMLoss(imratio=xxx)
Use optimizer.update_regularizer(decay_factor=10) to update learning rate and regularizer in stagewise
Add activation layer, e.g., torch.sigmoid(logits), before passing model outputs to loss function
Reshape both variables preds and targets to (N, 1) before calling loss function

I don't think the final point is valid for multi label scenario because AUCM_MultiLabel already iterates over the second dimension of predictions and labels, computes loss for every label and accumulates it to total_loss. At this point I can't figure out what I'm missing here.

I'm also providing a portion of my training logs.

train_loss: 0.081627 - lr: 0.00011867: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:49<00:00,  2.32it/s]
val_loss: 0.552354:  91%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋             | 80/88 [00:25<00:02,  3.71it/s]
val_loss: 0.550620: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:41<00:00,  2.11it/s]
Epoch 1 - Training Loss: 0.081627 - Validation Loss: 0.550620 - Validation Micro Average ROC AUC Score: 0.893272 - Macro Average ROC AUC Score: 0.855293
Saving model to ../models/efficientnet/efficientnetv2_s_auc.pt (validation loss decreased from inf to 0.550620)
Reducing learning rate to 0.00001 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.069679 - lr: 0.00092689: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:54<00:00,  2.29it/s]
val_loss: 0.409439: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:28<00:00,  3.12it/s]
Epoch 2 - Training Loss: 0.069679 - Validation Loss: 0.409439 - Validation Micro Average ROC AUC Score: 0.892139 - Macro Average ROC AUC Score: 0.855510
Saving model to ../models/efficientnet/efficientnetv2_s_auc.pt (validation loss decreased from 0.550620 to 0.409439)
Reducing learning rate to 0.00009 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.061587 - lr: 0.00025882: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:52<00:00,  2.30it/s]
val_loss: 0.359555: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.14it/s]
Epoch 3 - Training Loss: 0.061587 - Validation Loss: 0.359555 - Validation Micro Average ROC AUC Score: 0.891677 - Macro Average ROC AUC Score: 0.855467
Saving model to ../models/efficientnet/efficientnetv2_s_auc.pt (validation loss decreased from 0.409439 to 0.359555)
Reducing learning rate to 0.00003 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.053611 - lr: 0.00073131: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:51<00:00,  2.31it/s]
val_loss: 0.343951: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.15it/s]
Epoch 4 - Training Loss: 0.053611 - Validation Loss: 0.343951 - Validation Micro Average ROC AUC Score: 0.890916 - Macro Average ROC AUC Score: 0.855372
Saving model to ../models/efficientnet/efficientnetv2_s_auc.pt (validation loss decreased from 0.359555 to 0.343951)
Reducing learning rate to 0.00007 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.048404 - lr: 0.00049360: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:51<00:00,  2.31it/s]
val_loss: 0.351462: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:28<00:00,  3.13it/s]
Epoch 5 - Training Loss: 0.048404 - Validation Loss: 0.351462 - Validation Micro Average ROC AUC Score: 0.890097 - Macro Average ROC AUC Score: 0.855150
Reducing learning rate to 0.00005 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.042902 - lr: 0.00047681: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:51<00:00,  2.31it/s]
val_loss: 0.351917: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.16it/s]
Epoch 6 - Training Loss: 0.042902 - Validation Loss: 0.351917 - Validation Micro Average ROC AUC Score: 0.889476 - Macro Average ROC AUC Score: 0.855386
Reducing learning rate to 0.00005 @ T=1227!
Updating regularizer @ T=1227!

train_loss: 0.039794 - lr: 0.00074670: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:49<00:00,  2.32it/s]
val_loss: 0.359203: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.16it/s]
Epoch 7 - Training Loss: 0.039794 - Validation Loss: 0.359203 - Validation Micro Average ROC AUC Score: 0.889033 - Macro Average ROC AUC Score: 0.855199
Reducing learning rate to 0.00007 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.035798 - lr: 0.00024610: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:52<00:00,  2.31it/s]
val_loss: 0.363825: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.15it/s]
Epoch 8 - Training Loss: 0.035798 - Validation Loss: 0.363825 - Validation Micro Average ROC AUC Score: 0.887879 - Macro Average ROC AUC Score: 0.855122
Reducing learning rate to 0.00002 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.033877 - lr: 0.00093589: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:53<00:00,  2.30it/s]
val_loss: 0.368633: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:28<00:00,  3.12it/s]
Epoch 9 - Training Loss: 0.033877 - Validation Loss: 0.368633 - Validation Micro Average ROC AUC Score: 0.887842 - Macro Average ROC AUC Score: 0.854925
Reducing learning rate to 0.00009 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.031171 - lr: 0.00011414: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:50<00:00,  2.31it/s]
val_loss: 0.373346: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.15it/s]
Epoch 10 - Training Loss: 0.031171 - Validation Loss: 0.373346 - Validation Micro Average ROC AUC Score: 0.887144 - Macro Average ROC AUC Score: 0.855042
Reducing learning rate to 0.00001 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.034546 - lr: 0.00099968: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:50<00:00,  2.31it/s]
val_loss: 0.368587: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.15it/s]
Epoch 11 - Training Loss: 0.034546 - Validation Loss: 0.368587 - Validation Micro Average ROC AUC Score: 0.887355 - Macro Average ROC AUC Score: 0.855000
Reducing learning rate to 0.00010 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.033058 - lr: 0.00012381: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:49<00:00,  2.32it/s]
val_loss: 0.374046: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:28<00:00,  3.10it/s]
Epoch 12 - Training Loss: 0.033058 - Validation Loss: 0.374046 - Validation Micro Average ROC AUC Score: 0.886223 - Macro Average ROC AUC Score: 0.854960
Reducing learning rate to 0.00001 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.030131 - lr: 0.00091735: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:50<00:00,  2.31it/s]
val_loss: 0.379541: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.18it/s]
Epoch 13 - Training Loss: 0.030131 - Validation Loss: 0.379541 - Validation Micro Average ROC AUC Score: 0.886065 - Macro Average ROC AUC Score: 0.854729
Reducing learning rate to 0.00009 @ T=1227!
Updating regularizer @ T=1227!

train_loss: 0.027916 - lr: 0.00027196: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:52<00:00,  2.30it/s]
val_loss: 0.385755: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.16it/s]
Epoch 14 - Training Loss: 0.027916 - Validation Loss: 0.385755 - Validation Micro Average ROC AUC Score: 0.883915 - Macro Average ROC AUC Score: 0.854616
Reducing learning rate to 0.00003 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.025726 - lr: 0.00071566: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:53<00:00,  2.30it/s]
val_loss: 0.387343: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.17it/s]
Epoch 15 - Training Loss: 0.025726 - Validation Loss: 0.387343 - Validation Micro Average ROC AUC Score: 0.884937 - Macro Average ROC AUC Score: 0.854766
Reducing learning rate to 0.00007 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.024255 - lr: 0.00051047: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:55<00:00,  2.29it/s]
val_loss: 0.393512: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.14it/s]
Epoch 16 - Training Loss: 0.024255 - Validation Loss: 0.393512 - Validation Micro Average ROC AUC Score: 0.883381 - Macro Average ROC AUC Score: 0.854501
Reducing learning rate to 0.00005 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.022581 - lr: 0.00046013: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:54<00:00,  2.30it/s]
val_loss: 0.400079: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.14it/s]
Epoch 17 - Training Loss: 0.022581 - Validation Loss: 0.400079 - Validation Micro Average ROC AUC Score: 0.883271 - Macro Average ROC AUC Score: 0.854286
Reducing learning rate to 0.00005 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.021331 - lr: 0.00076182: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:52<00:00,  2.31it/s]
val_loss: 0.396512: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:27<00:00,  3.15it/s]
Epoch 18 - Training Loss: 0.021331 - Validation Loss: 0.396512 - Validation Micro Average ROC AUC Score: 0.882822 - Macro Average ROC AUC Score: 0.854248
Reducing learning rate to 0.00008 @ T=1227!
Updating regularizer @ T=1227!


train_loss: 0.020128 - lr: 0.00023381: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1227/1227 [08:51<00:00,  2.31it/s]
val_loss: 0.401782: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 88/88 [00:28<00:00,  3.13it/s]
Epoch 19 - Training Loss: 0.020128 - Validation Loss: 0.401782 - Validation Micro Average ROC AUC Score: 0.882145 - Macro Average ROC AUC Score: 0.854304
Early stopping (validation loss didn't increase for 15 epochs/steps)
Best validation loss is 0.343951

opened by gunesevitan 5

Negative Loss

I have been using AUC Maximization for experiments relating to chest X-Ray datasets. For some experiments, I have been encountering negative loss values; I don't think this is supposed to be theoretically possible. My implementation simply uses AUCMLoss and PESG from LibAUC. Any clarification on this issue would be appreciated.

Thanks!

opened by k78ma 4

The right way t predict an image

I was wondering if this is the right way to predict an image from path

import torch
import torchvision.transforms as transforms
from PIL import Image
import numpy as np
from libauc.models import DenseNet121
model = DenseNet121(pretrained=False, last_activation='sigmoid', activations='relu', num_classes=1)
PATH = 'edema_optimized_model.pth'
model.load_state_dict(torch.load(PATH,map_location='cpu'))
# Create the preprocessing transformation here
transform = transforms.Compose([
transforms.Resize((224,224)),
transforms.ToTensor(),
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))
])

# load your image(s)
image = Image.open("CheXpert-v1.0-small/valid/patient64552/study1/view1_frontal.jpg").convert('RGB')
# Transform
input = transform(image)
# # unsqueeze batch dimension, in case you are dealing with a single image
input = input.unsqueeze(0)
# Set model to eval
model.eval()
# Get prediction
output = model(input)
print(output)

opened by EngMoAmri 4

The difference on AUC margin loss V1 & V2

Great job by the authors. We are now optimizing the AUC margin loss on a new federated optimization scheme, but we found two versions of AUC margin loss methods in the /libauc/losses, respectively named AUCMLoss_V1 & AUCMLoss_V2 in auc.py and losses_v1.py. The main difference is about the positive ratio p (V1 includes but v2 didn't), and it directly influences the optimization. Then, why or which should I concentrate on?

opened by wxr99 3
Integrating torchvision models and Lightning

Hi, awesome work on AUC maximization. I'm trying to reproduce the second step from https://arxiv.org/abs/2012.03173 , I sort of have a different pre-training approach.

However, I'm using a torchvision.models.resnet instead of models from libauc, it appears it should fairly easy to use the former when replicated the mentioned tutorial for CheXpert. (Essentially is libauc.models.ResNet18 replaceable with torchvision.models.resnet18 ?)

Also, I'm using libraries like pytorch-lightning and wandb, do you expect any possible irregularities I might face if I use them with LibAUC?

Thanks for your time and amazing work!

opened by vasudev13 3
Where to find codes loss and optimizers, metrics?

Hello, I'm a student that studies medical AI.

First, I truly appreciate to share your library. I want to use AUCM loss and PESG optimizers with variety optimizers and losses. But there are some parameters to check before using the loss or optimizers. (like a, b parameters in PESG, issue on #24 )

So I want to see codes what parameters do actually! However, I cannot find the codes about them. Where can I find?

Thank you so much.

opened by seoulsky-field 2
Multi-Label Training

Hello! I was wondering for the Multi-Label training example, do we need to use Sigmoid after getting the output from densenet 121? Also, is this Multi-Label training referring to the case where a single sample can have multiple classes, or is it the case where a single sample can only be one of many classes? Thanks!

opened by Tom2096 2
Updating a and b in the PESG optimizer

Hello, I see that you update the model's parameters using the regime (momentum, decay etc) and you update the alphas according to the update equations. I wanted to know why you don't explicitly update the primal variables 'a' and 'b' in the PESG code. Is that happening internally in the autograd engine? Thanks!

opened by rohan1561 1
ImbalancedDataGenerator() ISSUE

Hello, nice work! I was just wondering why you convert every label <= split index into 0 and all others into 1 (lines 62 and 63 in utils/generator.py)? That makes the task binary but it just pools several classes into two. Have I misunderstood something? Why would one want to do this?

opened by rohan1561 1

Dose AUCMLoss be sensitive to hyper-params?

In my experiments, I use AUCMLoss implemented by LibAUC, while its performance seems not good.

[19:35:26] Epoch:001 Train AUC: 0.688; Validate AUC: 0.676; Test AUC: 0.685                                                                         Trainer.py:159
[19:35:28] Epoch:002 Train AUC: 0.689; Validate AUC: 0.678; Test AUC: 0.687                                                                         Trainer.py:159
[19:35:30] Epoch:003 Train AUC: 0.690; Validate AUC: 0.679; Test AUC: 0.688                                                                         Trainer.py:159
[19:35:32] Epoch:004 Train AUC: 0.691; Validate AUC: 0.680; Test AUC: 0.689                                                                         Trainer.py:159
[19:35:34] Epoch:005 Train AUC: 0.692; Validate AUC: 0.681; Test AUC: 0.690                                                                         Trainer.py:159
[19:35:36] Epoch:006 Train AUC: 0.693; Validate AUC: 0.683; Test AUC: 0.691                                                                         Trainer.py:159
[19:35:38] Epoch:007 Train AUC: 0.694; Validate AUC: 0.684; Test AUC: 0.692                                                                         Trainer.py:159
[19:35:41] Epoch:008 Train AUC: 0.695; Validate AUC: 0.685; Test AUC: 0.692                                                                         Trainer.py:159
[19:35:43] Epoch:009 Train AUC: 0.695; Validate AUC: 0.685; Test AUC: 0.693                                                                         Trainer.py:159
[19:35:45] Epoch:010 Train AUC: 0.696; Validate AUC: 0.686; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:47] Epoch:011 Train AUC: 0.696; Validate AUC: 0.686; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:49] Epoch:012 Train AUC: 0.696; Validate AUC: 0.686; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:51] Epoch:013 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:53] Epoch:014 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:55] Epoch:015 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:57] Epoch:016 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:35:59] Epoch:017 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:36:01] Epoch:018 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:36:04] Epoch:019 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:36:06] Epoch:020 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:36:08] Epoch:021 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:36:10] Epoch:022 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159
[19:36:13] Epoch:023 Train AUC: 0.696; Validate AUC: 0.687; Test AUC: 0.694                                                                         Trainer.py:159

This log was produced when I use AUCMLoss and PESG optimizer at a MNIST Dataset with ResNet18 and my training process code are

if self.method == "dam":
            from libauc.losses import AUCMLoss
            from libauc.optimizers import PESG

            self.dam_loss = AUCMLoss()
            self.dam_optimizer = PESG(
                model=self.mi.model,
                loss_fn=self.dam_loss,
                momentum=0.9,
                lr=0.1,
                margin=1,
                epoch_decay=0.003,
                weight_decay=1e-4,
                verbose=False,
            )
            self.dam_dataloader = DataLoader(self.di.train_raw, batch_size=self.batch_size, shuffle=True)
...
def dam_twoset_epoch_training_step(self):
        total_risk = 0.0
            
        for x, y in self.dam_dataloader:
            x, y = x.cuda(), y.cuda()
            logits = self.mi(x)
            preds = torch.sigmoid(logits)
            self.dam_optimizer.zero_grad()
            risk = self.dam_loss(preds, y)
            risk.backward()
            self.dam_optimizer.step()
            total_risk += risk.detach().cpu()

        return total_risk / len(self.dam_dataloader)

I am wondering if I need to tune it's hyper-params or check other things?

opened by hehyuan 0

how to use pAUC loss on a validation dataset

Hello, thank you for the nice package. I have a question - is it possible to use partial AUC loss on a validation dataset? I saw that the indices of the positive images are meaningful for loss calculation, and I was wondering if I can add the validation dataset images to the array without impacting the loss calculation (they will stay 0 during training) and getting a meaningful value during the validation step. Thanks.
enhancement

opened by BoredGeo 0

Releases(v1.2.0)

v1.2.0(Jul 31, 2022)
What's New

We continuously update our library by making improvements and adding new features. If you use or like our library, please star⭐ this repo. Thank you!

Major Improvements

In this version,AUCMLoss can automatically compute imratio without requiring this input from users.

Renamed gamma to epoch_decay for PESG and PDSCA optimizers, i.e., epoch_decay = 1/gamma

Reimplemented ImbalancedDataGenerator for constructing imbalanced dataset for benchmarking. Tutorial is available here.

Improved implementations of APLoss by removing some redundant computations.

Merged SOAP_ADAM and SOAP_SGD optimizers into one optimizer SOAP. Tutorial is provided here.

Removed dependency of TensorFlow and now LibAUC only requires PyTorch installed .

Updated existing tutorials to match the new version of LibAUC. Tutorials are available here.

New Features

Introduced DualSampler, TriSampler for sampling data that best fit the x-risk optimization to balance inner and outer estimation error.

Introduced CompositionAUCLoss and PDSCA optimizer. Tutorial is provided here.

Introduced SogCLR with Dynamic Contrastive Loss for training Self-Supervised Learning models using small batch size. Tutorial and code are provided here.

Introduced NDCG_Loss and SONG optimizer for optimizing NDCG. Tutorials are provided here.

Introduced pAUCLoss with three optimizers: SOPA, SOPAs, SOTAs for optimizing Partial AUROC. Tutorials are provided here.

Added three evaluation functions: auc_roc_score (binary/multi-task), auc_prc_score (binary/multi-task) and pauc_roc_score(binary).

Feedback

If you have any feedback/suggestions, please contact us @ Zhuoning Yuan [[email protected]] and Tianbao Yang [[email protected]].

Source code(tar.gz)
Source code(zip)
libauc-1.2.0-py3-none-any.whl(71.88 KB)
v1.1.8(Jan 11, 2022)
What's New

Fixed some bugs and improved the training stability

Source code(tar.gz)
Source code(zip)
libauc-1.1.8-py3-none-any.whl(37.94 KB)
v1.1.6(Nov 11, 2021)
What's New

Added Support for Multi-Label Training. Tutorial for training CheXpert is available here!

Fixed some bugs and improved the training stability

Source code(tar.gz)
Source code(zip)
libauc-1.1.6-py3-none-any.whl(34.64 KB)

An End-to-End Machine Learning Library to Optimize AUC (AUROC, AUPRC).

Related tags

Overview

LibAUC: A Machine Learning Library for AUC Optimization

🔍 Why LibAUC?

⭐ Key Features

⚙️ Installation

📔 Usage

Official Tutorials:

Quickstart for Beginners:

Optimizing AUROC (Area Under the Receiver Operating Characteristic)

Optimizing AUPRC (Area Under the Precision-Recall Curve)

⚡ Useful Tips

📃 Citation

📧 Contact

Comments

Releases(v1.2.0)

v1.2.0(Jul 31, 2022)

What's New

Major Improvements

New Features

Feedback

v1.1.8(Jan 11, 2022)

What's New

v1.1.6(Nov 11, 2021)

What's New

Owner

Optimization for AI

🐤 Nix-TTS: An Incredibly Lightweight End-to-End Text-to-Speech Model via Non End-to-End Distillation

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

An end-to-end machine learning web app to predict rugby scores (Pandas, SQLite, Keras, Flask, Docker)

End-to-end machine learning project for rices detection

Open-L2O: A Comprehensive and Reproducible Benchmark for Learning to Optimize Algorithms

Official implementation for "Symbolic Learning to Optimize: Towards Interpretability and Scalability"

TensorFlow implementation for Bayesian Modeling and Uncertainty Quantification for Learning to Optimize: What, Why, and How

NVIDIA Merlin is an open source library providing end-to-end GPU-accelerated recommender systems, from feature engineering and preprocessing to training deep learning models and running inference in production.

Avalanche RL: an End-to-End Library for Continual Reinforcement Learning

Sequential model-based optimization with a `scipy.optimize` interface

Let Python optimize the best stop loss and take profits for your TradingView strategy.

OCTIS: Comparing Topic Models is Simple! A python package to optimize and evaluate topic models (accepted at EACL2021 demo track)

Optimize Trading Strategies Using Freqtrade

Pytorch library for end-to-end transformer models training and serving

A PyTorch library and evaluation platform for end-to-end compression research

Learning recognition/segmentation models without end-to-end training. 40%-60% less GPU memory footprint. Same training time. Better performance.