OOD Generalization and Detection (ACL 2020)

Last update: Jan 9, 2023

Overview

Pretrained Transformers Improve Out-of-Distribution Robustness

How does pretraining affect out-of-distribution robustness? We create an OOD benchmark and use it to show that pretrained transformers such as BERT have substantially higher OOD accuracy and OOD detection rates compared to traditional NLP models.

This repository most of the code for the paper Pretrained Transformers Improve Out-of-Distribution Robustness, ACL 2020.

Requires Python 3+ and PyTorch 1.0+.

To correctly use RoBERTa model, see allennlp_glue_patch/notes.py.

Citation

If you find this useful in your research, please consider citing:

@inproceedings{hendrycks2020pretrained,
    Author = {Dan Hendrycks and Xiaoyuan Liu and Eric Wallace and Adam Dziedzic Rishabh Krishnan and Dawn Song},
    Booktitle = {Association for Computational Linguistics},                            
    Year = {2020},
    Title = {Pretrained Transformers Improve Out-of-Distribution Robustness}}

Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization This is an official implementation of "Feature Stylization and Domain-

22 Sep 22, 2022

This repo includes our code for evaluating and improving transferability in domain generalization (NeurIPS 2021)

Transferability for domain generalization This repo is for evaluating and improving transferability in domain generalization (NeurIPS 2021), based on

9 Nov 29, 2022

CSAC - Collaborative Semantic Aggregation and Calibration for Separated Domain Generalization

CSAC Introduction This repository contains the implementation code for paper: Co

5 Jul 22, 2022

[CVPR 2022] Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions" paper

template-pose Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions

92 Dec 28, 2022

MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

MOT-Tracking-by-Detection-Pipeline Tracking-by-Detection形式のMOT(Multi Object Trac

41 Nov 23, 2022

Data and Code for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning"

Introduction Code and data for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning". We cons

81 Dec 27, 2022

Library for machine learning stacking generalization.

stacked_generalization Implemented machine learning *stacking technic[1]* as handy library in Python. Feature weighted linear stacking is also availab

114 Jul 19, 2022

A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization

sam.pytorch A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization ( Foret+2020) Paper, Official implementa

102 Dec 28, 2022

[CVPR'21] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space

FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space by Quande Liu, Cheng Chen, Ji

178 Jan 6, 2023

Comments

OOD Detection

Hello, thanks for publishing the code used in your interesting paper! I am wondering if this repo also contains the code for the OOD Detection part, i.e. for recording the confidence scores on the SST-2 dataset and on the 5 validation datasets. I am curious how you calculated the FPR95 and anomaly. Going through all files I cannot find the related code. Do I miss it or is it not added? Thank you very much for your time, @tpatzelt

opened by tpatzelt 2
requriments.txt is missing.
Hi can you mention the version of allennlp and other modules you have used in a requirments.txt. It seems allennlp has changed a lot and the commands mentioned in the readme does not work. For example while running the command

allennlp train config/test/sst-lstm.json -s model/test-lstm[sst-2] --include-package allennlp_glue_patch

I encountered the following error.

"No module named allennlp.run"
opened by islamazhar 1
Json decode error while using BERT and ROBERTA models while using SST-2 dataset and imdb dataset

While I try to use the bert or roberta config using allennlp I get the following error

"json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)"

I was wondering which version of allennlp do you recommend for the code or is there some specific resolution you know regarding this issue.

opened by bhavdeep98 0

Owner

littleRound

Ph.D. Student, EECS, UC Berkeley B.Eng., Computer Science, SJTU

GitHub https://arxiv.org/abs/2004.06100

Github for the conference paper GLOD-Gaussian Likelihood OOD detector

FOOD - Fast OOD Detector Pytorch implamentation of the confernce peper FOOD arxiv link. Abstract Deep neural networks (DNNs) perform well at classifyi

17 Jun 19, 2022

The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

Domain Generalization for Medical Imaging Classification with Linear Dependency Regularization The code release of paper 'Domain Generalization for Me

56 Dec 28, 2022

OOD Generalization and Detection (ACL 2020)

Related tags

Overview

Pretrained Transformers Improve Out-of-Distribution Robustness

Citation

You might also like...

Official pytorch implementation of "Feature Stylization and Domain-aware Contrastive Loss for Domain Generalization" ACMMM 2021 (Oral)

This repo includes our code for evaluating and improving transferability in domain generalization (NeurIPS 2021)

CSAC - Collaborative Semantic Aggregation and Calibration for Separated Domain Generalization

[CVPR 2022] Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions" paper

MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

Data and Code for ACL 2021 Paper "Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning"

Library for machine learning stacking generalization.

A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization

[CVPR'21] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space

Comments

OOD Detection

requriments.txt is missing.

Json decode error while using BERT and ROBERTA models while using SST-2 dataset and imdb dataset

Owner

littleRound

Github for the conference paper GLOD-Gaussian Likelihood OOD detector

The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

Code for SentiBERT: A Transferable Transformer-Based Architecture for Compositional Sentiment Semantics (ACL'2020).

Code for Graph-to-Tree Learning for Solving Math Word Problems (ACL 2020)

Code for the paper "Balancing Training for Multilingual Neural Machine Translation, ACL 2020"

Data from "HateCheck: Functional Tests for Hate Speech Detection Models" (Röttger et al., ACL 2021)

UDP++ (ECCVW 2020 Oral), (Winner of COCO 2020 Keypoint Challenge).

Adaout is a practical and flexible regularization method with high generalization and interpretability

CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)

CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)