Official repo of the paper "Surface Form Competition: Why the Highest Probability Answer Isn't Always Right"

Last update: Dec 23, 2022

Related tags

Overview

Surface Form Competition

This is the official repo of the paper "Surface Form Competition: Why the Highest Probability Answer Isn't Always Right" We provide scripts for downloading/processing datasets and for reproducing our results on GPT-2 and GPT-3. We do not guarantee exact reproducibility, as library versions and GPUs may cause small differences, but these should be extremely minor.

Dependencies

We use python3 and pytorch 1.7.0, but we do not use cutting-edge features from either and expect to be largely forward and backward compatible. That is not a guarantee or promise.

You can use pip install -r requirements.txt to install the required libraries.

OpenAI Beta

To use GPT-3 you must use OpenAI Beta, which is limited access. You can apply for access here. Once you have access you will need to point the score.py to your API key with the --key argument or put your key in api.key which is the default path.

Downloading Datasets

DATA_README.md has thorough instructions for downloading and processing datasets. We provide automatic downloaders and processers for datasets where possible in data_downloaders/ but see DATA_README for full instructions.

Running Scorers

Once you have a dataset downloaded, running all the zero-shot scoring strategies at once is as simple as:

python score.py 
   
     --model

where is the abbreviation for a given dataset used for table rows in the paper. If there is any confusion, simply look in score.py to see how dataset selection works. is the name of either a GPT-2 or GPT-3 model e.g. xl, davinci, etc. To speed things up you can use a larger --batch if you have enough GPU memory.

You might also like...

This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations at CVPR'21. According to some product reasons, we are not planning to release the training/testing codes and models. However, we will release the dataset and the scripts to prepare the dataset.

TransFill-Reference-Inpainting This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transf

80 Dec 8, 2022

This repo contains the official implementations of EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis This repo contains the official implementations of EigenDamage: Structured Prunin

107 Apr 20, 2022

Official repo for SemanticGAN https://nv-tlabs.github.io/semanticGAN/

SemanticGAN This is the official code for: Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizat

151 Dec 28, 2022

The official repo for CVPR2021——ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search.

ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search [paper] Introduction This is the official implementation of ViPNAS: Efficient V

42 Sep 26, 2022

Official git repo for the CHIRP project

CHIRP Project This is the official git repository for the CHIRP project. Pull requests are accepted here, but for the moment, the main repository is s

77 Jan 8, 2023

This repo is official PyTorch implementation of MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices(CVPRW 2021).

Github Code of "MobileHumanPose: Toward real-time 3D human pose estimation in mobile devices" Introduction This repo is official PyTorch implementatio

203 Jan 5, 2023

Official Repo of my work for SREC Nandyal Machine Learning Bootcamp

About the Bootcamp A 3-day Machine Learning Bootcamp organised by Department of Electronics and Communication Engineering, Santhiram Engineering Colle

1 Nov 29, 2021

The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

325 Jan 5, 2023

Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

105 Dec 23, 2022

Comments

Still puzzled about your PMI-domain...

Hi, I am very interested in this work. It seems to alleviate the issues of surface form competition by proposing another scoring function, e.g. PMI-domain. The domain premise can be a prompt for classification tasks. I still think this method is more like a calibration technique... Or do I have any misunderstandings? How can we resolve the surface form competition issue in our LM predicted distribution?

Thanks for your help!

opened by MM-IR 1
Questions about Template Design

Hi,

Thank you for this interesting work, and congratulations on the EMNLP acceptance! After reading the paper, one question I am having is that how those domain-specific templates are designed and selected. In the get_sst2_variant_template function (data_loaders.py), I see a couple of template candidates, so I guess what you did is to first manually design a couple of template candidates for each domain/task and then select the best one based on the dev set performance (if both dev and test sets exist), right? Please correct me if I am wrong and look forward to your reply. Thanks!

opened by bflashcp3f 1
Preset functions for True/False summatives?

Fantastic idea from the insight on surface form competition, and it does solve the problem well. It's impressive indeed.

Have you considered adding preset functions, for the case of "Does this make sense?" / "Is this truthful?" calls with True/False 1-token responses and so forth?

Right now we maintain a list of FALSE, false ... etc ... this seems much better. It seems like how we as humans psychologically process the world and are affected by similar words/multiple word senses, fascinating...

I was sad to see this must use restricted pre=specified set of tokens to rebalance, but that makes sense by necessity of reducing complexity.

opened by jmdetect 1

Owner

Peter West

PhD student at UW, focussing on ML and NLP

GitHub

The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection .

GCoNet The official repo of the CVPR 2021 paper Group Collaborative Learning for Co-Salient Object Detection . Trained model Download final_gconet.pth

46 Nov 17, 2022

Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

83 Nov 27, 2022

Official repo for BMVC2021 paper ASFormer: Transformer for Action Segmentation

ASFormer: Transformer for Action Segmentation This repo provides training & inference code for BMVC 2021 paper: ASFormer: Transformer for Action Segme

42 Dec 23, 2022

Official Repo for Ground-aware Monocular 3D Object Detection for Autonomous Driving

Visual 3D Detection Package: This repo aims to provide flexible and reproducible visual 3D detection on KITTI dataset. We expect scripts starting from

305 Dec 19, 2022

This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer This repo is the official implementation for TransBTS: Multimodal Brain Tumor Segmenta

247 Dec 28, 2022

Official repo for QHack—the quantum machine learning hackathon

Note: This repository has been frozen while we consider the submissions for the QHack Open Hackathon. We hope you enjoyed the event! Welcome to QHack,

118 Jan 5, 2023

This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.

Skeleton Aware Multi-modal Sign Language Recognition By Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li and Yun Fu. Smile Lab @ Northeastern

128 Dec 8, 2022