Black-Box-Tuning - Black-Box Tuning for Language-Model-as-a-Service

Related tags

Deep Learning Black-Box-Tuning
Overview

Black-Box-Tuning

Source code for paper "Black-Box Tuning for Language-Model-as-a-Service".

Being busy recently, the code in this repo and this tutorial will be very brief. Please let me know if you find any issues.

Prepare your environment

The implementation of Black-Box Tuning is quite simple, you can check our code and easily implement it in your own environment. Or you can create a new environment to run our implementation, which is based on Nevergrad, Transformers and FastNLP. Optionally, we use fitlog to monitor experimental results. You can uncomment the fitlog-related lines in our code to use it.

conda create --name bbt python=3.8
conda activate bbt
pip install transformers==4.1.1
pip install datasets
pip install fastNLP
pip install nevergrad
pip install sklearn
git clone https://github.com/txsun1997/Black-Box-Tuning
cd Black-Box-Tuning

Optimize your prompt without gradients

Now you can run Black-Box Tuning with run.sh:

bash run.sh

Results will be saved in a directory named results/. In general, you will obtain the following results:

SST-2 split Best Accuracy
Train 100
Dev 96.87
Test 88.19

To reproduce other experiments in our paper, change the arguments of bbt.py, for example,

python bbt.py --task_name "agnews" --n_prompt_tokens 50 --intrinsic_dim 500 --k_shot 16 --device "cuda:0" --seed 42 --loss_type "hinge" --cat_or_add "add" --budget 8000

Cite

If you find this work helpful, please cite:

@article{sun2022bbt,
  title={Black-Box Tuning for Language-Model-as-as-Service}, 
  author={Tianxiang Sun and Yunfan Shao and Hong Qian and Xuanjing Huang and Xipeng Qiu},
  journal={arXiv preprint arXiv:2201.03514},
  year={2022}
}
Comments
  • 结果复现问题

    结果复现问题

    您好，我在sst2，yelpp和agnews上的(seed=42)复现结果为87.61(90.33)，91.00(92.86)，82.53(85.28)。 括号中是论文报告的结果，我所使用的机器是V100，配置如下： python -u bbt.py
    --task_name XXX
    --n_prompt_tokens 50
    --intrinsic_dim 500
    --k_shot 16
    --device "cuda:3"
    --seed 42
    --loss_type "ce"
    --cat_or_add "add"
    --budget 8000
    --print_every 50
    --eval_every 100 请问是我的参数设置有问题吗？或者其他什么原因导致的结果上的差异吗？

    opened by hccngu 2
  • 关于Dbpedia

    关于Dbpedia

    /remote-home/txsun/zfhe/dbpedia_csv.tar.gz 这个文件可以提供下吗？使用datasets自动下载dbpedia出问题了： requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Read timed out. (read timeout=100.0) 谢谢哈～

    opened by hccngu 2
  • Can't replicate results of BBTv2 paper

    Can't replicate results of BBTv2 paper

    Hi, I tried your BBTv2 code but failed to get comparable results as reported in your paper.

    In my case, using the command

    python deepbbt.py   --model_name "roberta-large"  --task_name "snli"   --n_prompt_tokens 50   --intrinsic_dim 500   --k_shot 16   --device "cuda:0"   --seed 42   --loss_type "ce"   --cat_or_add "add"   --random_proj "normal"   --sigma1 1   --sigma2 0.2   --popsize 20   --bound 0   --budget 8000   --print_every 50   --eval_every 100

    gives the following results

    Done. Elapsed time: 39.49383888641993 (mins)
Evaluate on test data...
Evaluate data in 75.54 seconds!                                                                                                                                                                                     
[tester] 
SNLIMetric: acc=0.5509975570032574, hinge=2.8394456026220167, ce=11.656479801339513
Test acc: 0.551

    which is higher than other gradient-free baselines but much smaller that the number reported in your paper (60.62).

    I'm wondering why. Do I need to tune the random seed?

    opened by jordane95 2
  • DeepBBT : the outputs of modified Roberta don't have hidden states

    DeepBBT : the outputs of modified Roberta don't have hidden states

    Hello, there is a bug in RobertaL Model ouput when I run deepbbt because it do not have hidden state even the config.output_hidden_states = True (Line439).

    Then I found the Line 975 of deep_modeling_roberta.py only output a dict with logit rather than the original MskedLMOutput, which leads to the above bug.

    When I try to directly replace the logit of MaskedLMOutput to fix it, it suggest the inner model also do not output hidden state. I would appreciate if you could help to fix that !

    opened by Desein-Yang 2
  • Truncation Length

    Truncation Length

    Hi, very nice work. Also, we are currently trying to reproduce your work as our baseline.

    Can I ask you what strategies do you try with in our Yelp P. dataset? Especially for very long sentences. In my current experiments, it tells me that I have to perform truncation...

    Thanks!

    opened by MM-IR 2
  • Can the user-side can obtain the final prompt in literal?

    Can the user-side can obtain the final prompt in literal?

    I found this optimization is about vector after user obtain the results from server. I wander if there any possible to get the final prompt text to help other people to design prompts

    opened by LemonQC 1
  • Why not calculate prompt on the server side?

    Why not calculate prompt on the server side?

    In the paper, it is assumed that the user can access only an inference API provided by the server, and thus we need a solution to calculate the prompt on the user side. This limit is from the requirement of keeping PLM parameters secret.

    Why not calculate the prompt on the server side. Returning the prompt, or, the gradient on the prompt, does not leak PLM parameters, too.

    We do not prefer the above solution, because of additional burden at the server side?

    opened by hdzhang-code 1
  • add web demo/models/datasets to ICML organization on Hugging Face

    add web demo/models/datasets to ICML organization on Hugging Face

    Hi, congrats on acceptance at ICML 2022 for Black-Box-Tuning. We are having a event on Hugging Face for ICML 2022, where you can submit spaces(web demos), models, and datasets for papers for a chance to win prizes, after joining the organization using this link https://huggingface.co/organizations/ICML2022/share/BpynfJtfsOTktlmXYoKNqqCnyufKLFXuay, let me know if you need any help with the above steps, thanks

    opened by AK391 0
Owner
Tianxiang Sun
@FudanNLP
Tianxiang Sun
GitHub
Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet

Attack classification models with transferability, black-box attack; unrestricted adversarial attacks on imagenet, CVPR2021 安全AI挑战者计划第六期：ImageNet无限制对抗攻击 决赛第四名（team name: Advers）

null 51 Dec 1, 2022
transfer attack; adversarial examples; black-box attack; unrestricted Adversarial Attacks on ImageNet; CVPR2021 天池黑盒竞赛

transfer_adv CVPR-2021 AIC-VI: unrestricted Adversarial Attacks on ImageNet CVPR2021 安全AI挑战者计划第六期赛道2：ImageNet无限制对抗攻击 介绍 ： 深度神经网络已经在各种视觉识别问题上取得了最先进的性能。

null 25 Dec 8, 2022
Code for "Diversity can be Transferred: Output Diversification for White- and Black-box Attacks"

Output Diversified Sampling (ODS) This is the github repository for the NeurIPS 2020 paper "Diversity can be Transferred: Output Diversification for W

null 50 Dec 11, 2022
[CVPR 2021] Pytorch implementation of Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs

Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANs In this work, we propose a framework HijackGAN, which enables non-linear latent space travers

Hui-Po Wang 46 Sep 5, 2022
Explainer for black box models that predict molecule properties

Explaining why that molecule exmol is a package to explain black-box predictions of molecules. The package uses model agnostic explanations to help us

White Laboratory 172 Dec 19, 2022
A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

Yunxia Zhao 3 Dec 29, 2022
Code for "Retrieving Black-box Optimal Images from External Databases" (WSDM 2022)

Retrieving Black-box Optimal Images from External Databases (WSDM 2022) We propose how a user retreives an optimal image from external databases of we

joisino 5 Apr 13, 2022
This repository contains the code and models necessary to replicate the results of paper: How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective

Black-Box-Defense This repository contains the code and models necessary to replicate the results of our recent paper: How to Robustify Black-Box ML M

OPTML Group 2 Oct 5, 2022
This repository contains the code and models necessary to replicate the results of paper: How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective

Black-Box-Defense This repository contains the code and models necessary to replicate the results of our recent paper: How to Robustify Black-Box ML M

OPTML Group 2 Oct 5, 2022
EMNLP 2021 Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections Ruiqi Zhong, Kristy Lee*, Zheng Zhang*, Dan Klein EMN

Ruiqi Zhong 42 Nov 3, 2022
[CVPR'21 Oral] Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning [CVPR'21, Oral] By Zhicheng Huang*, Zhaoyang Zeng*, Yupan H

Multimedia Research 196 Dec 13, 2022
Omnidirectional Scene Text Detection with Sequential-free Box Discretization (IJCAI 2019). Including competition model, online demo, etc.

Box_Discretization_Network This repository is built on the pytorch [maskrcnn_benchmark]. The method is the foundation of our ReCTs-competition method

Yuliang Liu 266 Nov 24, 2022
Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera.

Tools to create pixel-wise object masks, bounding box labels (2D and 3D) and 3D object model (PLY triangle mesh) for object sequences filmed with an RGB-D camera. This project prepares training and testing data for various deep learning projects such as 6D object pose estimation projects singleshotpose, as well as object detection and instance segmentation projects.

null 305 Dec 16, 2022
In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Contrastive Learning of Object Representations Supervisor: Prof. Dr. Gemma Roig Institutions: Goethe University CVAI - Computational Vision & Artifici

Dirk Neuhäuser 6 Dec 8, 2022
Step by Step on how to create an vision recognition model using LOBE.ai, export the model and run the model in an Azure Function

Step by Step on how to create an vision recognition model using LOBE.ai, export the model and run the model in an Azure Function

El Bruno 3 Mar 30, 2022
PyTorch implementation of Interpretable Explanations of Black Boxes by Meaningful Perturbation

PyTorch implementation of Interpretable Explanations of Black Boxes by Meaningful Perturbation The paper: https://arxiv.org/abs/1704.03296 What makes

Jacob Gildenblat 322 Dec 17, 2022
A customisable game where you have to quickly click on black tiles in order of appearance while avoiding clicking on white squares.

W.I.P-Aim-Memory-Game A customisable game where you have to quickly click on black tiles in order of appearance while avoiding clicking on white squar

dE_soot 1 Dec 8, 2021
Ray tracing of a Schwarzschild black hole written entirely in TensorFlow.

TensorGeodesic Ray tracing of a Schwarzschild black hole written entirely in TensorFlow. Dependencies: Python 3 TensorFlow 2.x numpy matplotlib About

null 5 Jan 15, 2022
This Artificial Intelligence program can take a black and white/grayscale image and generate a realistic or plausible colorized version of the same picture.

Colorizer The point of this project is to write a program capable of taking a black and white / grayscale image, and generating a realistic or plausib

Maitri Shah 1 Jan 6, 2022