Optimizing Deeper Transformers on Small Datasets

Related tags

Deep Learning DT-Fixup
Overview

DT-Fixup

Optimizing Deeper Transformers on Small Datasets

Paper published in ACL 2021: arXiv

Detailed instructions to replicate our results in the paper can be found in the folders spider and reclor.

Cite

If you found this codebase or our work useful, please cite:

@InProceedings{xu2021optimizing,
  author = {Xu, Peng and Kumar, Dhruv and Yang, Wei and Zi, Wenjie and Tang, Keyi and Huang, Chenyang and Cheung, Jackie Chi Kit and Prince, Simon J.D. and Cao, Yanshuai},
  title = {Optimizing Deeper Transformers on Small Datasets}
  booktitle = {The 59th Annual Meeting of the Association for Computational Linguistics (ACL 2021)},
  month = {August},
  year = {2021},
  publisher = {ACL}
}
Comments
  • the value in SQL

    the value in SQL

    Hi, thanks for the sharing code.

    I am a little confused that the predicted results don't have a specified column value, is there any parameter that restricts the model?

    For example, the generated SQL below:

    SELECT students.cell_mobile_number FROM students WHERE students.first_name = "value" and students.last_name = "value"

    What is the parameter if I want to get a specified column value rather than a terminal symbol?

    Thanks.

    opened by Gyyz 2
  • Request the Docker env.

    Request the Docker env.

    Thank you for your great work! I would like to ask whether you have a docker environment that can run directly. At present, the project depends on a lot and is not easy to deploy directly. Thanks a lot!

    opened by huybery 1
  • Model not able to preprocess?

    Model not able to preprocess?

    When I run the model with the code below !python -m semparser.run --config_path config.yml --commit 0 --do_preprocess --do_training

    Then the model is not able to preprocess the SQL queries. I am getting the following error image

    image

    opened by premshanker-ai 0
  • Tables.json not found?

    Tables.json not found?

    Love this project however when I try to run it, I get this error

    PS: Running this in colab pro

    2022-10-10 19:45:42.737414: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected DEBUG:semparser.common.registry:instantiating rat_new_sl of preprocessor DEBUG:semparser.common.registry:instantiating spider of transition_system DEBUG:semparser.common.registry:instantiating asdl of grammar Creating schema with meta... ERROR:root:[' File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main\n "main", mod_spec)\n', ' File "/usr/lib/python3.7/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n', ' File "/content/DT-Fixup/spider/semparser/run.py", line 122, in \n logger.error(traceback.format_stack())\n'] 10/10/2022 07:45:45 [' File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main\n "main", mod_spec)\n', ' File "/usr/lib/python3.7/runpy.py", line 85, in _run_code\n exec(code, run_globals)\n', ' File "/content/DT-Fixup/spider/semparser/run.py", line 122, in \n logger.error(traceback.format_stack())\n'] ERROR:root:Traceback (most recent call last): File "/content/DT-Fixup/spider/semparser/run.py", line 103, in argument_resolver.resolve_argument(config['PREPROCESSOR']) File "/content/DT-Fixup/spider/semparser/common/argument_resolver.py", line 36, in resolve_argument return resolve_argument(argument_dict, caller) File "/content/DT-Fixup/spider/semparser/common/argument_resolver.py", line 56, in resolve_argument return caller(**resolved_arguments) File "/content/DT-Fixup/spider/semparser/modules/semantic_parser/preprocessor/rat_new_sl.py", line 188, in prepare_data schema_with_db_meta = update_schemas_with_meta(raw_schema, database_folder) File "/content/DT-Fixup/spider/semparser/modules/alanschema/scripts/generate_schema_with_db_meta.py", line 111, in update_schemas_with_meta with open(table_fpath, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: 'spider/tables.json'

    10/10/2022 07:45:45 Traceback (most recent call last): File "/content/DT-Fixup/spider/semparser/run.py", line 103, in argument_resolver.resolve_argument(config['PREPROCESSOR']) File "/content/DT-Fixup/spider/semparser/common/argument_resolver.py", line 36, in resolve_argument return resolve_argument(argument_dict, caller) File "/content/DT-Fixup/spider/semparser/common/argument_resolver.py", line 56, in resolve_argument return caller(**resolved_arguments) File "/content/DT-Fixup/spider/semparser/modules/semantic_parser/preprocessor/rat_new_sl.py", line 188, in prepare_data schema_with_db_meta = update_schemas_with_meta(raw_schema, database_folder) File "/content/DT-Fixup/spider/semparser/modules/alanschema/scripts/generate_schema_with_db_meta.py", line 111, in update_schemas_with_meta with open(table_fpath, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: 'spider/tables.json'

    Traceback (most recent call last): File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/usr/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/content/DT-Fixup/spider/semparser/run.py", line 124, in raise ex File "/content/DT-Fixup/spider/semparser/run.py", line 103, in argument_resolver.resolve_argument(config['PREPROCESSOR']) File "/content/DT-Fixup/spider/semparser/common/argument_resolver.py", line 36, in resolve_argument return resolve_argument(argument_dict, caller) File "/content/DT-Fixup/spider/semparser/common/argument_resolver.py", line 56, in resolve_argument return caller(**resolved_arguments) File "/content/DT-Fixup/spider/semparser/modules/semantic_parser/preprocessor/rat_new_sl.py", line 188, in prepare_data schema_with_db_meta = update_schemas_with_meta(raw_schema, database_folder) File "/content/DT-Fixup/spider/semparser/modules/alanschema/scripts/generate_schema_with_db_meta.py", line 111, in update_schemas_with_meta with open(table_fpath, 'r') as f: FileNotFoundError: [Errno 2] No such file or directory: 'spider/tables.json'

    opened by premshanker-ai 2
  • About the experimental results

    About the experimental results

    I directly ran the code of the code base without any modification. The results are as follows

    08/28/2021 06:50:45 [Epoch 100] dev acc: 0.70696 (took 220s) 08/28/2021 06:50:45 checkpoint: tmp/dtfixup 08/28/2021 06:50:45 best dev accuracy: 0.72340 08/28/2021 06:50:45 checkpoint: tmp/dtfixup

    The best dev accuracy is only 72.3%, Maybe I missed something? For the Experiment Configuration, I found that the batch in the code is 32 and the batch in the paper is 16. Is this the reason for my failure?

    opened by huybery 2
Owner
null
GitHub
U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection

The code for our newly accepted paper in Pattern Recognition 2020: "U^2-Net: Going Deeper with Nested U-Structure for Salient Object Detection."

Xuebin Qin 6.5k Jan 9, 2023
[Preprint] "Bag of Tricks for Training Deeper Graph Neural Networks A Comprehensive Benchmark Study" by Tianlong Chen*, Kaixiong Zhou*, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, Zhangyang Wang

Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study Codes for [Preprint] Bag of Tricks for Training Deeper Graph

VITA 101 Dec 29, 2022
Deeper DCGAN with AE stabilization

AEGeAN Deeper DCGAN with AE stabilization Parallel training of generative adversarial network as an autoencoder with dedicated losses for each stage.

Tyler Kvochick 36 Feb 17, 2022
MogFace: Towards a Deeper Appreciation on Face Detection

MogFace: Towards a Deeper Appreciation on Face Detection Introduction In this repo, we propose a promising face detector, termed as MogFace. Our MogFa

null 48 Dec 20, 2022
Image morphing without reference points by applying warp maps and optimizing over them.

Differentiable Morphing Image morphing without reference points by applying warp maps and optimizing over them. Differentiable Morphing is machine lea

Alex K 380 Dec 19, 2022
Adversarial Color Enhancement: Generating Unrestricted Adversarial Images by Optimizing a Color Filter

ACE Please find the preliminary version published at BMVC 2020 in the folder BMVC_version, and its extended journal version in Journal_version. Datase

null 28 Dec 25, 2022
Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Implementation for Iso-Points (CVPR 2021) Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations paper |

Yifan Wang 66 Nov 8, 2022
Optimizing DR with hard negatives and achieving SOTA first-stage retrieval performance on TREC DL Track (SIGIR 2021 Full Paper).

Optimizing Dense Retrieval Model Training with Hard Negatives Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, Shaoping Ma This repo provi

Jingtao Zhan 99 Dec 27, 2022
Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

Minimal PyTorch implementation of Generative Latent Optimization This is a reimplementation of the paper Piotr Bojanowski, Armand Joulin, David Lopez-

Thomas Neumann 117 Nov 27, 2022
Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt

Feed forward VQGAN-CLIP model, where the goal is to eliminate the need for optimizing the latent space of VQGAN for each input prompt. This is done by

Mehdi Cherti 135 Dec 30, 2022
⚡️Optimizing einsum functions in NumPy, Tensorflow, Dask, and more with contraction order optimization.

Optimized Einsum Optimized Einsum: A tensor contraction order optimizer Optimized einsum can significantly reduce the overall execution time of einsum

Daniel Smith 653 Dec 30, 2022
PHOTONAI is a high level python API for designing and optimizing machine learning pipelines.

PHOTONAI is a high level python API for designing and optimizing machine learning pipelines. We've created a system in which you can easily select and

Medical Machine Learning Lab - University of Münster 57 Nov 12, 2022
The implementation of "Optimizing Shoulder to Shoulder: A Coordinated Sub-Band Fusion Model for Real-Time Full-Band Speech Enhancement"

SF-Net for fullband SE This is the repo of the manuscript "Optimizing Shoulder to Shoulder: A Coordinated Sub-Band Fusion Model for Real-Time Full-Ban

Guochen Yu 36 Dec 2, 2022
Implementation of "Debiasing Item-to-Item Recommendations With Small Annotated Datasets" (RecSys '20)

Debiasing Item-to-Item Recommendations With Small Annotated Datasets This is the code for our RecSys '20 paper. Other materials can be found here: Ful

Microsoft 34 Aug 10, 2022
Minimal But Practical Image Classifier Pipline Using Pytorch, Finetune on ResNet18, Got 99% Accuracy on Own Small Datasets.

PyTorch Image Classifier Updates As for many users request, I released a new version of standared pytorch immage classification example at here: http:

JinTian 106 Nov 6, 2022
Re-implementation of 'Grokking: Generalization beyond overfitting on small algorithmic datasets'

Re-implementation of the paper 'Grokking: Generalization beyond overfitting on small algorithmic datasets' Paper Original paper can be found here Data

Tom Lieberum 38 Aug 9, 2022
PyTorch implementation of Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets

Simple PyTorch Implementation of "Grokking" Implementation of Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets Usage Running

Teddy Koker 15 Sep 29, 2022
SPT_LSA_ViT - Implementation for Visual Transformer for Small-size Datasets

Vision Transformer for Small-Size Datasets Seung Hoon Lee and Seunghyun Lee and Byung Cheol Song | Paper Inha University Abstract Recently, the Vision

Lee SeungHoon 87 Jan 1, 2023
An easy way to build PyTorch datasets. Modularly build datasets and automatically cache processed results

EasyDatas An easy way to build PyTorch datasets. Modularly build datasets and automatically cache processed results Installation pip install git+https

Ximing Yang 4 Dec 14, 2021