Intent parsing and slot filling in PyTorch with seq2seq + attention

Related tags

Text Data & NLP pytorch-seq2seq-intent-parsing
Overview

PyTorch Seq2Seq Intent Parsing

Reframing intent parsing as a human - machine translation task. Work in progress successor to torch-seq2seq-intent-parsing

The command language

This is a simple command language developed for the "home assistant" Maia living in my apartment. She's designed as a collection of microservices with services for lights (Hue), switches (WeMo), and info such as weather and market prices.

A command consists of a "service", a "method", and some number of arguments.

lights setState office_light on
switches getState teapot
weather getWeather "San Francisco"
price getPrice TSLA

These can be represented with variable placeholders:

lights setState $device $state
switches getState $device
weather getWeather $location
price getPrice $symbol

We can imagine a bunch of human sentences that would map to a single command:

"Turn the office light on."
"Please turn on the light in the office."
"Maia could you set the office light on, thank you."

Which could similarly be represented with placeholders.

TODO: Specific vs. freeform variables

A shortcoming of the approach so far is that the model has to learn translations of specific values, for example mapping all of the device names to their equivalent device_name. If we added a "basement light" the model would have no basement_light in the output vocabulary unless it was re-trained.

The bigger the potential input space, the more obvious the problem - consider the getWeather command, where the model would need to be trained with every possible location we might ask about. Worse yet, consider a playMusic command that could take any song or artist name...

This can be solved with a technique which I have implemented in Torch here. The training pairs have "variable placeholders" in the output translation, which the model generates during an intial pass. Then the network fills in the values of these placeholders with an additional pass over the input.

You might also like...
:hot_pepper: R²SQL: "Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing." (AAAI 2021)

R²SQL The PyTorch implementation of paper Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing. (AAAI 2021) Requirement

huybery 60 Dec 31, 2022
GAP-text2SQL: Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

GAP-text2SQL: Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training Code and model from our AAAI 2021 paper

Amazon Web Services - Labs 83 Jan 9, 2023
PIZZA - a task-oriented semantic parsing dataset

The PIZZA dataset continues the exploration of task-oriented parsing by introducing a new dataset for parsing pizza and drink orders, whose semantics cannot be captured by flat slots and intents.

null 17 Dec 14, 2022
TweebankNLP - Pre-trained Tweet NLP Pipeline (NER, tokenization, lemmatization, POS tagging, dependency parsing) + Models + Tweebank-NER

TweebankNLP This repo contains the new Tweebank-NER dataset and Twitter-Stanza p

Laboratory for Social Machines 84 Dec 20, 2022
Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch
Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memories using approximate nearest neighbors, in Pytorch

Memorizing Transformers - Pytorch Implementation of Memorizing Transformers (ICLR 2022), attention net augmented with indexing and retrieval of memori

Phil Wang 364 Jan 6, 2023
pytorch implementation of Attention is all you need

A Pytorch Implementation of the Transformer: Attention Is All You Need Our implementation is largely based on Tensorflow implementation Requirements N

null 230 Dec 7, 2022
A PyTorch implementation of the Transformer model in
A PyTorch implementation of the Transformer model in "Attention is All You Need".

Attention is all you need: A Pytorch Implementation This is a PyTorch implementation of the Transformer model in "Attention is All You Need" (Ashish V

Yu-Hsiang Huang 7.1k Jan 5, 2023
[ICCV 2021] Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification
[ICCV 2021] Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

Counterfactual Attention Learning Created by Yongming Rao*, Guangyi Chen*, Jiwen Lu, Jie Zhou This repository contains PyTorch implementation for ICCV

Yongming Rao 89 Dec 18, 2022
HAIS_2GNN: 3D Visual Grounding with Graph and Attention
HAIS_2GNN: 3D Visual Grounding with Graph and Attention

HAIS_2GNN: 3D Visual Grounding with Graph and Attention This repository is for the HAIS_2GNN research project. Tao Gu, Yue Chen Introduction The motiv

Yue Chen 1 Nov 26, 2022
Comments
  • intent and slot joint detection

    intent and slot joint detection

    hey spro,

    Thanks for sharing many Pytorch tutorials in NLP, the code is clean and really helps.

    I am wondering is there a plan for you to release the intent and slot joint detection in this project ? if it's possible, that would be great.

    Have a good day. :）

    opened by walkacross 1
  • Index 1 is out of range

    Index 1 is out of range

    Hi Sean, Thanks for making this open source! I have been working on a similar implementation in tensorflow, and wanted to port to pytorch. To this end, I was checking out your project to try to learn how to do it, and I came upon the following error right away. Do you have an idea of what's going on here? I am obviously very new to pytorch, so even basic suggestions could be helpful. (also note that you have to use 0.1.1 of torchtext for this to work.)

    output lang = DictionaryLang(size = 41)
  0%|          | 0/1193514 [00:00<?, ?it/s]Loading word vectors from /Users/thomas/Downloads/glove.twitter.27B/glove.twitter.27B.100d.txt
100%|██████████| 1193514/1193514 [03:48<00:00, 5215.09it/s]
input lang = GloVeLang(size = 100)
Training for 200 epochs...
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1596, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1023, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/thomas/projects/apollo/torchtest/pytorch-seq2seq-intent-parsing/train.py", line 83, in <module>
    loss = train(input_variable, target_variable)
  File "/Users/thomas/projects/apollo/torchtest/pytorch-seq2seq-intent-parsing/train.py", line 48, in train
    decoder_output, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_hidden, encoder_outputs)
  File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/thomas/projects/apollo/torchtest/pytorch-seq2seq-intent-parsing/model.py", line 59, in forward
    rnn_output, hidden = self.gru(word_embedded, last_hidden)
  File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Python/2.7/site-packages/torch/nn/modules/rnn.py", line 162, in forward
    output, hidden = func(input, self.all_weights, hx)
  File "/Library/Python/2.7/site-packages/torch/nn/_functions/rnn.py", line 351, in forward
    return func(input, *fargs, **fkwargs)
  File "/Library/Python/2.7/site-packages/torch/nn/_functions/rnn.py", line 244, in forward
    nexth, output = func(input, hidden, weight)
  File "/Library/Python/2.7/site-packages/torch/nn/_functions/rnn.py", line 84, in forward
    hy, output = inner(input, hidden[l], weight[l])
  File "/Library/Python/2.7/site-packages/torch/autograd/variable.py", line 76, in __getitem__
    return Index.apply(self, key)
  File "/Library/Python/2.7/site-packages/torch/autograd/_functions/tensor.py", line 16, in forward
    result = i.index(ctx.index)
IndexError: index 1 is out of range for dimension 0 (of size 1)
    opened by unicornmafia 0
  • Freeform variables

    Freeform variables

    Hello Sean,

    First of all, thank you so much for open sourcing your code in torch7/pytorch. I have a simple question for you: When do you plan to port the freeform variables capability ?

    Thanks again

    opened by Cadene 1
Owner
Sean Robertson
I sure do like websites.
Sean Robertson
GitHub
multi-label，classifier，text classification，多标签文本分类，文本分类，BERT，ALBERT，multi-label-classification，seq2seq，attention，beam search

multi-label，classifier，text classification，多标签文本分类，文本分类，BERT，ALBERT，multi-label-classification，seq2seq，attention，beam search

hellonlp 30 Dec 12, 2022
An open source framework for seq2seq models in PyTorch.

pytorch-seq2seq Documentation This is a framework for sequence-to-sequence (seq2seq) models implemented in PyTorch. The framework has modularized and

International Business Machines 1.4k Jan 2, 2023
🏖 Easy training and deployment of seq2seq models.

Headliner Headliner is a sequence modeling library that eases the training and in particular, the deployment of custom sequence models for both resear

Axel Springer Ideas Engineering GmbH 231 Nov 18, 2022
🏖 Easy training and deployment of seq2seq models.

Headliner Headliner is a sequence modeling library that eases the training and in particular, the deployment of custom sequence models for both resear

Axel Springer Ideas Engineering GmbH 220 Feb 10, 2021
voice2json is a collection of command-line tools for offline speech/intent recognition on Linux

Command-line tools for speech and intent recognition on Linux

Michael Hansen 988 Jan 4, 2023
Unsupervised intent recognition

INTENT author: steeve LAQUITAINE description: deployment pattern: currently batch only Setup & run git clone https://github.com/slq0/intent.git bash

sl 1 Apr 8, 2022
PhoNLP: A BERT-based multi-task learning toolkit for part-of-speech tagging, named entity recognition and dependency parsing

PhoNLP is a multi-task learning model for joint part-of-speech (POS) tagging, named entity recognition (NER) and dependency parsing. Experiments on Vietnamese benchmark datasets show that PhoNLP produces state-of-the-art results, outperforming a single-task learning approach that fine-tunes the pre-trained Vietnamese language model PhoBERT for each task independently.

VinAI Research 109 Dec 2, 2022
Code and checkpoints for training the transformer-based Table QA models introduced in the paper TAPAS: Weakly Supervised Table Parsing via Pre-training.

End-to-end neural table-text understanding models.

Google Research 914 Jan 7, 2023
Neural network models for joint POS tagging and dependency parsing (CoNLL 2017-2018)

Neural Network Models for Joint POS Tagging and Dependency Parsing Implementations of joint models for POS tagging and dependency parsing, as describe

Dat Quoc Nguyen 152 Sep 2, 2022
Mednlp - Medical natural language parsing and utility library

Medical natural language parsing and utility library A natural language medical

Paul Landes 3 Aug 24, 2022