Intent parsing and slot filling in PyTorch with seq2seq + attention

Related tags

Deep Learning pytorch-seq2seq-intent-parsing
Overview

PyTorch Seq2Seq Intent Parsing

Reframing intent parsing as a human - machine translation task. Work in progress successor to torch-seq2seq-intent-parsing

The command language

This is a simple command language developed for the "home assistant" Maia living in my apartment. She's designed as a collection of microservices with services for lights (Hue), switches (WeMo), and info such as weather and market prices.

A command consists of a "service", a "method", and some number of arguments.

lights setState office_light on
switches getState teapot
weather getWeather "San Francisco"
price getPrice TSLA

These can be represented with variable placeholders:

lights setState $device $state
switches getState $device
weather getWeather $location
price getPrice $symbol

We can imagine a bunch of human sentences that would map to a single command:

"Turn the office light on."
"Please turn on the light in the office."
"Maia could you set the office light on, thank you."

Which could similarly be represented with placeholders.

TODO: Specific vs. freeform variables

A shortcoming of the approach so far is that the model has to learn translations of specific values, for example mapping all of the device names to their equivalent device_name. If we added a "basement light" the model would have no basement_light in the output vocabulary unless it was re-trained.

The bigger the potential input space, the more obvious the problem - consider the getWeather command, where the model would need to be trained with every possible location we might ask about. Worse yet, consider a playMusic command that could take any song or artist name...

This can be solved with a technique which I have implemented in Torch here. The training pairs have "variable placeholders" in the output translation, which the model generates during an intial pass. Then the network fills in the values of these placeholders with an additional pass over the input.

You might also like...
Using a Seq2Seq RNN architecture via TensorFlow to predict future Bitcoin prices

Recurrent Bitcoin Network A Data Science Thesis Project About This repository contains the source code for implementing Bitcoin price prediciton using

Frizu 6 Sep 8, 2022
Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch
Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image classification, in Pytorch

Transformer in Transformer Implementation of Transformer in Transformer, pixel level attention paired with patch level attention for image c

Phil Wang 272 Dec 23, 2022
PyTorch code for our paper "Attention in Attention Network for Image Super-Resolution"

Under construction... Attention in Attention Network for Image Super-Resolution (A2N) This repository is an PyTorch implementation of the paper "Atten

Haoyu Chen 71 Dec 30, 2022
Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding
Official Pytorch Implementation of Relational Self-Attention: What's Missing in Attention for Video Understanding

Relational Self-Attention: What's Missing in Attention for Video Understanding This repository is the official implementation of "Relational Self-Atte

mandos 43 Dec 7, 2022
Implementation of Deformable Attention in Pytorch from the paper
Implementation of Deformable Attention in Pytorch from the paper "Vision Transformer with Deformable Attention"

Deformable Attention Implementation of Deformable Attention from this paper in Pytorch, which appears to be an improvement to what was proposed in DET

Phil Wang 128 Dec 24, 2022
Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification
Implementation of STAM (Space Time Attention Model), a pure and simple attention model that reaches SOTA for video classification

STAM - Pytorch Implementation of STAM (Space Time Attention Model), yet another pure and simple SOTA attention model that bests all previous models in

Phil Wang 109 Dec 28, 2022
Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms
Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

Chenglin Yang 20 Dec 31, 2021
Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones
Implementation of the 😇 Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones

HaloNet - Pytorch Implementation of the Attention layer from the paper, Scaling Local Self-Attention For Parameter Efficient Visual Backbones. This re

Phil Wang 189 Nov 22, 2022
Attention-driven Robot Manipulation (ARM) which includes Q-attention
Attention-driven Robot Manipulation (ARM) which includes Q-attention

Attention-driven Robotic Manipulation (ARM) This codebase is home to: Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation I

Stephen James 84 Dec 29, 2022
Comments
  • intent and slot joint detection

    intent and slot joint detection

    hey spro,

    Thanks for sharing many Pytorch tutorials in NLP, the code is clean and really helps.

    I am wondering is there a plan for you to release the intent and slot joint detection in this project ? if it's possible, that would be great.

    Have a good day. :）

    opened by walkacross 1
  • Index 1 is out of range

    Index 1 is out of range

    Hi Sean, Thanks for making this open source! I have been working on a similar implementation in tensorflow, and wanted to port to pytorch. To this end, I was checking out your project to try to learn how to do it, and I came upon the following error right away. Do you have an idea of what's going on here? I am obviously very new to pytorch, so even basic suggestions could be helpful. (also note that you have to use 0.1.1 of torchtext for this to work.)

    output lang = DictionaryLang(size = 41)
  0%|          | 0/1193514 [00:00<?, ?it/s]Loading word vectors from /Users/thomas/Downloads/glove.twitter.27B/glove.twitter.27B.100d.txt
100%|██████████| 1193514/1193514 [03:48<00:00, 5215.09it/s]
input lang = GloVeLang(size = 100)
Training for 200 epochs...
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1596, in <module>
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1023, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/thomas/projects/apollo/torchtest/pytorch-seq2seq-intent-parsing/train.py", line 83, in <module>
    loss = train(input_variable, target_variable)
  File "/Users/thomas/projects/apollo/torchtest/pytorch-seq2seq-intent-parsing/train.py", line 48, in train
    decoder_output, decoder_hidden, decoder_attention = decoder(decoder_input, decoder_hidden, encoder_outputs)
  File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/thomas/projects/apollo/torchtest/pytorch-seq2seq-intent-parsing/model.py", line 59, in forward
    rnn_output, hidden = self.gru(word_embedded, last_hidden)
  File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Python/2.7/site-packages/torch/nn/modules/rnn.py", line 162, in forward
    output, hidden = func(input, self.all_weights, hx)
  File "/Library/Python/2.7/site-packages/torch/nn/_functions/rnn.py", line 351, in forward
    return func(input, *fargs, **fkwargs)
  File "/Library/Python/2.7/site-packages/torch/nn/_functions/rnn.py", line 244, in forward
    nexth, output = func(input, hidden, weight)
  File "/Library/Python/2.7/site-packages/torch/nn/_functions/rnn.py", line 84, in forward
    hy, output = inner(input, hidden[l], weight[l])
  File "/Library/Python/2.7/site-packages/torch/autograd/variable.py", line 76, in __getitem__
    return Index.apply(self, key)
  File "/Library/Python/2.7/site-packages/torch/autograd/_functions/tensor.py", line 16, in forward
    result = i.index(ctx.index)
IndexError: index 1 is out of range for dimension 0 (of size 1)
    opened by unicornmafia 0
  • Freeform variables

    Freeform variables

    Hello Sean,

    First of all, thank you so much for open sourcing your code in torch7/pytorch. I have a simple question for you: When do you plan to port the freeform variables capability ?

    Thanks again

    opened by Cadene 1
Owner
Sean Robertson
I sure do like websites.
Sean Robertson
GitHub
pytorch bert intent classification and slot filling

pytorch_bert_intent_classification_and_slot_filling 基于pytorch的中文意图识别和槽位填充 说明 基本思路就是：分类+序列标注（命名实体识别）同时训练。 使用的预训练模型：hugging face上的chinese-bert-wwm-ext 依

西西嘛呦 33 Dec 15, 2022
Open source code for Paper "A Co-Interactive Transformer for Joint Slot Filling and Intent Detection"

A Co-Interactive Transformer for Joint Slot Filling and Intent Detection This repository contains the PyTorch implementation of the paper: A Co-Intera

null 67 Dec 5, 2022
SlotRefine: A Fast Non-Autoregressive Model forJoint Intent Detection and Slot Filling

SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling Reference Main paper to be cited (Di Wu et al., 2020) @article

Moore 34 Nov 3, 2022
Object-Centric Learning with Slot Attention

Slot Attention This is a re-implementation of "Object-Centric Learning with Slot Attention" in PyTorch (https://arxiv.org/abs/2006.15055). Requirement

Untitled AI 72 Jan 2, 2023
Implementation for "Manga Filling Style Conversion with Screentone Variational Autoencoder" (SIGGRAPH ASIA 2020 issue)

Manga Filling with ScreenVAE SIGGRAPH ASIA 2020 | Project Website | BibTex This repository is for ScreenVAE introduced in the following paper "Manga F

null 30 Dec 24, 2022
Implementation of fast algorithms for Maximum Spanning Tree (MST) parsing that includes fast ArcMax+Reweighting+Tarjan algorithm for single-root dependency parsing.

Fast MST Algorithm Implementation of fast algorithms for (Maximum Spanning Tree) MST parsing that includes fast ArcMax+Reweighting+Tarjan algorithm fo

Miloš Stanojević 11 Oct 14, 2022
GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks This repository implements a capsule model Inten

Joel Huang 15 Dec 24, 2022
Citation Intent Classification in scientific papers using the Scicite dataset an Pytorch

Citation Intent Classification Table of Contents About the Project Built With Installation Usage Acknowledgments About The Project Citation Intent Cla

Federico Nocentini 4 Mar 4, 2022
An end-to-end implementation of intent prediction with Metaflow and other cool tools

You Don't Need a Bigger Boat An end-to-end (Metaflow-based) implementation of an intent prediction flow for kids who can't MLOps good and wanna learn

Jacopo Tagliabue 614 Dec 31, 2022
On-device speech-to-intent engine powered by deep learning

Rhino Made in Vancouver, Canada by Picovoice Rhino is Picovoice's Speech-to-Intent engine. It directly infers intent from spoken commands within a giv

Picovoice 510 Dec 30, 2022