Visualization Toolbox for Long Short Term Memory networks (LSTMs)

Overview

Visual Analysis for State Changes in RNNs

More information about LSTMVis, an introduction video, and the link to the live demo can be found at lstm.seas.harvard.edu

Also check out our new work on Sequence-to-Sequence models on github or the live demo at http://seq2seq-vis.io/

Changes in V2.1

  • update to Python 3.7++ (thanks to @nneophyt)

Changes in V2

  • new design and server-backend
  • discrete zooming for hidden-state track
  • added annotation tracks for meta-data and prediction
  • added training and extraction workflow for tensorflow
  • client is now ES6 and D3v4
  • some performance enhancements on client side
  • Added Keras tutorial here (thanks to Mohammadreza Ebrahimi)

Install

Please use python 3.7 or later to install LSTMVis.

Clone the repository:

git clone https://github.com/HendrikStrobelt/LSTMVis.git; cd LSTMVis

Install python (server-side) requirements using pip:

python -m venv  venv3
source venv3/bin/activate
pip install -r requirements.txt

Download & Unzip example dataset(s) into /data/05childbook:

Children Book - Gutenberg - 2.2 GB

Parens Dataset - 10k small - 0.03 GB

start server:

source venv3/bin/activate
python lstm_server.py -dir <datadir>

For the example dataset, use python lstm_server.py -dir data

open browser at http://localhost:8888 - eh voila !

Adding Your Own Data

If you want to train your own data first, please read the Training document. If you have your own data at hand, adding it to LSTMVis is very easy. You only need three files:

  • HDF5 file containing the state vectors for each time step (e.g. states.hdf5)
  • HDF5 file containing a word ID for each time step (e.g. train.hdf5)*
  • Dict file containing the mapping from word ID to word (e.g. train.dict)*

A schematic representation of the data:

Data Format

*If you don't have these files yet, but a space-separated .txt file of your training data instead, check out our text conversion tool

Data Directory

LSTMVis parses all subdirectories of for config files lstm.yml. A typical might look like this:


├── paren  		        <--- project directory
│   ├── lstm.yml 		<--- config file
│   ├── states.hdf5 	        <--- states for each time step
│   ├── train.hdf5 		<--- word ID for each time step
│   └── train.dict 		<--- mapping word ID -> word
├── fun .. 

Config File

a simple example of an lstm.yml is:

name: children books  # project name
description: children book texts from the Gutenberg project # little description

files: # assign files to reference name
  states: states.hdf5 # HDF5 files have to end with .h5 or .hdf5 !!!
  train: train.hdf5 # word ids of training set
  words: train.dict # dict files have to end with .dict !!

word_sequence: # defines the word sequence
  file: train # HDF5 file
  path: word_ids # path to table in HDF5
  dict_file: words # dictionary to map IDs from HDF5 to words

states: # section to define which states of your model you want to look at
  file: states # HDF5 files containing the state for each position
  types: [
        {type: state, layer: 1, path: states1}, # type={state, output}, layer=[1..x], path = HDF5 path
        {type: state, layer: 2, path: states2},
        {type: output, layer: 2, path: output2}
  ]

Intrigued ? Here is more..

Check out our documents about:

Credits

LSTMVis is a collaborative project of Hendrik Strobelt, Sebastian Gehrmann, Bernd Huber, Hanspeter Pfister, and Alexander M. Rush at Harvard SEAS.

Comments
  • SwaggerValidationError 'states,words' is not of type 'array'

    SwaggerValidationError 'states,words' is not of type 'array'

    Upon running python lstm_server.py -dir data I get the error below. I'm running Python 2.7.15 in a virtualenv and I haven't altered any files or directories save for creating /data and downloading the two corpus packs. Any help would be greatly appreciated, thank you :)

      File "lstm_server.py", line 169, in <module>
        app.add_api('lstm_server.yaml')
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/connexion/app.py", line 168, in add_api
        validator_map=self.validator_map)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/connexion/api.py", line 108, in __init__
        validate_spec(spec)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/validator20.py", line 97, in validate_spec
        validate_apis(apis, bound_deref)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/validator20.py", line 310, in validate_apis
        idx=idx,
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/validator20.py", line 243, in validate_parameter
        validate_default_in_parameter(param, deref)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/validator20.py", line 172, in validate_default_in_parameter
        deref=deref,
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/common.py", line 29, in wrapper
        sys.exc_info()[2])
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/common.py", line 24, in wrapper
        return method(*args, **kwargs)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/validator20.py", line 155, in validate_value_type
        validate_schema_value(schema=deref(schema), value=value, swagger_resolver=swagger_resolver)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/swagger_spec_validator/ref_validators.py", line 101, in validate_schema_value
        create_dereffing_validator(swagger_resolver)(schema, resolver=swagger_resolver).validate(value)
      File "/Users/pkarpenko/Documents/Dev/python/LSTMVis/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 130, in validate
        raise error
    swagger_spec_validator.common.SwaggerValidationError: ('\'states,words\' is not of type \'array\'\n\nFailed validating \'type\' in schema:\n    {\'collectionFormat\': \'csv\',\n     \'default\': \'states,words\',\n     \'description\': "list of required data dimensions, like state values (\'states\'), token values (\'words\'),\\n  or meta information (\'meta_XYZ\')\\n",\n     \'in\': \'query\',\n     \'items\': {\'type\': \'string\'},\n     \'name\': \'dims\',\n     \'required\': False,\n     \'type\': \'array\'}\n\nOn instance:\n    \'states,words\'', <ValidationError: "'states,words' is not of type 'array'">)
    bug 
    opened by paulkarpenko 6
  • Python module helper_functions is missing

    Python module helper_functions is missing

    Dear developers, I ran the "install" instructions from the main page and downloaded the parens data as a test. However, I get the following error, indicating a missing module:

    Traceback (most recent call last): File "lstm_server.py", line 8, in <module> from lstmdata.data_handler import LSTMDataHandler File "/home/dnieuwenhuijse/Software/LSTMVis/lstmdata/data_handler.py", line 11, in <module> import helper_functions as hf ModuleNotFoundError: No module named 'helper_functions'

    I couldn't find the module on your github repo either, so I am guessing that you have removed the module from your repo by accident?

    Kind regards,

    David

    opened by dnieuw 4
  • model/get_states.lua:78: bad argument #1 to 'copy'

    model/get_states.lua:78: bad argument #1 to 'copy'

    Getting the above error when I attempt to execute get_states.lua. This is on my own data. Here is what I do before executing get_states.lua First, I download the tiny-Shakespeare dataset in jcjohson's github repo, torch-rnn; and split it into 2 datasets: train_tiny-shakespeare.txt and validation_tiny-shakespeare.txt.

    Then, I run preprocess like so: python model/preprocess.py data/tinyshakespeare/train_tiny-shakespeare.txt
    data/tinyshakespeare/validation_tiny-shakespeare.txt 50 64
    data/tinyshakespeare/convert/tiny-shakespeare

    Then, I run main.lua to train on this data, like so: th model/main.lua -rnn_size 128 -word_vec_size 64-num_layers 2
    -epochs 50 -data_file data/tinyshakespeare/convert/tiny-shakespeare.hdf5
    -val_data_file data/tinyshakespeare/convert/tiny-shakespeareval.hdf5
    -gpuid 0 -savefile cv/tinyshakespeare
    | tee train-tinyshakespeare.log

    Then, I run get_states.lua like so: th model/get_states.lua
    -data_file data/tinyshakespeare/convert/tiny-shakespeare.hdf5
    -checkpoint_file cv/tinyshakespeare_epoch20.00_420.16.t7
    -output_file data/reads/tinyshakespeare_states.h5

    This is where I get the error.

    opened by tlhumphrey2 4
  • error when starting server

    error when starting server

    1. data in /LSTMVis/lstmdata/05childbook
    2. starting server with.... root@b6d19b96355f:/LSTMVis# python server.py -dir /LSTMVis/lstmdata/05childbook File "server.py", line 267 print args ^ SyntaxError: Missing parentheses in call to 'print'

    Any ideas what I'm missing? @HendrikStrobelt Thx!

    opened by Heavy02011 4
  • Clarification on states HDF5 input

    Clarification on states HDF5 input

    Hi All,

    Came across this project and it looks really helpful. I'm looking to use it to explore LSTM in a current project.

    I'm a bit confused about the states.h5 input file. I imagine that the state vector from each timestep is the h_t output of the LSTM, for each t (as in the notation of this LSTM description). Is this correct?

    If so, the states are dependent on the input sequence. So I can put any sequence into the LSTM and then get these hidden states, but wouldn't this tool also need the states from all of my other input sequences for some of the analysis? Or does it only look at one input at a time? Is there an easy way to swap between or compare activations across input sequences?

    Thanks for the clarification, and for contributing this tool!

    opened by b-carter 3
  • Show class labels if available

    Show class labels if available

    I have a tagging task where each token has a bunch of annotations associated with it (the most important being the predicted and actual class). It would be great if the annotations could be shown below the main visualization. This ways, I could more easily see, e.g., which neurons lead to a wrong decision.

    opened by alex-j-j 3
  • KeyError:

    KeyError: "Unable to open object ..." - lstm.yml

    I generated the three files required for running the lstm server. But the main thing which I don't find in the documentation of the tool is what exactly is the path mentioned in lstm yaml file. I'm constantly getting errors in the line path!

    The image below shows my yaml file: image

    Running the server shows the following error: image

    My network only consists of 1 hidden layer. I'm not sure whether I understood that part of visualization right!

    Later I commented out those two lines in states and ran the server again. image

    which shows: image

    Could someone help me out as I don't find enough documentation on what does path exactly meant to do here?

    opened by sachn1 2
  • ModuleNotFoundError: No module named 'helper_functions'

    ModuleNotFoundError: No module named 'helper_functions'

    When I run this execution: python lstm_server.py -dir data

    Getting this error below. Ideas on how to fix?

    Traceback (most recent call last):
      File "lstm_server.py", line 8, in <module>
        from lstmdata.data_handler import LSTMDataHandler
      File "/home/user/Documents/GitHub/LSTMVis/lstmdata/data_handler.py", line 11, in <module>
        import helper_functions as hf
    ModuleNotFoundError: No module named 'helper_functions'
    
    opened by blindfish 2
  • Support for whitespace symbols in the words.dict file

    Support for whitespace symbols in the words.dict file

    If you do character-wise prediction and not word-wise prediction, you might want to predict whitespaces. But the way the words.dict file is parsed right now crashes on whitespaces. This pull request should fix this.

    opened by ernestum 2
  • Make  more generalizable

    Make more generalizable

    Hey guys, Fantastic project!

    As part of my research, I am looking into extending your platform outside of the NLP domain. In other words, I would like to be able to explore the activation states:

    • for n-dimensional vectors rather than word ids
    • with respect to the predicted and target value
    • of any RNN

    Would you be interested in collaborating on something of the kind?

    opened by bottydim 2
  • Move to python3.7

    Move to python3.7

    Small changes to allow moving to Python3.7 (Python2.7 is already depricated in most environments).

    • Requirements adjusted to latest version, and to include "swagger-ui-bundle" per warning thrown at startup.
    • The previous version of the project.config included nested dict_items structures which are not directly serializable in Python3 so a workaround was introduced.
    • Calls to iteritems() were replaced by items() for Python3
    • maps are no longer passed around due to differences between Python2/Python3. Instead we use list(map())
    • Localhost should be "127.0.0.1" for WSL on PC. Not sure if this breaks running on a mac.
    opened by nneophyt 1
  • Parens Dataset not accessible

    Parens Dataset not accessible

    https://github.com/HendrikStrobelt/LSTMVis/issues/42 "Parens Dataset - 10k small - 0.03 GB" https://drive.google.com/file/d/0B3yX0IkfCkLvWUowazhOZHFuSms/view?usp=sharing This dataset is not accessible to my google drive account. When I click the file sharing link, it says that I have no access to this file.

    However, "Children Book - Gutenberg" https://drive.google.com/file/d/0B542UFSlrvMjMHcxWHluNzh3clU/view?usp=sharing I have access to this file. The scale of Children Book is too large for a simple trying.

    opened by zhoyifan 0
  • Bump pyyaml from 5.3.1 to 5.4

    Bump pyyaml from 5.3.1 to 5.4

    Bumps pyyaml from 5.3.1 to 5.4.

    Changelog

    Sourced from pyyaml's changelog.

    5.4 (2021-01-19)

    Commits
    • 58d0cb7 5.4 release
    • a60f7a1 Fix compatibility with Jython
    • ee98abd Run CI on PR base branch changes
    • ddf2033 constructor.timezone: _copy & deepcopy
    • fc914d5 Avoid repeatedly appending to yaml_implicit_resolvers
    • a001f27 Fix for CVE-2020-14343
    • fe15062 Add 3.9 to appveyor file for completeness sake
    • 1e1c7fb Add a newline character to end of pyproject.toml
    • 0b6b7d6 Start sentences and phrases for capital letters
    • c976915 Shell code improvements
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • python2 import connexion will show error! What can i do?

    python2 import connexion will show error! What can i do?

    Excuse me, i ran the command:pip install connexion and download this package. But when i use python2 to import connexion, i found that

    File "lstm_server.py", line 2, in import connexion File "C:\Users\10259.conda\envs\python27\lib\site-packages\connexion_init_.py", line 5, in File "C:\Users\10259.conda\envs\python27\lib\site-packages\connexion\apis_init_.py", line 1, in from .abstract import AbstractAPI # NOQA File "C:\Users\10259.conda\envs\python27\lib\site-packages\connexion\apis\abstract.py", line 32 class AbstractAPI(metaclass=AbstractAPIMeta): ^ SyntaxError: invalid syntax

    I really want to know how to solve this problem. Thanks!

    opened by xlwang1995 3
  • use LSTMVis with OpenNMT

    use LSTMVis with OpenNMT

    Not sure if this is addressed in the doc or elsewhere; I'll close this as soon as possible if I found out...

    I did some development recently with OpenNMT. Is there a way to use this visualization tool for OpenNMT models? Especially since both OpenNMT and LSTMVi are developed by the same lab?

    Thanks in advance!

    opened by anonymous10101010101 2
Releases(2.1)
Owner
Hendrik Strobelt
Updates on Twitter: @hen_str
Hendrik Strobelt
GNNLens2 is an interactive visualization tool for graph neural networks (GNN).

GNNLens2 is an interactive visualization tool for graph neural networks (GNN).

Distributed (Deep) Machine Learning Community 143 Jan 7, 2023
FairML - is a python toolbox auditing the machine learning models for bias.

======== FairML: Auditing Black-Box Predictive Models FairML is a python toolbox auditing the machine learning models for bias. Description Predictive

Julius Adebayo 338 Nov 9, 2022
python partial dependence plot toolbox

PDPbox python partial dependence plot toolbox Motivation This repository is inspired by ICEbox. The goal is to visualize the impact of certain feature

Li Jiangchun 722 Dec 30, 2022
⬛ Python Individual Conditional Expectation Plot Toolbox

⬛ PyCEbox Python Individual Conditional Expectation Plot Toolbox A Python implementation of individual conditional expecation plots inspired by R's IC

Austin Rochford 140 Dec 30, 2022
🎆 A visualization of the CapsNet layers to better understand how it works

CapsNet-Visualization For more information on capsule networks check out my Medium articles here and here. Setup Use pip to install the required pytho

Nick Bourdakos 387 Dec 6, 2022
Logging MXNet data for visualization in TensorBoard.

Logging MXNet Data for Visualization in TensorBoard Overview MXBoard provides a set of APIs for logging MXNet data for visualization in TensorBoard. T

Amazon Web Services - Labs 327 Dec 5, 2022
null 131 Jun 25, 2021
Neural network visualization toolkit for tf.keras

Neural network visualization toolkit for tf.keras

Yasuhiro Kubota 262 Dec 19, 2022
Pytorch implementation of convolutional neural network visualization techniques

Convolutional Neural Network Visualizations This repository contains a number of convolutional neural network visualization techniques implemented in

Utku Ozbulak 7k Jan 3, 2023
Convolutional neural network visualization techniques implemented in PyTorch.

This repository contains a number of convolutional neural network visualization techniques implemented in PyTorch.

null 1 Nov 6, 2021
Interactive convnet features visualization for Keras

Quiver Interactive convnet features visualization for Keras The quiver workflow Video Demo Build your model in keras model = Model(...) Launch the vis

Keplr 1.7k Dec 21, 2022
Visual Computing Group (Ulm University) 99 Nov 30, 2022
A Practical Debugging Tool for Training Deep Neural Networks

Cockpit is a visual and statistical debugger specifically designed for deep learning!

null 31 Aug 14, 2022
Portal is the fastest way to load and visualize your deep neural networks on images and videos 🔮

Portal is the fastest way to load and visualize your deep neural networks on images and videos ??

Datature 243 Jan 5, 2023
Making decision trees competitive with neural networks on CIFAR10, CIFAR100, TinyImagenet200, Imagenet

Neural-Backed Decision Trees · Site · Paper · Blog · Video Alvin Wan, *Lisa Dunlap, *Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah

Alvin Wan 556 Dec 20, 2022
LSTMs (Long Short Term Memory) RNN for prediction of price trends

Price Prediction with Recurrent Neural Networks LSTMs BTC-USD price prediction with deep learning algorithm. Artificial Neural Networks specifically L

null 5 Nov 12, 2021
Sign Language Recognition service utilizing a deep learning model with Long Short-Term Memory to perform sign language recognition.

Sign Language Recognition Service This is a Sign Language Recognition service utilizing a deep learning model with Long Short-Term Memory to perform s

Martin Lønne 1 Jan 8, 2022
PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short-Term Transformer for Online Action Detection".

Long Short-Term Transformer for Online Action Detection Introduction This is a PyTorch implementation for our NeurIPS 2021 Spotlight paper "Long Short

null 77 Dec 16, 2022
The Dual Memory is build from a simple CNN for the deep memory and Linear Regression fro the fast Memory

Simple-DMA a simple Dual Memory Architecture for classifications. based on the paper Dual-Memory Deep Learning Architectures for Lifelong Learning of

null 1 Jan 27, 2022
This is a super simple visualization toolbox (script) for transformer attention visualization ✌

Trans_attention_vis This is a super simple visualization toolbox (script) for transformer attention visualization ✌ 1. How to prepare your attention m

Mingyu Wang 3 Jul 9, 2022