Just an idea, because I don't think we are supposed to run a ton of 2-5 minute jobs. But it might too complex to setup.

~~work in progress: I didn't test it at all. It is kinda a lot of bash magic, and I'm not strong in bash. I would properly do the same for babi and imdb.~~

Based on https://github.com/AndreasMadsen/python-comp550-interpretability/pull/33

Give that we will be setting riemann_samples=50 which can take 1.5h just to compute, we could cache the importance measure which will benefit the non-recursive case.

Still need to set up the slurm job dependencies and make sure it actually works.

I need the datasets to be subclasses for TorchScript support, and we will anyway need it if we add more datasets. Unfortunately the cache will have to be rebuild, becuase this changes some of the .pkl file formats and filenames.

• This fixes #14.
• This sets the seed for some of the split generation. That could properly be done better, but at least it is set.
• For babi it just hardcodes in the labels, instead of computing them dynammically. That simplifies things a lot. I noticed there were some odd labels, I don't know if they are actually used. But may be worth looking into.
• If the encode dataset and vocab exists, it will not build any of the intermediate files. Should make syncing less expensive.
• The test dataset now uses the test dataset instead of the validation dataset. (Recently introduced bug)

~~work in progress: Everything should work. I ran a few iterations locally and rebuild the cache. I need to rerun experiments on compute-canada to make sure it works completely.~~

Scripts for quantitatively evaluating the sparsity of attention.

The general idea is to compute the number of tokens that make up the largest 95% of the attention mass (i.e. the smallest number of tokens that have a cumulative mass of 0.95).

Other ideas for quantifying the sparsity of attention:

• Compute Gini index for each attention distribution (from each example) in each dataset and aggregate.

This should work. I've tested batch_size=8 aganist batch_size=1 and get identical results.

~~What is left is to run this on compute-canada and make sure there is no out-of-memory issues. Also, maybe compute_batch_size should just be base_dataset.batch_size. I will test that too.~~

Jobs to pick a suitable Riemann sample size for integrated gradients.

~~Branched from #17. I believe the merge order will now be #18 -> #17 -> #24.~~

~~Based on https://github.com/AndreasMadsen/python-comp550-interpretability/pull/22, which will need to land first.~~

Properly the last fixes I will make before submitting to compute-canada

Finally managed to make some significant progress on the performance issues:

• Use TorchScript for ROAR computation. Only makes a small difference.
• Load and save list instead of torch.tensor in the ROAR .pkl. This makes a large difference for the garbage collector. For some reason loading torch.tensor via pickle does not seam to be GC'ed correctly.
• Use pin_memory=True in dataloader. Appears to make training a bit faster and is generally recommended for performance on CUDA devices.
• Compute gradient wrt. embedding instead of one_hot encoding. This makes a huge difference in both memory and performance. Constructing the one_hot encoding took up an enormous amount of memory and had to be done on the CPU, hence transfering it to the GPU was also slow. The new version gives identical gradients, but computes gradients over the embedding layer manually.

I've updated the batch_jobs allocated time and also annotated how much time each job actually took. Looks like we could reduce the memory allocation too, but it shouldn't really matter since 1GPU=32GB RAM on Beluga anyway.

~~I sent a support request to make torchtext=0.9.1 available and fix sklearn for python 3.8, this needs to happen first. Before this can land.~~

This removes 'en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz' from setup.py. It is still a dependency but it's such a pain to deal with, it's properly best to just have it installed with python -m spacy download en_core_web_sm locally.

Creates a script for each cluster and cpu/gpu. There is then a bash function that detects which to use. There is very little difference, so maybe there is a better way. But this was the simplest I could think of now.

Note there is no python_cedar_cpu_job.sh because we don't have an allocation on cedar. We could add a script that just uses the default allocation.

Features:

• [x] Implemented Gradient Importance Measure
• [x] Implemented recursive ROAR
• [x] Merged ROAR and non-roar experiment scripts
• [x] Compute Canada integration

Issues:

• ~~Times too short for gradient measure (IMDB and Babi-3).~~
• Mimic vocab size and embedding size don't match (diabetese: 18881 vs 18883, anemia: 16261 vs 16263). This directly prevents gradient calculations. The underlying bug properly affects MIMIC results a bit.
• Checkpoints not found in the recursive case, even though the files are there (Mimic-d, IMDB, Babi). Kinda confused why only some job categories suffer.

This is minor issue and can potentially be dealt with later. This is for documentation.

When uncompressing some gzipped CSVs (as done in export/riemann_samples.py), the gzip module sometimes throws a BadGzipFile exception. This looks to be caused by "trailing garbage" that is appended to the end of some gzip files. Additional information on this issue can be found here:

• https://stackoverflow.com/questions/4928560/how-can-i-work-with-gzip-files-which-contain-extra-data
• https://bugs.python.org/issue24301#msg245369

To reproduce this issue, execute the following on Beluga:

import pandas as pd

pd.read_csv("/scratch/anmadc/shared/babi-2-pre_s-0_m-i_rs-50.csv.gz", compression="infer")

which should output:

BadGzipFile: Not a gzipped file (b'\x17\xc2')

For now, I've avoided this issue by manually uncompressing /scratch/anmadc/shared/babi-2-pre_s-0_m-i_rs-50.csv.gz using gunzip. Some of the MIMIC files may have this issue.

must do

• [ ] Remove reference to compute canada. i.e. delete batch_jobs, *_job.sh, and Makefile from zip.
• [x] Rename comp550 to nlproar or something similar.
• [ ] Anonymize setup.py in zip.

should do

• [x] update doc-strings