Hi,

Thank you for this very clean repository! I wonder whether any checkpoints will be released? If yes, I'm interested in adding those to the HuggingFace Transformers library (the Reformer and LongFormer are already there, but the Linformer and the Nystromformer aren't yet). Since we only have to replace the self-attention mechanism, this seems quite straightforward in terms of modeling.

Kind regards,

Niels

When the input of iterative_inv is a 4-tensor (which is how it is used in NystromAttention), then the calculation of of the initial value of V in iterative_inv here takes the max over all dimensions of the 4-tensor K rather than just the matrix dimensions as claimed in lemma 1:

This has a significant effect on the accuracy of the approximation on random data I've tried it on.

Hello,

I'm unclear on the pretraining procedures, in particular the preprocessing of the datasets.

Unless I'm mistaken https://github.com/mlpen/Nystromformer/blob/main/data-preprocessing/preprocess_data_512.py suggest we just put segments of size 512. I'm not sure I understand how SOP is defined in this case? Actually SOP doesn't seem to be used at all in the pretrain script.

Hello, first of all great work! This looks very promising.

As I'm reading a bit more in depth, I'm wondering if you might have an error on parenthesis in lemma 1. Both the code, and the proof seem to indicate it.

The last term should apply only to AZ instead of (15I - AZ) no?

This would "fix" the following lines in the proof

Hi, I saw the tweet about this paper and wanted to know the performance of Nystromformer on each individual task in Long Range Arena. (Also what is the "Standard" in the graph posted? Standard transformer does much worse in LRA)

Hello, I noticed that in your paper, Table 3 shown the Text (4K). However, max_length was set to 1000 in your code LRA/datasets/text.py. Is the Text task length 4K or 1K for the scores given in Table 3?

Thanks for your great work.

I noticed that in the paper and also in this repo, you report the results on LRA tasks. I am trying to train your model on LRA with the code given in this repo. For the Pathfinder task, I follow the dataset preparing guidance and I get 3 sets of pickle files, e.g. pathfinder32-curv_baseline, pathfinder32-curv_contour_length_9, and pathfinder32-curv_contour_length_14, each with train/dev/test splits.

I would like to know that which set gives the 70.94 result in table 3.

Hi, thanks for sharing the code.

I have read your paper recently and I have some questions about some details. I would be very grateful if you can help me with those questions.

1. How do you derive eqn.(4) ? As far as I know, Nystrom method only applies to semidefinite matrix, not the product of matrices, let alone the softmax of the matrix product. I do read through the reference above eqn.(4) ``Improving CUR Matrix Decomposition and the Nystr¨om Approximation via Adaptive Sampling'', but I failed to find any relevant results. Could you please point to me where can I find these results?

2. The other question is about the key simplification, which first uses Nystrom method inside the softmax then commutes the matrix production and the softmax operation (calcuate softmax first and then the matrix production). While the first step (Nystrom) is indeed unbiased, the second step (commute operations) is not necessary the case. In summary, it’s hard to say what the final equation (13) really is, and why it works at all.

When trying to run python3 cifar10.py to prepare the cifar10 dataset, the script gets into an infinite loop because input_pipeline.get_cifar10_datasets returns a dataset with infinite sample (it uses the repeat function). What do you think I should do to fix it?

First, thank you for sharing this project with us!

Could you please add an explicit LICENSE file to the repo so that it's clear under what terms the content is provided, and under what terms user contributions are licensed?

Per GitHub docs on licensing:

[...] without a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work. If you're creating an open source project, we strongly encourage you to include an open source license.

Thanks!

Hi,

Thanks for the excellent work!

I found some issues in my humble trials (I didn't change anything in the code):

1. using softmax attention on Text4k I got ~63.7 acc instead of 65.02 you posted in your paper.
2. again I tried linear attention Text4k I got ~64 acc, it's even higher than vanilla transformer, I wonder did you get the same result from your side?
3. the attention types linformer-256 and nystrom-64 doesn't work, the errors are either dimensions mismatching or config key error. It seems like not all the attention types can successfully run when you release the code. Btw I didn't try out all the choices.

Thank you for your time, I look forward to your reply~

Ziwei

Hi all,

Nice work. I have a question however. I see with in my problem setting that the self-attention (SA) weights don't always sum to ~1 (row-wise). I assume this is due to the out-of-sample approximation and structural properties of the true SA matrix. Are there ways to reduce the error and obtain a more accurate approximation or other ways to account for this? I myself will try with more landmarks and see how the dynamics change.

Thanks in advance,

Dennis

I wonder why the Retrieval accuracy is almost 20% higher than the official JAX/FLAX implementation. As the paper says, "While we achieve consistent results reported in (Tay et al. 2020) for most tasks in our PyTorch reimplementation, the performance on Retrieval task is higher for all models following the hyperparameters in (Tay et al. 2020)." Is there any difference aside from the hyperparameters?