Hi Yikang and other Contributors,

Thank you for making public the source code! I am trying to reproduce your results, but I am not sure what path to use as the command line argument of test_phrase_grammar --data. I downloaded PTB data and I am currently using treebank_3/parsed/mrg as the data argument. It does not work.

The listings under treebank_3/parsed/mrg: atis brown readme.mrg swbd wsj

The listings under treebank_3/parsed/mrg/wsj:

00 06 12 18 24 01 07 13 19 MERGE.LOG 02 08 14 20 03 09 15 21 04 10 16 22 05 11 17 23

Thank you for your time! Ian

In your paper, you use a unidirectional ON-LSTM to trained a language model and then phrase grammar with the output distance of the pretrained language model. How can we explain that the level of first token is independent with the future tokens? Is there any bidirectional way to do it?

Hello Yikang:

Hi~, I'm a research intern of HIT-SCIR lab, Yangming Li. It's great for your contribution about this repository. But I found some problems about the dataset construction (including test set):

1, the use of pytorch API "narrow" will unexpectedly abandon some words and result in incorrect PPL score.

2, It seems that your slide window on the whole corpus is not continuous and thus generate far less data than usual.

Great thanks again for your contribution about this repository. Yangming, 19/08/02

Hi I wonder how to train parsing. The main.py seems to be only for training LM.

Besides, when I try testing the parsing, python test_phrase_grammar.py --cuda gives error of No such file or directory: 'PTB.pt'.

Best regards, Ron

Hello, I am interested in ON-LSTM and I want to reimplement it under Keras.

I trained a Chinese language model with ON-LSTM and then export the distance (as you do in https://github.com/yikangshen/Ordered-Neurons/blob/master/ON_LSTM.py#L89 ). However, I found all elements of distance is quite close to 1 (0.995, 0.99, 0.98, ...).

Is it a normal phenomenon in your experience ?

Really appreciate for releasing the code.

I found when I testing the baseline of right-branching strategy on WSJ test set, the F1 is really high (39.87), which does not match the result in the paper (16.5).

I have just changed the code

distance = model.distance[0].squeeze().data.cpu().numpy()
distance_in = model.distance[1].squeeze().data.cpu().numpy()

into

distance = numpy.array([numpy.arange(len(sen), 0, -1)] * 3)
distance_in = numpy.array([numpy.arange(len(sen), 0, -1)] * 3)

, which represent a right-branching strategy.

And the result on WSJ test set is:

So, what my be the reason? Thanks a lot if u could help me out.

Dear Yikang,

I am new to NLP but I really like this paper and appreciate your work. Now I am reading the paper and have a question about Eq. 15 in section 5.2. What does y_t mean? Thank you!

The hyperlink of "Penn Treebank Parsed" in ReadMe.md is broken and I can not find the correct download link. Is this the same? I found this link on the page NLTK data.

Hi, Yikang thanks a lot for this awesome paper!

When I try to run the below command,

python main.py --batch_size 20 --dropout 0.45 --dropouth 0.3 --dropouti 0.5 --wdrop 0.45 --chunk_size 10 --seed 141 --epoch 1000

Such error triggered at certain(5th) epoch:

File "main.py", line 269, in train() File "main.py", line 245, in train elapsed * 1000 / args.log_interval, cur_loss, math.exp(cur_loss), cur_loss / math.log(2))) OverflowError: math range error

My initial found is that the cur_loss suddenly increases to a larger number(from ~5 to more than 10000), which results in such error. However, I am not sure what cause this sudden huge increment.

It seems that there are two evaluation result in test_phrase_grammar.py(one comes from evalb software, the other is computed by yourself), which is reported in the paper? what is the difference between them?

I have a masked LM pretrained with bert.

The embeddings are poor on the sentence level, but do well for base tokens. There is a natural tree structure to my corpus that I believe stands to gain from something like on-lstm.

Do you think swapping out the embedding layer of the on-lstm with pretrained bert embeddings could be fruitful?

Hi, thanks for sharing the source code.

According to the Equation (10) in your paper, I guess the last elements of $\tilde{i}_t$ will always be zero, e.g., [0.8, 0.3, 0.1, 0]. Is this on purpose? If yes, could you please explain why? I just think this will let the upmost neuron chunk keep copying history without writing in anything new, is this correct?

Hi Yikang,

If I want to reproduce your work, what parameters should I use?

In the readme, you suggest use the default parameters in main.py. At the same time, you provide another set of parameters: "python main.py --batch_size 20 --dropout 0.45 --dropouth 0.3 --dropouti 0.5 --wdrop 0.45 --chunk_size 10 --seed 141 --epoch 1000 --data /path/to/your/data".

Which one should I use? I tried both ones and after 48 hours, the quoted parameters outperform the default one, so I would like to double-check with you.

Thanks, Ian

I install Pytorch0.4 which choose cuda none.And after i run the main.py ,i got a error：torch.cuda.LongTensor is not enabled.，which i haven't found a simillar problem online.whether i need use a computer with GPU and CUDA?

Hi,when I run the test_phrase_grammar.py ,I will get this return like following:

ZeroDivisionError: float division by zero

This is the specific error:

I'm sorry to bother you that when I try to test this model, I have no where to download the checkpoint. With the link you have proved ,I only found the ‘.txt’ files, where can I download the 'PTB.pt'

FileNotFoundError: [Errno 2] No such file or directory: 'PTB.pt'