Hello!

Now I am trying to inference the trained model, but I found mismatch dimension error. Here are the full logs:

/opt/conda/lib/python3.7/site-packages/transformers/models/t5/tokenization_t5_fast.py:166: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.
For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.
- Be aware that you SHOULD NOT rely on t5-small automatically truncating your input to 512 when padding/encoding.
- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.
- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.
  FutureWarning,
/opt/conda/lib/python3.7/site-packages/pytorch_lightning/core/lightning.py:23: LightningDeprecationWarning: pytorch_lightning.core.lightning.LightningModule has been deprecated in v1.7 and will be removed in v1.9. Use the equivalent class from the pytorch_lightning.core.module.LightningModule class instead.
  "pytorch_lightning.core.lightning.LightningModule has been deprecated in v1.7"
Downloading: 100%|███████████████████████████| 231M/231M [00:04<00:00, 55.9MB/s]

--------------------------------------------------------------------------------
./transformers-into-vaes/finetune.py 280 <module>
use_cache=True,

/kaggle/working/transformers-into-vaes/generate.py 89 generate
sampled_z=sampled_z,

.../conda/lib/python3.7/site-packages/torch/nn/modules/module.py 1110 _call_implreturn forward_call(*input, **kwargs)

/kaggle/working/transformers-into-vaes/vendor_t5.py 142 forward
return_dict=return_dict,

.../conda/lib/python3.7/site-packages/torch/nn/modules/module.py 1110 _call_implreturn forward_call(*input, **kwargs)

...ib/python3.7/site-packages/transformers/models/t5/modeling_t5.py 1044 forwardoutput_attentions=output_attentions,

.../conda/lib/python3.7/site-packages/torch/nn/modules/module.py 1110 _call_implreturn forward_call(*input, **kwargs)

...lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py 671 forwardoutput_attentions=output_attentions,

.../conda/lib/python3.7/site-packages/torch/nn/modules/module.py 1110 _call_implreturn forward_call(*input, **kwargs)

...lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py 577 forwardoutput_attentions=output_attentions,

.../conda/lib/python3.7/site-packages/torch/nn/modules/module.py 1110 _call_implreturn forward_call(*input, **kwargs)

...lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py 500 forwardhidden_states, self.k, key_value_states, past_key_value[0] if past_key_value is not None else None

...lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py 489 projecthidden_states = torch.cat([past_key_value, hidden_states], dim=2)

RuntimeError:
Sizes of tensors must match except in dimension 2. Expected size 64 but got size 1 for tensor number 1 in the list.

It seems the problem occurs on decoding phase, then goes deep into the internal class of T5. I am trying to figure out what's the problem but I can't find it. Could you please help to check with this issue?

Thank you very much!)

Hello again! Thank you for the great code. As stated in the paper, there is a KL Thresholding technique. But, I feel lost looking for this function in the code. Could you please point out where I could find it?

Thank you very much!

Hello, first of all thanks for your amazing work! I am trying to barely run your code in Google Colab but got the training loss always nan when starting pretraining encoder. Among 15 epochs, it just stop after 1st epoch as the EarlyStopCallback noticed that the loss keep infinite since the beginning. If it possible, could you please check to reproduce my errors?

My environment:

• Transformers == 4.23.1
• Pytorch Lightning == 1.7.7

Looking forward for your response, thank you)

Hello, thank you very much for making the code available. I'm confused about the mutual information math, more specifically about the line

E_{q(z|x)}log(q(z|x)) = -0.5nzlog(2*\pi) - 0.5*(1+logvar).sum(-1) neg_entropy = (-0.5 * nz * math.log(2 * math.pi) - 0.5 * (1 + logv).sum(-1)).mean()

When I derive it, it gives me neg_entropy = (-0.5 * nz * math.log(2 * math.pi) - 0.5 * (logv).sum(-1)).mean()

So I think I must have made a mistake somewhere? Thank you

Thanks for this nice work and reproducible code. If I understood your approach well, I think I am missing the code line that you passed latent space z to decoder as encoder_hidden_states. I will be glad if you point that line. Thanks.