作者你好，论文提供的思路对我启发很大!

我有两个问题想请教一下:

1. 关于训练的pipeline

我看到论文中描述的训练流程只包含language-free和language-aware两个环节，类似于代码中的LF_2和LA，但代码中还额外增加了LF_1来专门预训练backbone + VRM部分，并且在LF_2过程的optimizer中还针对LF_1训练过的params采用了不同的lr。请问LF_1 --> LF_2 --> LA和LF_2 --> LA两种训练方式差别大吗？

2. 关于并行解码环节的attention计算方式

问题是我对实现逻辑不太理解

这里我和SRN中视觉部分(PVAM)中的attention过程作对比:

(a) SRN-PVAM中的attention过程(伪代码，假设qkv的维度都是d_model):

# e.g. d_model = 512, max_seq_len = seq_len_q = 25, vocab_size = 37
key2att = nn.Linear(d_model, d_model)
query2att = nn.Linear(d_model, d_model)
embedding = nn.Embedding(max_seq_len, d_model)
score = nn.Linear(d_model, 1)
classifier = nn.Linear(d_model, vocab_size)

# input is encoder_out
reading_order = torch.arange(max_seq_len, dtype=torch.long)
Q = embedding(reading_order)  # (max_seq_len, d_model)
K, V = encoder_out  # (batch_size, seq_len_k, d_model)

# 这里计算att_weight的过程很容易理解，和经典的attention模型比如ASTER的attention过程相同
######
att_q = key2att(Q).unsqueeze(0).unsqueeze(2)  # (1, seq_len_q, 1, d_model)
att_k = query2att(K).unsqueeze(1)  # (batch_size, 1, seq_len_k, d_model)
att_weight = score(torch.tanh(att_q + att_k)).squeeze(3)  # (batch_size, seq_len_q, seq_len_k)
######

att_weight = F.softmax(att_weight, dim=-1)
decoder_out = torch.bmm(att_weight, K)  # (batch_size, seq_len_q, d_model)
logits = classifier(decoder_out)  # (batch_size, seq_len_q, vicab_size)

(b) VisionLAN中的attention过程:

# e.g. d_model = 512, max_seq_len = seq_len_q = 25, vocab_size = 37
embedding = nn.Embedding(max_seq_len, d_model)
w0 = nn.Linear(max_seq_len, seq_len_k)
wv = nn.Linear(d_model, d_model)
we = nn.Linear(d_model, max_seq_len)
classifier = nn.Linear(d_model, vocab_size)

# input is encoder_out
K, V = encoder_out  # (batch_size, seq_len_k, d_model)
reading_order = torch.arange(max_seq_len, dtype=torch.long)

# 如何理解下面这段计算att_weight的代码?
#####
reading_order = embedding(reading_order)  # (seq_len_q, d_model)
reading_order = reading_order.unsqueeze(0).expand(K.size(0), -1)  # (batch_size, seq_len_q, d_model)
t = w0(reading_order.permute(0, 2, 1))  # (batch_size, d_model, seq_len_q) ==> (batch_size, d_model, seq_len_k)
t = torch.tanh(t.permute(0, 2, 1) + wv(K))  # (batch_size, seq_len_k, d_model)
att_weight = we(t)  # (batch_size, seq_len_k, d_model) ==> (batch_size, seq_len_k, seq_len_q)
att_weight = att_weight.permute(0, 2, 1)
######

att_weight = F.softmax(att_weight, dim=-1)
decoder_out = torch.bmm(att_weight, K)  # (batch_size, seq_len_q, d_model)
logits = classifier(decoder_out)  # (batch_size, seq_len_q, vicab_size)

期待你的回复，谢谢！

How many images do you use in Benchmark Datasets IIIT5K, SVT, IC13, IC15, SVTP, and CUTE80 respectively? I am confused with the 4832 images you mentioned in your paper. Thank you!

Thanks for your code. I found my evaluation results are different from the results post in README.md. I run the provided models and eval.py, and I get zero accuracy on all datasets.

I used the test datasets and models that you provided on Ruike.

The command I executed:

python eval.py

The result I get:

------Average on 6 benchmarks--------                                                                                                          
                                                                                                                                               
test accuracy:                                                                                                                                 
Accuracy: 0.000000, AR: 0.815073, CER: 0.184927, WER: 1.000000, best_acc: 0.000000                                                             
------IIIT--------                                                                                                                             
                                                                                                                                               
test accuracy:                                                                                                                                 
Accuracy: 0.000000, AR: 0.821556, CER: 0.178444, WER: 1.000000, best_acc: 0.000000                                                             
------IC13--------                                                                                                                             
                                                                                                                                               
test accuracy:                                                                                                                                 
Accuracy: 0.000000, AR: 0.846731, CER: 0.153269, WER: 1.000000, best_acc: 0.000000                                                             
------IC15--------                                                                                                                             
                                                                                                                                               
test accuracy:                                                                                                                                 
Accuracy: 0.000000, AR: 0.790097, CER: 0.209903, WER: 1.000000, best_acc: 0.000000
------SVT--------

test accuracy: 
Accuracy: 0.000000, AR: 0.831195, CER: 0.168805, WER: 1.000000, best_acc: 0.000000
------SVTP--------

test accuracy: 
Accuracy: 0.000000, AR: 0.797383, CER: 0.202617, WER: 1.000000, best_acc: 0.000000
------CUTE--------

Hope for your response~

Hello! It is an excellent work that inspires me a lot ;D As I read your code, I found something strange (maybe a typo):

train_LA.py

line145: text_pre, test_rem, text_mas, att_mask_sub = model(data, label_id, cfgs.global_cfgs['step'])

Should "test_rem" be modified to "text_rem"?

Thanks for sharing the code and data of your amazing work; I tried to download the OST datasets from both Baidu and Ruike, but I could not. It needed login. Can you please upload on google drive or share with me the link that I can download directly?

感谢您的代码，运行“CUDA_VISIBLE_DEVICES=0 python eval.py”命令后，报出如下错误： Traceback (most recent call last): File "eval.py", line 11, in import cfgs.cfgs_eval as cfgs File "/workspace/VisionLAN-main/cfgs/cfgs_eval.py", line 5, in from data.dataset_scene import * File "/workspace/VisionLAN-main/data/dataset_scene.py", line 16, in from transforms import CVColorJitter, CVDeterioration, CVGeometry ModuleNotFoundError: No module named 'transforms' 看起来似乎是环境的问题，但我的环境是严格按照requirements.txt配置的，这是怎么回事呢？热切盼望您的解答

Thanks for your excellent contributions! I try to use your pre-trained LF_2 model to visualize the mask map, I pick the same image that was shown in the Visualization character-wise mask map (P=0) of ReadMe. But I got a different result from you. I resize the mask map to the original height and width and add it with the original image and got the result like this:

Then I just resize the mask map to the original height and width and directly visualize the mask map and got a result like this:

It seems that the above two visualizations are different from yours, was there a problem?

Dear yuxin, sorry to bother you again. When I use your code, I found two new questions: 1. When I executed python train_LF_1.py, I got a CUDA error in ClassNLLCriterion.cu. 2. When I modify the code into Chinese training, the model could not converge.

Question 1: CUDA error in ClassNLLCriterion.cu.

The error info:

THCudaCheck FAIL file=/pytorch/aten/src/THC/generic/THCTensorMath.cu line=29 error=710 : device-side assert triggered                          
/pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:108: cunn_ClassNLLCriterion_updateOutput_kernel: block: [0,0,0], thread: [12,0,0] Assertion `t >= 0 && t < n_classes` failed.

solution:

The problem seems like a bug (My PyTorch version is 1.71). When I modified the nclass from 37 to 38, the problem is gone. I think 38 is reasonable: 36 normal chars, 1 , and 1 . I modified these two lines:

VisionLAN.py
71:        self.Prediction = Prediction(n_position=256, N_max_character=26, n_class=37) # N_max_character = 1 eos + 25 characters
72:        self.nclass = 37

Question 2: Chinese training is failed.

I modified the codes for Chinese training, but the model could not be coverged. The loss drops very slowly. Do you modify the training config when training TRW15?