Hello,

Thank you very much for the great work!

In order to evaluate some the Code-ML models, I have two questions regarding the AVATAR paper (https://arxiv.org/pdf/2108.11590.pdf) :

(1) In Section 2: "To train models, we chose a maximum of k (we set k to 3 based on validation performances) solutions in each language to form a maximum of k^2 training examples and consider all the accepted solutions as reference translations for validation and testing." I am wondering how exactly these k^2 pairs are selected during training ? Assume there are 5 candidates in each language (JAVA/Python) for each problem, there are 25 pairs that one can choose from -- when k is set to 3, that means we ought to choose 9 out of these 25 pairs, I don't quite understand how exactly it is done as there are many ways to choose 9 pairs out of 25 pairs. Could you point to where in the code it is done ? Also, to validate and test, does it mean as long as the translated code matches with any one of the python reference targets (could be as many as 5), it should be counted as correct ?

(2) The caption of Table 2 reads: CA stands for Computational Accuracy. Does CA actually stand for Compilation Accuracy ?

Thanks!

Wei

Hi @wasiahmad, I'm trying to evaluate a gpt-2 model with your code. Thus, I run run.py with microsoft/CodeGPT-small-py in pretrain_dir parameter and do_infer. In eval_blue script outputs equal to model(inputs)[1] – these are hidden states of pretrained gpt – and it's a tuple of 12 elements (n_layers) consisting of 2 elements each, and these two have [1, 12, 48, 64]. When it goes to this line past_hidden = [x[:, i:i + 1].expand(-1, beam_size, -1, -1, -1) for x in outputs] an error occurs: TypeError: tuple indices must be integers or slices, not tuple – and it also implies that the shape of each element in outputs should have 5 dimensions. Which corrections should be done in this case?

Hi @wasiahmad, I am using https://github.com/wasiahmad/AVATAR/blob/main/transcoder/run.sh to evaluate transcoder on AVATAR test data (test.java-python.java, test.java-python.java and test.jsonl ), 1699 samples. The scores do not match exactly with the paper. Could you please tell me if I am missing anything.

Also I think the success and error variables are swapped in the below line in format function due to which the current output for Python to Java is Success - 1699, Errors - 0 instead Success - 0, Errors - 1699. In that case the CA is 0 for Python to Java. Please correct me if I am wrong. https://github.com/wasiahmad/AVATAR/blob/7bcabc9c7ff82190e878ed0fd3d93e9eb86fea2e/evaluation/compile.py#L90

Thanks and Regards, Kunal Pagarey

Hi! Sorry to bother you again. I found that there are 5132 geeksforgeeks samples in the whole AVATAR dataset, but only 3411 samples in g4g_functions. How did you select these 3411 samples? Did you filter out the problems in TransCoder-Eval?

Thank you!

Hi!

I downloaded data from AVATAR/data/data.zip and also using script AVATAR/data/download.sh, and it seems that a lot of python functions in the dataset miss new_lines and indentation. For example CodeForces/421/A/solution1.py:

n, a, b = map(int, input().split())athur = map(int, input().split())alex = map(int, input().split()) total = [1] * n for i in alex:    total[i-1] = 2 print(*total)

or CodeForces/981/A/solution1.py:

s=input()c=len(s)for i in range(len(s)-1,0,-1):    k=s[0:i+1]    if(k!=k[::-1]):        print(c)        exit()    c-=1if(c==1):    print("0")

According to my simple heuristic calculation, about 50% of python functions look like this.

Is there way to fix it? Thanks in advance for your help!

----------------------------------- Edit: able to reproduce results using different bs / lr ---------------------------------------------

Hi, I was trying to finetune CodeT5 for java-python translation but cannot reproduce the results. I got BLEU 63.77 / EM 1.88, which is much lower than your reported results: BLEU 67.0 / EM 2.8. The hyper-params are:

TRAIN_BATCH_SIZE=2 GRAD_ACCUM_STEP=16 LR=5e-5 NUM_TRAIN_EPOCHS=20 tokenizer_path=codet5/bpe; source_length=510 target_length=510 --do_eval_bleu

All other hyper-params are as default in https://github.com/wasiahmad/AVATAR/blob/main/codet5/run.sh

Could you give me some suggestions on how to reproduce your results? Thank you!

Hi! I was trying to download the pretrained CodeT5-base and PLBART models by running download.sh, but I got the awk: fatal: cannot open file './cookie' for reading (No such file or directory) error. I also tried to directly open the download link in my web browser but it shows "404 file not found". Are you using the same pretrained model as in huggingface (i.e., Salesforce/codet5-base)?

Also, will you release the finetuned ckpts of those models?

Thank you!

https://github.com/wasiahmad/AVATAR/blob/bc9257091eb5125376eec6fb305faac1b71f9f5c/evaluation/CodeBLEU/parser/DFG.py#L254

Hi, I am working on a project that makes uses of your DFG_java code in the linked file, I found that there is a 'referenced before defined' error in this code. There should be another line define tag=False after flag=False.

I also found that other versions of DFG_java in this repository has the tag=False line defined, so I think this is just a typo. (Also, I didn't make use of DFG_csharp but the DFG_csharp in this file evaluation/CodeBLEU/parser/DFG.py seem to be missing the tag initialization as well.)

Thanks for the good work!

Best

Hi, can you please let me know how you preprocessed your data to run on the Transcoder-DOBF model? A few brief steps would really be helpful.

Thank you in advance!

Hi @wasiahmad , I wonder if it’s possible for you to share the trained models?Since l am in urgent need of evaluating the experimental effects of these models on datasets (such as AVATAR and G4G), including assigned specific codes after translation, but my GPU is in short supply. Thank you in advance!