Hi, jingtao

I find that the adore model released is not defined in your code.

The config.json file indicates that the model architecture is "RobertaDot_NLL_LN", however , it seems not defined in model.py.

Hi，jingtao

When I train the STAR model. There exist an error.

File "./dataset.py", line 176, in getitem hardpids = random.sample(self.rankdict[str(qid)], self.hard_num) KeyError: '18337'

The valid keys of rankdict should be 1,2,...,6980. Am I right?

您好，感谢您开源代码！

我在尝试运行的时候发现，您在README里提到，transformers的版本要使用2.8.0，因为3以上的版本里tokenizer的行为不一致；但是在setup.py文件中，却指定了 transformers==3.4.0，这是为什么呢？我应该使用哪个版本的transformers库呢？

Thanks for your released beautiful and easy-to-follow code ! It is very helpful to the dense retrieval researchers as me.

However, maybe I have met a bug in the preprocess.py. As the model requires to use the RoBERTa-base model for initialization, the native pad token is 1 but not 0. While I found that in this file, the pad token is set to 0 (the [CLS] token id), which has caused that the pre-trained model checkpoint can not achieve the same results as reported in your paper. https://github.com/jingtaozhan/DRhard/blob/main/preprocess.py#L23

In https://github.com/jingtaozhan/DRhard/blob/dc17f3d1f7f59d13d15daa1a728dc8d6efc48b92/dataset.py, if we take a look at the data collator,

def get_collate_function(max_seq_length):
    cnt = 0
    def collate_function(batch):
        nonlocal cnt
        length = None
        if cnt < 10:
            length = max_seq_length
            cnt += 1

        input_ids = [x["input_ids"] for x in batch]
        attention_mask = [x["attention_mask"] for x in batch]
        data = {
            "input_ids": pack_tensor_2D(input_ids, default=1, 
                dtype=torch.int64, length=length),
            "attention_mask": pack_tensor_2D(attention_mask, default=0, 
                dtype=torch.int64, length=length),
        }
        ids = [x['id'] for x in batch]
        return data, ids
    return collate_function  

we see that there is a cnt variable which is deciding if the collate_function should pad or not. I couldn't get why it is needed. Could you please explain the significance of cnt ?

Thank you AM

Hi Jingtao,

I try to reproduce the results showing in the README. The models are downloaded from google drive. For the transformers version, preprocessing is 2.8.0 and for inference is 4.8.2.

I ran the following commands: python ./star/inference.py --data_type passage --max_doc_length 256 --mode dev
python ./msmarco_eval.py ./data/passage/preprocess/dev-qrel.tsv ./data/passage/evaluate/star/dev.rank.tsv

And I got the following results: Eval Started ##################### MRR @10: 0.010382669304589082 QueriesRanked: 6980 #####################

Could you help to figure out what I did wrong? Thanks!

There is an ImportError when I am trying to replicate your work

$python ./star/inference.py --data_type passage --max_doc_length 256 --mode dev
Traceback (most recent call last):
  File "./star/inference.py", line 15, in <module>
    from model import RobertaDot
  File "/home/yicheng.fyc/DRhard/./model.py", line 10, in <module>
    from transformers.modeling_roberta import RobertaPreTrainedModel
ImportError: cannot import name 'RobertaPreTrainedModel' from 'transformers.modeling_roberta' (/home/yicheng.fyc/miniconda2/envs/adore/lib/python3.8/site-packages/transformers-2.8.0-py3.8.egg/transformers/modeling_roberta.py)

It happens for both 2.8.0 and 3.4.0 of transformers. upgrade transformers to 4.2 will fixed this problems but it will leads a huge gap between the MRR index.

Hello,

I have read your paper and am quite interested in your work! There is a question about the tokens. I notice you truncat the passage tokens with 120 in MSMARCO Passage Retrieval, however, for ANCE, the original paper uses 512 tokens. So does the number of tokens have the impact on the accuracy?

Hi,

I can't find the instruction to replicate the nDCG performance on TREC 19. Could you tell me how to run the evaluation on TREC 19 test set.

Thanks.

Hello, I found the result of proviededi nbatch-neg model on test dataset is so bad. Is TREC DL Passgae data a test dataset?
What I should do to reproduce the NDCG@10 and R@100 on TREC DL dataset?

• Command: python ./msmarco_eval.py ./data/passage/preprocess/test-qrel.tsv ./data/passage/evaluate/download_inbatch/test.rank.tsv

• Results: Eval Started ##################### MRR @10: 0.04559967102039234 QueriesRanked: 43 #####################

Is the generating of passage_embeddings of this program the same as the RepBERT program?:

python precompute.py --load_model_path ./data/ckpt-350000 --task doc python precompute.py --load_model_path ./data/ckpt-350000 --task query_dev.small python precompute.py --load_model_path ./data/ckpt-350000 --task query_eval.small

Hi, thank you for publishing the code for your interesting paper. I was just trying to reproduce STAR results in ANCE setup, i.e. I am using static hard negatives and in-batch negatives. But I am unable to achieve an MRR@10 score of 0.34. Also, the STAR checkpoint provided in this repo is not producing MRR@10 result of 0.34 when evaluated using ANCE repo. I am getting MRR@10 of 0.299 instead. I see there are some differences in the training setups in your repo and the ANCE one. Can you please highlight those?

Hi, I am trying to understand how you combined the hard negative loss L_s with the in-batch random negative loss L_r, as in the paper the in-batch random negative loss is scaled by an alpha hyperparameter but there is no mention of the value of alpha you used in the experiments.

Following star/train.py I found the RobertaDot_InBatch model, whose forward function calls the inbatch_train method.

A the end of the inbatch_train method (line 182), I found

return ((first_loss + second_loss) / (first_num + second_num),)

which is different from the combined loss proposed in the paper (Eq. 13).

Am I missing something?

Also, for each query in the batch, did you consider all the possible in-batch random negatives or just one?

Thanks in advance!