Hi. Thanks for sharing the great project. I'm new to this and tried the first example script. Training seems fine but testing fails as follows. Any idea?

• Ubuntu 20.04
• Python 3.8.10

Thanks

(venv) me@DESKTOP-FCVGD1U:~/workspace/github/ULTRA_pytorch$ bash example/toy/offline_exp_pipeline.sh
Reading data in ./example/toy/data/
Read data from ./example/toy/data//train in libsvm format.
Reading finish: 0 lines
Remove 7 invalid queries.
Data reading finish!
Finished reading 20 queries with lists.
Read data from ./example/toy/data//valid in libsvm format.
Reading finish: 0 lines
Remove 3 invalid queries.
Data reading finish!
Finished reading 6 queries with lists.
Train Rank list size 9
Valid Rank list size 9
Users can only see the top 9 documents for each query in training.
Creating model...
Build DLA

Loss Function is softmax_loss
Created model with fresh parameters.
Create simluated clicks feed

Create direct label feed with list size 9 with feature size 136
max_train_iter:  10
 Loss 4.346205 at Global Step 0: 
 Loss 6.528031 at Global Step 1: 
 Loss 8.008065 at Global Step 2: 
 Loss 4.675643 at Global Step 3: 
 Loss 8.523232 at Global Step 4: 
 Loss 4.580126 at Global Step 5: 
 Loss 5.745630 at Global Step 6: 
 Loss 4.260286 at Global Step 7: 
 Loss 4.781512 at Global Step 8: 
 Loss 4.347456 at Global Step 9: 
 Loss 4.397479 at Global Step 10: 
 Loss 4.239964 at Global Step 11: 
 Loss 4.159795 at Global Step 12: 
 Loss 5.694969 at Global Step 13: 
 Loss 4.179561 at Global Step 14: 
 Loss 4.300393 at Global Step 15: 
 Loss 4.171987 at Global Step 16: 
 Loss 4.124520 at Global Step 17: 
 Loss 5.438332 at Global Step 18: 
 Loss 4.089553 at Global Step 19: 
 Loss 5.418541 at Global Step 20: 
 Loss 4.175595 at Global Step 21: 
 Loss 4.735269 at Global Step 22: 
 Loss 4.107638 at Global Step 23: 
 Loss 4.677073 at Global Step 24: 
 Loss 4.106503 at Global Step 25: 
 Loss 4.410158 at Global Step 26: 
 Loss 4.117153 at Global Step 27: 
 Loss 4.603172 at Global Step 28: 
 Loss 4.095118 at Global Step 29: 
 Loss 4.604589 at Global Step 30: 
 Loss 4.101539 at Global Step 31: 
 Loss 4.362749 at Global Step 32: 
 Loss 4.077655 at Global Step 33: 
 Loss 4.241923 at Global Step 34: 
 Loss 4.122931 at Global Step 35: 
 Loss 4.558557 at Global Step 36: 
 Loss 4.125799 at Global Step 37: 
 Loss 4.322569 at Global Step 38: 
 Loss 4.135751 at Global Step 39: 
 Loss 4.521419 at Global Step 40: 
 Loss 4.149001 at Global Step 41: 
 Loss 4.357186 at Global Step 42: 
 Loss 4.099861 at Global Step 43: 
 Loss 4.441446 at Global Step 44: 
 Loss 4.093635 at Global Step 45: 
 Loss 4.306568 at Global Step 46: 
 Loss 4.094622 at Global Step 47: 
 Loss 4.529568 at Global Step 48: 
 Loss 4.184571 at Global Step 49: 
global step 50 learning rate 0.0500 step-time 0.13 loss 4.6094
mrr_3 tensor(0.5972, device='cuda:0')
mrr_5 tensor(0.5972, device='cuda:0')
mrr_10 tensor(0.5972, device='cuda:0')
ndcg_3 tensor(0.4604, device='cuda:0')
ndcg_5 tensor(0.5386, device='cuda:0')
ndcg_10 tensor(0.6191, device='cuda:0')
Save model, valid ndcg_10:0.619
current_step:  50
Reading data in ./example/toy/data/
Read data from ./example/toy/data//test in libsvm format.
Reading finish: 0 lines
Remove 3 invalid queries.
Data reading finish!
Finished reading 6 queries with lists.
Build DLA

Traceback (most recent call last):
  File "main.py", line 291, in <module>
    main(argv)
  File "main.py", line 284, in main
    test(exp_settings)
  File "main.py", line 236, in test
    model = create_model(exp_settings, test_set)
  File "main.py", line 70, in create_model
    model = ultra.utils.find_class(exp_settings['learning_algorithm'])(data_set, exp_settings)
  File "/home/me/workspace/github/ULTRA_pytorch/ultra/learning_algorithm/dla.py", line 97, in __init__
    self.rank_list_size = exp_settings['selection_bias_cutoff']
KeyError: 'selection_bias_cutoff'

Hi,

Thanks for open source this amazing work.

I have a question for the current example in main.py. Specifcially, for the following code,

https://github.com/ULTR-Community/ULTRA_pytorch/blob/ec4fe329e4239b588a940cb4bcdd6a321aade679/main.py#L92

If I understand correctly, is this should be ultra.utils.find_class(exp_settings['valid_input_feed']).preprocess_data(valid_set, exp_settings['valid_input_hparams'], exp_settings)

since for the valid set, we would like to evaluate based on the human annotation for LTR, thus the key should be valid_input_feed instead of train_input_feed, and for the same reason, the key for the second parameter shall be altered as well.

In addition, should be the test_set need to altered correspondingly.

Thanks for your time and consideration.

i'm trying to reproduce the algorithm on my local machine, w/ different IDE, i has these settings:

exp_settings = {
    "train_input_feed":"ultra.input_layer.ClickSimulationFeed",
    "train_input_hparams":"",
    "valid_input_feed":"ultra.input_layer.DirectLabelFeed",
    "valid_input_hparams":"",
    "test_input_feed":"ultra.input_layer.DirectLabelFeed",
    "test_input_hparams":"",
    "ranking_model":"ultra.ranking_model.SetRank",
    "ranking_model_hparams":" ",
    "learning_algorithm": "ultra.learning_algorithm.dla.DLA",
    "learning_algorithm_hparams":"",
    "metrics": [
        "mrr", "ndcg"
    ],
    "metrics_topn" : [3,5,10],
    "objective_metric": "ndcg_10",
    "max_candidate_num": 10 #added after error in DLA default setting setrank*
}

the error i got, when i execute model = create_model(exp_settings, train_set)

Reading finish: 0 lines
Remove 7 invalid queries.
Data reading finish!
Finished reading 20 queries with lists.
Read data from /ULTRA/example/toy/data//valid in libsvm format.
Reading finish: 0 lines
Remove 3 invalid queries.
Data reading finish!
Finished reading 6 queries with lists.
Train Rank list size 9
Valid Rank list size 9
Users can only see the top 9 documents for each query in training.
Creating model...
Build DLA
Traceback (most recent call last):

  File "/tmp/ipykernel_1350348/2504700013.py", line 151, in <module>
    model = create_model_main(exp_settings, train_set)

  File "/tmp/ipykernel_1350348/2504700013.py", line 99, in create_model_main
    model = ultra.utils.find_class(exp_settings['learning_algorithm'])(data_set, exp_settings)

  File "/ULTRA/ultra/learning_algorithm/dla.py", line 102, in __init__
    self.rank_model = self.create_model(self.feature_size)

  File "/ULTRA/ultra/learning_algorithm/base_algorithm.py", line 164, in create_model
    model = utils.find_class(

TypeError: 'module' object is not callable

There something wrong w/ function or nesting parameters here:

def create_model(self, feature_size):
      if not hasattr(self, "model"):
              model = utils.find_class(
                  self.exp_settings['ranking_model'])(
                  self.exp_settings['ranking_model_hparams'], feature_size)
      return model

debugging lead to setrank() model inside of DLA learning algorithm which is stuck w/ this line of code:

self.exp_settings['ranking_model_hparams'], feature_size)"

https://github.com/ULTR-Community/ULTRA_pytorch/blob/ec4fe329e4239b588a940cb4bcdd6a321aade679/ultra/learning_algorithm/dla.py#L266

Refer to https://discuss.pytorch.org/t/cpu-ram-usage-increasing-for-every-epoch/24475/9

When I run DLA for 10k steps, the CPU memory will be used up. A simple fixation is to modify it as self.loss.item().

A similar issue also occurs in regression_EM, pdgd, nsgd, etc.

Hi, Thanks for sharing this work.

I re-run the DLA on Yahoo! data, and the data preprocessing follows the example/Yahoo/offline_exp_pipeline.sh. However, I find the following metrics triggers errors:

1. average_relevance_position has an issue that tensors on different devices as the following tensor is on CPU instead of GPU. https://github.com/ULTR-Community/ULTRA_pytorch/blob/ec4fe329e4239b588a940cb4bcdd6a321aade679/ultra/utils/metrics.py#L363

2. precision is not callable. Since it is computed without repeat, it will trigger an error as TypeError: zip argument #2 must support iteration

Thanks for this amazing work.

I am trying to follow the experiments on Yahoo! Learning-to-Rank dataset, and find the variable $Data_path/tmp_data/ is not specified previously.

I assume it should be folder containing normalized train/test/valid, so is this should be Data_path/normalized/ instead of Data_path/tmp_data/?

https://github.com/ULTR-Community/ULTRA_pytorch/blob/d2d98b859ff49ad20335d0287d4de8906f9708b9/example/Yahoo/offline_exp_pipeline.sh#L82

And then the model will load train/valid/test, respectively for unbiased learning-to-rank.

oops, sorry that I miss that is the output from ./libsvm_tools/prepare_exp_data_with_svmrank.py

Issue closed.

I tried to run the code with DLA algorithm on Yahoo dataset. Following is the output attached. I am not sure of the following observation where I am getting almost constant training loss of about 4 (with each rank loss and exam loss as about 2), and high validation and testing metric values of more than 0.9. I did try to observe the parameter values of 2 models, which are actually updating. Also the loss is just fluctuating in range of 3.9 to 4.5 always. Is there something I should do with hyperparameters, have kept the default learning rate of 0.05 and selection_bias_cutoff = 10. This is with respect to the pytorch implementation of the code

I'm a little rusty at this stuff but I believe there might be a bug in the implementation of the position bias EM algorithm here for p_r1 = reshaped_train_labels + (1 - reshaped_train_labels) * p_e0_r1_c0. In particular, I think p_r1 for unclicked documents, (the 1-reshaped_train_labels case), is incorrect.

Using notation from the paper, page 614, we want to compute $P(R=1|C=0, q, k, d)$ by marginalizing $P(R=1, E|C=0, q, k, d)$ over $E$:

$$ \begin{aligned} P(R=1|C=0, q, k, d) =& \underbrace{P(R=1, E=1|C=0, q, k, d)}_{=0} P(E=1|C=0, q, k, d) \ & + P(R=1, E=0|C=0, q, k, d) P(E=0|C=0, q, k, d) \ =& P(R=1, E=0|C=0, q, k, d) (1- P(E=1|C=0, q, k, d)) \end{aligned} $$

The implementation just has $P(R=1|C=0, q, k, d) = P(R=1, E=0|C=0, q, k, d)$. That is, it's missing the $(1- P(E=1|C=0, q, k, d))$ term.

Am I missing something or is this a bug? Unfortunately, the paper just says "From this, we can compute the marginals P(E = 1|c, q, d, k) and P(R = 1|c, q, d, k)" without additional details and I can't find any other implementations of this paper to see how they're working. Like I said, I'm rusty at this so I may be missing something obvious.

Note, I think the same issue comes up here with $P(E=1|C=0, q,k, d)$, which is not explicitly named in the code like p_r1, but is computed as reshaped_train_labels + (1 - reshaped_train_labels) * p_e1_r0_c0 if I'm reading things correctly.

I think it should instead be

$$ P(E=1|C=0, q, k, d) = P(R=0, E=1|C=0, q, k, d) (1- P(R=1|C=0, q, k, d)) $$

Of course, these depend on each other, but with these two equations we can solve for $P(E=1|C=0, q, k, d)$ and $P(R=1|C=0, q, k, d)$ in terms of $P(R, E|C=0, q, k, d)$ which we can compute from gamma and self.propensity.

Hi,

Thanks for releasing this interesting platform.

I'm interested in how ULTRA could be used from another Python library - e.g. say I wanted to use an ULTRA model to re-rank results in PyTerrier.

With sklearn, xgboost, fastrank etc, I can give it an array of feature values for a given document, and it will return a score - see https://pyterrier.readthedocs.io/en/latest/ltr.html#learning for our integration.

Does ULTRA have a similar API?