Hi, I have trouble replicating the results that you reported in Table 3 in the paper for the ARPL method (so far).

I downloaded you git repository and run this command to train the ARPL model (I run it three times to account for random initialization using different out dirs): python ood.py --dataset cifar10 --out-dataset svhn --model arpl --loss ARPLoss --outf log_arpl Here is a training log from one of the model training script logs.txt.

To evaluate I run this command (for each trained model): python ood.py --dataset cifar10 --out-dataset svhn --model arpl --loss ARPLoss --outf log_arpl --eval

I get the following results:

Acc: 92.58000
       TNR    AUROC  DTACC  AUIN   AUOUT
Bas    25.496 82.803 78.414 65.403 90.405
Acc (%): 92.580  AUROC (%): 82.803       OSCR (%): 79.600

Acc: 92.68000
       TNR    AUROC  DTACC  AUIN   AUOUT
Bas    21.589 78.534 74.769 54.259 88.407
Acc (%): 92.680  AUROC (%): 78.534       OSCR (%): 75.498

Acc: 92.68000
       TNR    AUROC  DTACC  AUIN   AUOUT
Bas    37.930 82.561 77.525 78.463 82.497
Acc (%): 92.680  AUROC (%): 82.561       OSCR (%): 78.857

There were no errors or warnings during the running of the scripts. All metrics are significantly below the reported numbers. Do you have any idea what may be the issue? Thank you.

I noticed that in the experimental code, you use known class as a positive sample to calculate related indicators. This does not seem to be consistent with the openset/ODD field. Most of the openset open source codes choose unknown class as a positive sample, such as the OSRCI method you compared.

Hi there! Thanks for your inspiring work and releasing the code. I have a small question regarding the baseline results. I did not modify the code and run it with command python osr.py --dataset cifar10 --loss Softmax. If I understand correctly, this would be the baseline method, and according to the Table 1 in your paper, the result AUROC should be 67.7 for the CIFAR10 dataset. However, the log I obtained is as follows:

,0,1,2,3,4
TNR,34.0,30.625000000000004,22.624999999999996,35.175,30.500000000000004
AUROC,86.9976125,85.59652499999999,84.34554583333333,86.77233749999999,86.72411666666667
DTACC,80.25416666666668,79.03333333333333,78.1625,79.575,79.97916666666667
AUIN,92.19165744328468,90.77076215295214,90.76942678194541,91.79811544137021,91.64555773527704
AUOUT,77.31935437681796,75.15246512319506,72.03227915144676,77.35292209454035,76.84887201588833
ACC,94.16666666666667,95.58333333333333,91.35,95.3,95.18333333333334
OSCR,84.46526250000002,84.14960000000002,80.82762291666654,84.88179583333343,84.94574374999975
unknown,"[0, 8, 3, 5]","[2, 3, 4, 5]","[0, 8, 2, 6]","[8, 2, 3, 5]","[8, 2, 3, 5]"
known,"[2, 4, 1, 7, 9, 6]","[8, 6, 1, 9, 0, 7]","[1, 5, 7, 3, 9, 4]","[7, 6, 4, 9, 0, 1]","[0, 6, 4, 9, 1, 7]"

And the average AUROC is about 86.09, which is significantly higher than the results reported. I'd like to know if there is anything that I haven't done properly. Thanks in advance!

I noticed that the classes of close set are assigned in split.py. Are the results only reimplemented via the specific classes ? Two phenomenons shown in experiments.

1. I tried to change the class number into random set. The performance decrease dramatically. The average AUROC in 5 times is only about 50.
2. I trained the close set with close set with cross entropy loss, and use the softmax probability as logits in evaluation. Then, I get similar perfromance (about 74~ in AUROC) in results. May I miss some something in your code ?

我跑你的代码的时候，我发现当我把compute_oscr函数的条件改成跟计算AUROC一样的时候，log输出的AUROC的值要大于OSCR的值。然后我阅读compute_oscr中计算不同阈值TP和FP的代码时候，发现TP是K+1但是FP没有K+1（我知道+1是为了取消最大的置信度当阈值时的计算） CC = s_k_target[k+1:].sum() FP = s_u_target[k:].sum() 我想问这里的FP的计算是不是索引也应该是k+1

Could you provide the code as well as pretrained models for the experiment on ImageNet1k? I'm trying to evaluate ARPL on various other OOD datasets. Thanks !