Hi,

Thanks for making the code public which helps a lot. I'm wondering if you were using the fall11 version of ImageNet-21K for the open splits. I'm now using the winter21 version and some of the classes in the 'Hard' split do not exist anymore (e.g., 'n10506915').

If this is true, then a problem is that the ImageNet website doesn't seem to host fall11 version anymore (correct me if I'm wrong). If this is the case, then would it be possible for you to update the 'Hard' split based on the available winter21 version?

Thanks

Thanks for your wonderful work

I want to know why the checkpoint is not saved when I only use the CEloss, I read the code but did not find where should be modify.

Hi again,

I read from the paper that "[...] we propose the use of the maximum logit rather than softmax probability for the open-set scoring rule. Logits are the raw outputs of the final linear layer in a deep classifier, while the softmax operation involves a normalization such that the outputs can be interpreted as a probability vector summing to one. As the softmax operation normalizes out much of the feature magnitude information present in the logits, we find logits lead to better open-set detection results" . Then you have figure 6c that shows AUROC on the test set(s) and how it evolves as training goes on, using both max-logits and max-softmax for scoring, showing how it might be better to use max-of-logits.

However, the ARPL code for the Softmax loss (found here), which you are inheriting and using for testing, is a bit weird: it calls logits to the post-softmax activation, see here.

Since you are taking the (false) logits from calling the criterion (here) during testing, and then you have a few lines below the option of (re-)applying softmax to them if we are running with 'use_softmax_in_eval, I am wondering if what you are calling in your experiments from the paper "logits" are actually softmax(logits), and what you call softmax activations are indeed softmax(softmax(logits))?

Thanks!

Adrian

Hello again! I'm working on a task with your code, i have seen that all methods are evaluate by AUROC, a threshold independent metric, but now i need to predict the label by threshold, is there any func in code to do that?

Thanks for your work

When i train the model with parameters "--model=timm_resnet50_pretrained --resnet50_pretrain=places", and i download pretrained model from https://github.com\nanxuanzhao\Good_transfer, i get the error :"Missing key(s) in state_dict: "conv1.weight", "bn1.weight", "bn1.bias", "bn1.running_mean",.....................".

i think this error causes by mismatched weight and model, could you please tell me the right method to train model with pretrained weight, or right link to download weight?

Hello, thank you for maintaining this repo!

I have one question regarding a low accuracy using cub dataset.

Here is the bash file that I am using:

LOSS='ARPLoss'          # For TinyImageNet, ARPLoss and Softmax loss have the same
                        # RandAug and Label Smoothing hyper-parameters, but different learning rates

# Fixed hyper params for both ARPLoss and Softmax
DATASET='cub'
AUG_M=30
AUG_N=2
LABEL_SMOOTHING=0.1

# LR different for ARPLoss and Softmax
if [ $LOSS = "Softmax" ]; then
   LR=0.01
elif [ $LOSS = "ARPLoss" ]; then
   LR=0.001
fi

# GPU0-0 MIG-GPU-7391bfa5-fd39-632b-ac8a-cbd1359e940b/5/0


# tinyimagenet
for SPLIT_IDX in 0 1 2 3 4; do

  EXP_NUM=$(ls ${SAVE_DIR} | wc -l)
  EXP_NUM=$((${EXP_NUM}+1))
  echo $EXP_NUM

  ${PYTHON} -m methods.ARPL.osr --lr=0.001 --seed=0 \
                             --transform='rand-augment' \
                            --rand_aug_m=${AUG_M} --rand_aug_n=${AUG_N} --loss=${LOSS} --label_smoothing=${LABEL_SMOOTHING} \
                            --dataset=${DATASET} --image_size=448 --cs --num_restarts=2 --gpus 0 --split_idx=${SPLIT_IDX} \
                            --scheduler='cosine_warm_restarts_warmup' --split_train_val='True' --batch_size=32 --num_workers=16 --max-epoch=600 \
> ${SAVE_DIR}logfile_${DATASET}_cs_${LOSS}_${EXP_NUM}.out
done

Batch 150/150    Net 4.238 (4.178) G 19.857 (19.787) D 0.000 (0.000)

I believe this is because the loss of D(discriminator) becomes 0.

However, I was not able to get the accuracy more than 20%. Could you point me out what I am doing wrong?

Thank you!

Hello again,

I have been able to train models for all benchmarks, old and newly proposed, so thanks for that!

I am just missing the Imagenet experiment, which appears in Table 3 of the paper (with an easy and a hard split). The ImageNet splits seem to be present in data/open_set_split/imagenet_osr_splits.pkl, but there is not pytorch dataset that implements them, nor do ImageNet hyperparameters appear in utils/paper_hyperparameters.csv. I was wondering if you have any plans to release the ImageNet experimental setup anytime soon? Thanks!!

Also, may I email you with a separate question about your work that does not fit in a github issue? Thank you very much!!

Adrian

https://github.com/sgvaze/osr_closed_set_all_you_need/blob/c3fee78818f83052cd8fed7826d5511a5a9d7165/data/open_set_splits/osr_splits.py#L19

Hi, first of all, thank you for sharing the code.

For the OSR-CIFAR10 experiment, the first and second splits are exactly the same. I am wondering if this is actually correct.

Hi! First, thank you very much for this work, it is very refreshing to see recent OSR methods put to test and finding out that mostly they are over-hyped over-complex approaches, and cross-entropy alone is so competitive if you give some care to training baselines properly, congratulations :)

I am trying to reproduce your results, but I am struggling to understand how to do it. I am starting from Tiny Imagenet, which I have been able to re-train successfuly, after:

1. running the create_val_img_folder function on the dataset folder, and
2. correcting lines 18 of methods/ARPL/core/train.py, as well as lines 25 and 42 of methods/ARPL/core/test.py, because options['use_gpu'] does not exist; those lines should probably be replaced by if not options['use_cpu'], which works ok.

Now, after properly manipulating config.py and bash_scripts/osr_train_tinyimagenet.sh, I carry out the entire training and I end up with a directory called, in this case, in methods/ARPL/log/(03.01.2022_|_32.677). Within this directory, one can find some tensorboard-related stuff, and two directories, namely checkpoints/ and arpl_models/tinyimagenet/checkpoints/. The former is empty and I guess it is created by mistake, whereas the latter contains a bunch of checkpoints, as it seems that you guys are storing a model checkpoint (and a "criterion checkpoint", which btw,I don't know what it is) each twenty epochs.

My question is, how exactly do I evaluate the final performance of this experiment? I.e.:

• How do I know which is the checkpoint with the highest closed-set performance, that I should then be using to compute Accuracy on the closed set, AUC on the open classes, plus the OSCR score, like in Table 5 or Table 3?
• Which piece of code should I use to evaluate the checkpoint, and how do I go about it?

I'm suspecting it might have something to do with methods/tests/openset_test.py, but I am not sure since there seem to be some hard-coded experiment names in there,and it seems to be only useful for evaluating the performance of an ensemble of five models. Could you please provide some instructions on how to assess final performance of a trained model?

Thanks!!

Adrian

P.S.: In the next days or weeks I might be asking some more questions about your work, thanks for the patience!

Hello! I am glad to see this work when I encounter OSR problems in practice. I'm wonder that how to give the unknown label to a sample with the classifier trained on the closed set.

Hello When I tried to reproduce the AUC results of the Easy and Medium datasets for the CUB and Aircraft datasets, I couldn't reproduce the results of the paper, for CUB, Easy and Medium were only 87.5 and 81.8; for Aircraft is 89.0 and 85.4. I trained according to the bash script/osr_finegrained_train.sh and fix corresponding Optimal Hyper-parameters, and used the places_moco pre-trainmodel, but the results in the paper could not be reproduced. What is the reason?

Hi,

I found that the evaluation metrics are unstable among different epochs when I conducted experiments on the custom dataset. I would like to ask whether you report the last or the best epoch results in your training exps.

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

Hi,

I am trying to use the RandAug instead of RandCrop and cosine learning rate schedule to train the whole Cifar100 dataset. But they do not work. The baseline is RandCrop with step learning rate schedule (initial learning rate is 0.1 and divided by 2 at [60, 120, 160] epochs with 300 epochs in total). The closed-set accuracy of baseline is 0.75, and RandAug and cosine learning rate schedule (restart 0 or 2 times for 600 epochs) does not work, can you give me some advise?

Best regards.