Hi, I'm new to out-of-distribution detection. After reading the paper and the code, I still cannot figure out how the out-of-distribution data is detected. I see there are two related lines of code as below.

out_score = get_ood_scores(ood_loader)
measures = get_measures(out_score, in_score)

It seems that the detection process is related to the in_score. But what about the prediction in a real application scenario? I'm confused. You could give me some references to read if that's easier to explain.

In the training script, the cross entropy between the softmax distribution and the uniform distribution is implemented with this line.

loss += 0.5 * -(x[len(in_set[0]):].mean(1) - torch.logsumexp(x[len(in_set[0]):], dim=1)).mean()

How does torch.logsumexp(x[len(in_set[0]):], dim=1) represent the uniform distribution?

Thanks.

I was going through the paper and in section 3 where the OE minimization objective is defined, there is no constraint on the type of loss that is being used. Hence I was curious if the methodology can be applied to tasks where we do pixel-level prediction (example image translation) and use L1 or L2 loss? Similar to classification loss, if we use uniform random ground truth images as the label for ood images, will it work or dealing with pixel-level predictions, require a completely different methodology?

Hello, I am a graduate student in school and I just read this paper of yours. I want to ask you some questions that I don’t understand. First of all, I don't understand the latter part of the formula in the paper.I don’t know what it means.Is f(x') the predicted output of the model on OOD data?Is the cross entropy calculated for the second half? Second, loss += 0.5 * -(x[len(in_set[0]):].mean(1) - torch.logsumexp(x[len(in_set[0]):], dim=1)).mean() I don't understand how this formula shows that the obtained is cross entropy?And where is softmax used in the first half（x[len(in_set[0]):].mean(1)） of the formula?How does the second half of the formula（torch.logsumexp(x[len(in_set[0]):], dim=1)） represent uniform distribution? Thansk,sir!

Hi,

In your paper, you mention that 'We remove all examples of 80 Million Tiny Images which appear in the CIFAR datasets'. I was trying to download 80 million Tinyimagenet but apparantly it is not available anymore. Instead, I am trying to use imagenet32X32. Is there an effective way to remove duplicates of cifar10 or cifar100 from imagenet32X32 ?

Hi, @hendrycks , thanks for your code!

I am new to Out of distribution detection, and I am not sure about the OOD score you used in the code.

_score.append(to_np((output.mean(1) - torch.logsumexp(output, dim=1))))

Don't we use the maximum probability predicted as the metric to discriminate in-distribution and out-distribution data? where does this score originate?

I appreciate it if you can refer me some papers that proposed this score

Thanks,

Hi,

I am not able to understand the loss function mentioned in the file oe_scratch.py in MNIST folder. It does not look like minimising the KL divergence between Uniform distribution and cross-entropy distribution.

The loss mentioned in the file is: loss += 0.5 * -(x[len(in_set[0]):].mean(1) - torch.logsumexp(x[len(in_set[0]):], dim=1)).mean()

Can you please help me in understanding what it is trying to do.

Hi,

Thank you very much for the response. After the modifications you suggested, I was able to reproduce the results of the paper for the SST dataset.

However, there is a problem for the 20 Newsgroups dataset. When I evaluate your pretrained model, the results are way different from the ones mentioned in the paper. Actually, when we run the evaluation script for your model, the results we get are the following:

OOD dataset mean FPR: 0.6828 OOD dataset mean AUROC: 0.7115 OOD dataset mean AUPR: 0.2773

So, what could be the issue? If the evaluation datasets are the same ones as the ones you used in the SST, I cannot find a reason for which your pretrained model does not give the results mentioned in the paper. Which version of the 20 Newsgroups did you use and how did you do the train/test split? we use the newsgroup available from sklearn library

Thank you,

Originally posted by @AristotelisPap in https://github.com/hendrycks/outlier-exposure/issues/4#issuecomment-517967141

Hello,

I am currently trying to replicate the results of the paper related to text classification. For example, let's say I want to replicate the results for sst dataset with OE on the wikitext2 dataset. If I load your oe_tuned model and I run the script eval_OOD_sst, then I get results similar to the ones mentioned in the paper.

However, whenever I run the baseline script in order to train the baseline model and then I run the oe script in order to fine-tune the model, then if I evaluate the fine-tuned model using the script eval_OOD_sst, the results are way different from the ones mentioned in the paper. I guess something is going on with the training process. But what could be the issue?

Thank you,

Aris

Hi, Is it possible to push the rest of the code including the language modeling task? I'm doing some related work and it would be of great help. Thanks!

Hi,

I was looking to understand the pre-processing involved prior to generating predictions from the neural network on OOD samples for computer vision(image classification). Specifically, how the OOD-test samples are standardised. Could you shed some light on it?

Regards.

Maybe I am not looking hard enough but I can't seem to find the code for density estimation using pixelcnn++. Can you direct me to the file for the same?

Hello! Thank you for your excellent work. I have some questions about the implementation. When I change the train_loader_ood with shuffle = True and the ID classification drops a lot after finetuning the pre-trained WideResNet with lr 0.001 and weight_decay 0.0005. The fine-tuned WideResNet in Cifar10 has 88.59% test classification accuracy compared with 94.39%. Instead, you use a random offset to induce the randomness. Could you please clarify the reason for this operation? Thanks again!