Dear Authors,

I have trained the model based on BDD. And I have evaluated the model on the COCO successfully. Now I use the same model to evaluate openimages. There is an error.

I want to use this method in yolo, but during the implementation process, I encountered some problems. Has anyone used this method in yolo? Or have any suggestions?

Greetings, I am attempting to involve your proposed model in my undertaking. I have utilized the accompanying cfg to prepare my model.

from vos.detection.default_trainer_gmm import DefaultTrainer from detectron2.config import CfgNode, get_cfg from vos.detection.core.setup import add_probabilistic_config

cfg = get_cfg()#CfgNode() add_probabilistic_config(cfg)

cfg_file_name = "vos/detection/configs/VOC-Detection/faster-rcnn/vos.yaml" cfg.merge_from_file(cfg_file_name) cfg.SOLVER.BASE_LR = 0.0001 cfg.SOLVER.IMS_PER_BATCH=6 cfg.MODEL.ROI_HEADS.BATCH_SIZE_PER_IMAGE = 64 cfg.MODEL.ROI_HEADS.POSITIVE_FRACTION = 0.25 cfg.MODEL.RPN.BATCH_SIZE_PER_IMAGE = 128 cfg.SOLVER.MAX_ITER = 20000 cfg.SOLVER.STEPS = (8000,12000) cfg.VOS.STARTING_ITER = 10000 cfg.SOLVER.CHECKPOINT_PERIOD = 8000 cfg.MODEL.ROI_HEADS.NUM_CLASSES = len(MetadataCatalog.get('my_data').thing_classes)

Some way or another, I am not obtain great outcomes. Would you be able to diagram why it isn't functioning admirably

Dataset info

Training Graphs:

Please let me know if you need more info, to help me.

Hi,

I tried applying the classification example with a different network. To be specific, I tried the Xception network which has a feature size of 2048 (the WideResNet model from your code has a feature size of 128). This introduces an error, raised by PyTorch:

ValueError: Expected parameter covariance_matrix (Tensor of shape (2048, 2048)) of distribution MultivariateNormal(loc: torch.Size([2048]), covariance_matrix: torch.Size([2048, 2048])) to satisfy the constraint PositiveDefinite(), but found invalid values:... Interesting enough, this does not happen with the efficientnet-b5 model (which also has a feature size of 2048). Do you maybe know what is going on?

Before evaluation FPR95 and AUROC, the predictions are filtered by "a threshold", why after this filter, the left predictions can be seen as "in-distribution object?", and what's meaning about this optimal_score_threshold. thank you~

    # Compute optimal micro F1 score threshold. We compute the f1 score for
    # every class and score threshold. We then compute the score threshold that
    # maximizes the F-1 score of every class. The final score threshold is the average
    # over all classes.
    precisions = results_api.eval['precision'].mean(0)[:, :, 0, 2]
    recalls = np.expand_dims(results_api.params.recThrs, 1)
    f1_scores = 2 * (precisions * recalls) / (precisions + recalls)
    optimal_f1_score = f1_scores.argmax(0)
    scores = results_api.eval['scores'].mean(0)[:, :, 0, 2]
    optimal_score_threshold = [scores[optimal_f1_score_i, i]
                               for i, optimal_f1_score_i in enumerate(optimal_f1_score)]
    optimal_score_threshold = np.array(optimal_score_threshold)
    optimal_score_threshold = optimal_score_threshold[optimal_score_threshold != 0]
    optimal_score_threshold = optimal_score_threshold.mean()

I was wondering if there is a missing python script for testing VOS trained CIFAR models on the OOD datasets to recreate Table 4 from the paper? In classification/CIFAR/test.py I am not seeing any of the VOS code from train_virtual.py referenced for any of the evaluation options.

Hopefully I'm not overlooking the code and missing it. Thanks.

Dear Author,

I'm trying to run your code and I have a trouble with preparing classification datasets. (places365, iSUN, LSUN-C, LSUN-Resize) I downloaded these datasets but there are only images in the folder. There are no class labels. Would you tell me where I can download classification datasets with class labels? Thank you.

Using main-MLP branch, and training command：

python train_net_gmm.py --dataset-dir /data/haitaozhang/dataset/object_detection/VOC --num-gpus 4 --config-file VOC-Detection/faster-rcnn/vos.yaml --random-seed 0 --resume

it's stucking in 12000 iter, and gpus usage do not work.

how can I solve this problem, maybe need use the main branch?

Hi,

Thanks for your public code. The OOD detection benchmarks in your paper are great. I want to know how to get results of the Mahalanobis method based on your public code?

Thanks.

Dear Authors,

this is the evaluation results based on VOC dataset. The results are not consistent with the given results in the paper.

I first run python apply_net.py --dataset-dir /data1/user/vos/VOC_0712_converted --test-dataset voc_custom_val --config-file VOC-Detection/faster-rcnn/vos.yaml --inference-config Inference/standard_nms.yaml --random-seed 0 --image-corruption-level 0 --visualize 0.

Then I run python apply_net.py --dataset-dir /data1/user/COCO2017/datasets/coco --test-dataset coco_ood_val --config-file VOC-Detection/faster-rcnn/vos.yaml --inference-config Inference/standard_nms.yaml --random-seed 0 --image-corruption-level 0 --visualize 0.

Finally, I run CUDA_VISIBLE_DEVICES=4 python voc_coco_plot.py --name vos --thres xxx --energy 1 --seed 0.

Could you give me some advice? Thank you.

Hello,

How can I calculate the OOD score on the classification models? I see the "get_ood_scores" function in "vos-main/classification/CIFAR/test.py". Does this function calculate the OOD score for VOS? (or this function only relates to the baselines considered in the paper?).