PyTorch implementation of paper “Unbiased Scene Graph Generation from Biased Training”

Kaihua Tang

Last update: Jan 3, 2023

Overview

Scene Graph Benchmark in Pytorch

Our paper Unbiased Scene Graph Generation from Biased Training has been accepted by CVPR 2020 (Oral).

Recent Updates

2020.06.23 Add no graph constraint mean Recall@K (ng-mR@K) and no graph constraint Zero-Shot Recall@K (ng-zR@K) [link]
2020.06.23 Allow scene graph detection (SGDet) on custom images [link]
2020.07.21 Change scene graph detection output on custom images to json files [link]
2020.07.21 Visualize detected scene graphs of custom images [link]
TODO: Using Background-Exempted Inference to improve the quality of TDE Scene Graph

Overview
Install the Requirements
Prepare the Dataset
Metrics and Results for our Toolkit
Faster R-CNN Pre-training
Scene Graph Generation as RoI_Head
Training on Scene Graph Generation
Evaluation on Scene Graph Generation
Detect Scene Graphs on Your Custom Images 🌟
Visualize Detected Scene Graphs of Custom Images 🌟
Other Options that May Improve the SGG
Tips and Tricks for TDE on any Unbiased Task
Frequently Asked Questions
Citations

Overview

This project aims to build a new CODEBASE of Scene Graph Generation (SGG), and it is also a Pytorch implementation of the paper Unbiased Scene Graph Generation from Biased Training. The previous widely adopted SGG codebase neural-motifs is detached from the recent development of Faster/Mask R-CNN. Therefore, I decided to build a scene graph benchmark on top of the well-known maskrcnn-benchmark project and define relationship prediction as an additional roi_head. By the way, thanks to their elegant framework, this codebase is much more novice-friendly and easier to read/modify for your own projects than previous neural-motifs framework(at least I hope so). It is a pity that when I was working on this project, the detectron2 had not been released, but I think we can consider maskrcnn-benchmark as a more stable version with less bugs, hahahaha. I also introduce all the old and new metrics used in SGG, and clarify two common misunderstandings in SGG metrics in METRICS.md, which cause abnormal results in some papers.

Benefit from the up-to-date Faster R-CNN in maskrcnn-benchmark, this codebase achieves new state-of-the-art Recall@k on SGCls & SGGen (by 2020.2.16) through the reimplemented VCTree using two 1080ti GPUs and batch size 8:

Models	SGGen R@20	SGGen R@50	SGGen R@100	SGCls R@20	SGCls R@50	SGCls R@100	PredCls R@20	PredCls R@50	PredCls R@100
VCTree	24.53	31.93	36.21	42.77	46.67	47.64	59.02	65.42	67.18

Note that all results of VCTree should be better than what we reported in Unbiased Scene Graph Generation from Biased Training, because we optimized the tree construction network after the publication.

The illustration of the Unbiased SGG from 'Unbiased Scene Graph Generation from Biased Training'

Installation

Check INSTALL.md for installation instructions.

Dataset

Check DATASET.md for instructions of dataset preprocessing.

Metrics and Results (IMPORTANT)

Explanation of metrics in our toolkit and reported results are given in METRICS.md

Pretrained Models

Since we tested many SGG models in our paper Unbiased Scene Graph Generation from Biased Training, I won't upload all the pretrained SGG models here. However, you can download the pretrained Faster R-CNN we used in the paper, which is the most time consuming step in the whole training process (it took 4 2080ti GPUs). As to the SGG model, you can follow the rest instructions to train your own, which only takes 2 GPUs to train each SGG model. The results should be very close to the reported results given in METRICS.md

After you download the Faster R-CNN model, please extract all the files to the directory /home/username/checkpoints/pretrained_faster_rcnn. To train your own Faster R-CNN model, please follow the next section.

The above pretrained Faster R-CNN model achives 38.52/26.35/28.14 mAp on VG train/val/test set respectively.

Faster R-CNN pre-training

The following command can be used to train your own Faster R-CNN model:

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10001 --nproc_per_node=4 tools/detector_pretrain_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 8 TEST.IMS_PER_BATCH 4 DTYPE "float16" SOLVER.MAX_ITER 50000 SOLVER.STEPS "(30000, 45000)" SOLVER.VAL_PERIOD 2000 SOLVER.CHECKPOINT_PERIOD 2000 MODEL.RELATION_ON False OUTPUT_DIR /home/kaihua/checkpoints/pretrained_faster_rcnn SOLVER.PRE_VAL False

where CUDA_VISIBLE_DEVICES and --nproc_per_node represent the id of GPUs and number of GPUs you use, --config-file means the config we use, where you can change other parameters. SOLVER.IMS_PER_BATCH and TEST.IMS_PER_BATCH are the training and testing batch size respectively, DTYPE "float16" enables Automatic Mixed Precision supported by APEX, SOLVER.MAX_ITER is the maximum iteration, SOLVER.STEPS is the steps where we decay the learning rate, SOLVER.VAL_PERIOD and SOLVER.CHECKPOINT_PERIOD are the periods of conducting val and saving checkpoint, MODEL.RELATION_ON means turning on the relationship head or not (since this is the pretraining phase for Faster R-CNN only, we turn off the relationship head), OUTPUT_DIR is the output directory to save checkpoints and log (considering /home/username/checkpoints/pretrained_faster_rcnn), SOLVER.PRE_VAL means whether we conduct validation before training or not.

Scene Graph Generation as RoI_Head

To standardize the SGG, I define scene graph generation as an RoI_Head. Referring to the design of other roi_heads like box_head, I put most of the SGG codes under maskrcnn_benchmark/modeling/roi_heads/relation_head and their calling sequence is as follows:

Perform training on Scene Graph Generation

There are three standard protocols: (1) Predicate Classification (PredCls): taking ground truth bounding boxes and labels as inputs, (2) Scene Graph Classification (SGCls) : using ground truth bounding boxes without labels, (3) Scene Graph Detection (SGDet): detecting SGs from scratch. We use two switches MODEL.ROI_RELATION_HEAD.USE_GT_BOX and MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL to select the protocols.

For Predicate Classification (PredCls), we need to set:

MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL True

For Scene Graph Classification (SGCls):

MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False

For Scene Graph Detection (SGDet):

MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False

Predefined Models

We abstract various SGG models to be different relation-head predictors in the file roi_heads/relation_head/roi_relation_predictors.py, which are independent of the Faster R-CNN backbone and relation-head feature extractor. To select our predefined models, you can use MODEL.ROI_RELATION_HEAD.PREDICTOR.

For Neural-MOTIFS Model:

MODEL.ROI_RELATION_HEAD.PREDICTOR MotifPredictor

For Iterative-Message-Passing(IMP) Model (Note that SOLVER.BASE_LR should be changed to 0.001 in SGCls, or the model won't converge):

MODEL.ROI_RELATION_HEAD.PREDICTOR IMPPredictor

For VCTree Model:

MODEL.ROI_RELATION_HEAD.PREDICTOR VCTreePredictor

For our predefined Transformer Model (Note that Transformer Model needs to change SOLVER.BASE_LR to 0.001, SOLVER.SCHEDULE.TYPE to WarmupMultiStepLR, SOLVER.MAX_ITER to 16000, SOLVER.IMS_PER_BATCH to 16, SOLVER.STEPS to (10000, 16000).), which is provided by Jiaxin Shi:

MODEL.ROI_RELATION_HEAD.PREDICTOR TransformerPredictor

For Unbiased-Causal-TDE Model:

MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor

The default settings are under configs/e2e_relation_X_101_32_8_FPN_1x.yaml and maskrcnn_benchmark/config/defaults.py. The priority is command > yaml > defaults.py

Customize Your Own Model

If you want to customize your own model, you can refer maskrcnn-benchmark/modeling/roi_heads/relation_head/model_XXXXX.py and maskrcnn-benchmark/modeling/roi_heads/relation_head/utils_XXXXX.py. You also need to add corresponding nn.Module in maskrcnn-benchmark/modeling/roi_heads/relation_head/roi_relation_predictors.py. Sometimes you may also need to change the inputs & outputs of the module through maskrcnn-benchmark/modeling/roi_heads/relation_head/relation_head.py.

The proposed Causal TDE on Unbiased Scene Graph Generation from Biased Training

As to the Unbiased-Causal-TDE, there are some additional parameters you need to know. MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE is used to select the causal effect analysis type during inference(test), where "none" is original likelihood, "TDE" is total direct effect, "NIE" is natural indirect effect, "TE" is total effect. MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE has two choice "sum" or "gate". Since Unbiased Causal TDE Analysis is model-agnostic, we support Neural-MOTIFS, VCTree and VTransE. MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER is used to select these models for Unbiased Causal Analysis, which has three choices: motifs, vctree, vtranse.

Note that during training, we always set MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE to be 'none', because causal effect analysis is only applicable to the inference/test phase.

Examples of the Training Command

Training Example 1 : (PreCls, Motif Model)

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --master_port 10025 --nproc_per_node=2 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL True MODEL.ROI_RELATION_HEAD.PREDICTOR MotifPredictor SOLVER.IMS_PER_BATCH 12 TEST.IMS_PER_BATCH 2 DTYPE "float16" SOLVER.MAX_ITER 50000 SOLVER.VAL_PERIOD 2000 SOLVER.CHECKPOINT_PERIOD 2000 GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/pretrained_faster_rcnn/model_final.pth OUTPUT_DIR /home/kaihua/checkpoints/motif-precls-exmp

where GLOVE_DIR is the directory used to save glove initializations, MODEL.PRETRAINED_DETECTOR_CKPT is the pretrained Faster R-CNN model you want to load, OUTPUT_DIR is the output directory used to save checkpoints and the log. Since we use the WarmupReduceLROnPlateau as the learning scheduler for SGG, SOLVER.STEPS is not required anymore.

Training Example 2 : (SGCls, Causal, TDE, SUM Fusion, MOTIFS Model)

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --master_port 10026 --nproc_per_node=2 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE none MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs  SOLVER.IMS_PER_BATCH 12 TEST.IMS_PER_BATCH 2 DTYPE "float16" SOLVER.MAX_ITER 50000 SOLVER.VAL_PERIOD 2000 SOLVER.CHECKPOINT_PERIOD 2000 GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/pretrained_faster_rcnn/model_final.pth OUTPUT_DIR /home/kaihua/checkpoints/causal-motifs-sgcls-exmp

Evaluation

Examples of the Test Command

Test Example 1 : (PreCls, Motif Model)

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL True MODEL.ROI_RELATION_HEAD.PREDICTOR MotifPredictor TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/motif-precls-exmp OUTPUT_DIR /home/kaihua/checkpoints/motif-precls-exmp

Test Example 2 : (SGCls, Causal, TDE, SUM Fusion, MOTIFS Model)

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10028 --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs  TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/causal-motifs-sgcls-exmp OUTPUT_DIR /home/kaihua/checkpoints/causal-motifs-sgcls-exmp

Examples of Pretrained Causal MOTIFS-SUM models

Examples of Pretrained Causal MOTIFS-SUM models on SGDet/SGCls/PredCls (batch size 12): (SGDet Download), (SGCls Download), (PredCls Download)

Corresponding Results (The original models used in the paper are lost. These are the fresh ones, so there are some fluctuations on the results. More results can be found in Reported Results):

Models	R@20	R@50	R@100	mR@20	mR@50	mR@100	zR@20	zR@50	zR@100
MOTIFS-SGDet-none	25.42	32.45	37.26	4.36	5.83	7.08	0.02	0.08	0.24
MOTIFS-SGDet-TDE	11.92	16.56	20.15	6.58	8.94	10.99	1.54	2.33	3.03
MOTIFS-SGCls-none	36.02	39.25	40.07	6.50	8.02	8.51	1.06	2.18	3.07
MOTIFS-SGCls-TDE	20.47	26.31	28.79	9.80	13.21	15.06	1.91	2.95	4.10
MOTIFS-PredCls-none	59.64	66.11	67.96	11.46	14.60	15.84	5.79	11.02	14.74
MOTIFS-PredCls-TDE	33.38	45.88	51.25	17.85	24.75	28.70	8.28	14.31	18.04

SGDet on Custom Images

Note that evaluation on custum images is only applicable for SGDet model, because PredCls and SGCls model requires additional ground-truth bounding boxes information. To detect scene graphs into a json file on your own images, you need to turn on the switch TEST.CUSTUM_EVAL and give a folder path that contains the custom images to TEST.CUSTUM_PATH. Only JPG files are allowed. The output will be saved as custom_prediction.json in the given DETECTED_SGG_DIR.

Test Example 1 : (SGDet, Causal TDE, MOTIFS Model, SUM Fusion) (checkpoint)

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE TDE MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/causal-motifs-sgdet OUTPUT_DIR /home/kaihua/checkpoints/causal-motifs-sgdet TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /home/kaihua/checkpoints/custom_images DETECTED_SGG_DIR /home/kaihua/checkpoints/your_output_path

Test Example 2 : (SGDet, Original, MOTIFS Model, SUM Fusion) (same checkpoint)

CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/relation_test_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.ROI_RELATION_HEAD.PREDICTOR CausalAnalysisPredictor MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE none MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE sum MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER motifs TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/kaihua/glove MODEL.PRETRAINED_DETECTOR_CKPT /home/kaihua/checkpoints/causal-motifs-sgdet OUTPUT_DIR /home/kaihua/checkpoints/causal-motifs-sgdet TEST.CUSTUM_EVAL True TEST.CUSTUM_PATH /home/kaihua/checkpoints/custom_images DETECTED_SGG_DIR /home/kaihua/checkpoints/your_output_path

The output is a json file. For each image, the scene graph information is saved as a dictionary containing bbox(sorted), bbox_labels(sorted), bbox_scores(sorted), rel_pairs(sorted), rel_labels(sorted), rel_scores(sorted), rel_all_scores(sorted), where the last rel_all_scores give all 51 predicates probability for each pair of objects. The dataset information is saved as custom_data_info.json in the same DETECTED_SGG_DIR.

Visualize Detected SGs of Custom Images

To visualize the detected scene graphs of custom images, you can follow the jupyter note: visualization/3.visualize_custom_SGDet.jpynb. The inputs of our visualization code are custom_prediction.json and custom_data_info.json in DETECTED_SGG_DIR. They will be automatically generated if you run the above custom SGDet instruction successfully. Note that there may be too much trivial bounding boxes and relationships, so you can select top-k bbox and predicates for better scene graphs by change parameters box_topk and rel_topk.

Other Options that May Improve the SGG

For some models (not all), turning on or turning off MODEL.ROI_RELATION_HEAD.POOLING_ALL_LEVELS will affect the performance of predicate prediction, e.g., turning it off will improve VCTree PredCls but not the corresponding SGCls and SGGen. For the reported results of VCTree, we simply turn it on for all three protocols like other models.
For some models (not all), a crazy fusion proposed by Learning to Count Object will significantly improves the results, which looks like f(x1, x2) = ReLU(x1 + x2) - (x1 - x2)**2. It can be used to combine the subject and object features in roi_heads/relation_head/roi_relation_predictors.py. For now, most of our model just concatenate them as torch.cat((head_rep, tail_rep), dim=-1).
Not to mention the hidden dimensions in the models, e.g., MODEL.ROI_RELATION_HEAD.CONTEXT_HIDDEN_DIM. Due to the limited time, we didn't fully explore all the settings in this project, I won't be surprised if you improve our results by simply changing one of our hyper-parameters

Tips and Tricks for any Unbiased TaskX from Biased Training

The counterfactual inference is not only applicable to SGG. Actually, my collegue Yulei found that counterfactual causal inference also has significant potential in unbiased VQA. We believe such an counterfactual inference can also be applied to lots of reasoning tasks with significant bias. It basically just runs the model two times (one for original output, another for the intervened output), and the later one gets the biased prior that should be subtracted from the final prediction. But there are three tips you need to bear in mind:

The most important things is always the causal graph. You need to find the correct causal graph with an identifiable branch that causes the biased predictions. If the causal graph is incorrect, the rest would be meaningless. Note that causal graph is not the summarization of the existing network (but the guidance to build networks), you should modify your network based on causal graph, but not vise versa.
For those nodes having multiple input branches in the causal graph, it's crucial to choose the right fusion function. We tested lots of fusion funtions and only found the SUM fusion and GATE fusion consistently working well. The fusion function like element-wise production won't work for TDE analysis in most of the cases, because the causal influence from multiple branches can not be linearly separated anymore, which means, it's no longer an identifiable 'influence'.
For those final predictions having multiple input branches in the causal graph, it may also need to add auxiliary losses for each branch to stablize the causal influence of each independent branch. Because when these branches have different convergent speeds, those hard branches would easily be learned as unimportant tiny floatings that depend on the fastest/stablest converged branch. Auxiliary losses allow different branches to have independent and equal influences.

Frequently Asked Questions:

Q: Fail to load the given checkpoints. A: The model to be loaded is based on the last_checkpoint file in the OUTPUT_DIR path. If you fail to load the given pretained checkpoints, it probably because the last_checkpoint file still provides the path in my workstation rather than your own path.
Q: AssertionError on "assert len(fns) == 108073" A: If you are working on VG dataset, it is probably caused by the wrong DATASETS (data path) in maskrcnn_benchmark/config/paths_catlog.py. If you are working on your custom datasets, just comment out the assertions.
Q: AssertionError on "l_batch == 1" in model_motifs.py A: The original MOTIFS code only supports evaluation on 1 GPU. Since my reimplemented motifs is based on their code, I keep this assertion to make sure it won't cause any unexpected errors.

Citations

If you find this project helps your research, please kindly consider citing our project or papers in your publications.

@misc{tang2020sggcode,
title = {A Scene Graph Generation Codebase in PyTorch},
author = {Tang, Kaihua},
year = {2020},
note = {\url{https://github.com/KaihuaTang/Scene-Graph-Benchmark.pytorch}},
}

@inproceedings{tang2018learning,
  title={Learning to Compose Dynamic Tree Structures for Visual Contexts},
  author={Tang, Kaihua and Zhang, Hanwang and Wu, Baoyuan and Luo, Wenhan and Liu, Wei},
  booktitle= "Conference on Computer Vision and Pattern Recognition",
  year={2019}
}

@inproceedings{tang2020unbiased,
  title={Unbiased Scene Graph Generation from Biased Training},
  author={Tang, Kaihua and Niu, Yulei and Huang, Jianqiang and Shi, Jiaxin and Zhang, Hanwang},
  booktitle= "Conference on Computer Vision and Pattern Recognition",
  year={2020}
}

Comments

Pretrained models with MODEL.ATTRIBUTE_ON set to true for SGDet on Custom Images

❓ Questions and Help

Thanks for this repository. I'm using the pretrained models you shared in Examples of Pretrained Causal MOTIFS-SUM models , the SGDet Download model for SGDet on Custom Images. The MODEL.ATTRIBUTE_ON was set to false in config.yaml of upload_causal_motif_sgdet(the SGDet Download model), and returned bbox.fields() got ['pred_labels', 'pred_scores', 'rel_pair_idxs', 'pred_rel_scores', 'pred_rel_labels'] successfully. But when I set MODEL.ATTRIBUTE_ON to true and wish to get 'pred_attributes', I got this bug:

RuntimeError: Error(s) in loading state_dict for GeneralizedRCNN:
        size mismatch for roi_heads.box.feature_extractor.fc7.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([2048, 4096]).
        size mismatch for roi_heads.box.feature_extractor.fc7.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([2048]).
        size mismatch for roi_heads.box.predictor.cls_score.weight: copying a param with shape torch.Size([151, 4096]) from checkpoint, the shape in current model is torch.Size([151, 2048]).
        size mismatch for roi_heads.box.predictor.bbox_pred.weight: copying a param with shape torch.Size([604, 4096]) from checkpoint, the shape in current model is torch.Size([604, 2048]).
        size mismatch for roi_heads.relation.union_feature_extractor.feature_extractor.fc7.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([2048, 4096]).
        size mismatch for roi_heads.relation.union_feature_extractor.feature_extractor.fc7.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([2048]).
        size mismatch for roi_heads.relation.box_feature_extractor.fc7.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([2048, 4096]).
        size mismatch for roi_heads.relation.box_feature_extractor.fc7.bias: copying a param with shape torch.Size([4096]) from checkpoint, the shape in current model is torch.Size([2048]).
        size mismatch for roi_heads.relation.predictor.context_layer.obj_ctx_rnn.weight_ih_l0: copying a param with shape torch.Size([2048, 4424]) from checkpoint, the shape in current model is torch.Size([2048, 4624]).
        size mismatch for roi_heads.relation.predictor.context_layer.obj_ctx_rnn.weight_ih_l0_reverse: copying a param with shape torch.Size([2048, 4424]) from checkpoint, the shape in current model is torch.Size([2048, 4624]).
        size mismatch for roi_heads.relation.predictor.context_layer.decoder_rnn.input_linearity.weight: copying a param with shape torch.Size([3072, 5136]) from checkpoint, the shape in current model is torch.Size([3072, 5536]).
        size mismatch for roi_heads.relation.predictor.context_layer.edge_ctx_rnn.weight_ih_l0: copying a param with shape torch.Size([2048, 4808]) from checkpoint, the shape in current model is torch.Size([2048, 5008]).
        size mismatch for roi_heads.relation.predictor.context_layer.edge_ctx_rnn.weight_ih_l0_reverse: copying a param with shape torch.Size([2048, 4808]) from checkpoint, the shape in current model is torch.Size([2048, 5008]).

I'm new to SSG and don't know why this happens. What should I do to using pretrained model to get attribute of custom images? I really appreciate your help.

opened by Opdoop 24

ImportError: cannot import name '_C' from 'maskrcnn_benchmark'

❓ Questions and Help

Hi, @KaihuaTang I am facing this issue with almost every repo I am trying to implement on scene graphs. Can you please tell what is the problem here?

from .nms import nms from maskrcnn_benchmark import _C ImportError: cannot import name '_C' from 'maskrcnn_benchmark'

opened by Kritz23 8
About the pair accuracy

❓ Questions and Help

I wonder if you can get the "PREDDET" (no graph constraints) results reported in the paper "Neural Motifs [CVPR 2018]" or the so-called "PREDCLS" in "Graph Contrastive Loss [CVPR 2019]". I know that this is actually the "PairAccuracy" in your codes. The results in those 2 papers are about 96%-98%. However, I only got about 84-85% with your PairAccuracy. I think that your code is right. But your paper did not report this result. So I may need your help. Thanks.

opened by Kenneth-Wong 8
About pretrained detection model
umm, it's me again. I notice that you release your pretrained faster-rcnn detection model. I download and evaluate it.

CUDA_VISIBLE_DEVICES=1 python -m torch.distributed.launch --master_port 10027 --nproc_per_node=1 tools/detector_pretest_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" TEST.IMS_PER_BATCH 1 DTYPE "float16" GLOVE_DIR /home/myname/glove MODEL.PRETRAINED_DETECTOR_CKPT ./checkpoints/pretrained_faster_rcnn OUTPUT_DIR ./checkpoints/pretrained_faster_rcnn

btw, I fixed a bug to run above command. The "last_checkpoint" string is the absolute path which contains "kaihua". We need modify some code in "./maskrcnn_benchmark/utils/checkpoint.py".

The result is "Detection evaluation mAp=0.2635", which is far from the value 0.296 mentioned in your paper. Is there something wrong with my evalution command?
opened by shine-lcy 8
Probable bug in image retrieval code

Hi, I was trying to use the code for the image retrieval task. The code in 'dataloader.pyand 'preprocessing.py seem to be somewhat inconsistent. In dataloader.py there is _generate_tensor_by_idx that looks for 'image_graph and 'text_graph key in the img_txt_sg variable

However in the preprocessing code the output dictionary created had only two keys img and txt

I was wondering if there is something missing in the code or if I am doing something wrong.

opened by mods333 5

Visualize SGDet

🐛 Bug

I have already prepared all required files, and other lines are all running correctly. But when I run the last line code, some errors are coming up.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-15-65157587e5d4> in <module>
----> 1 show_all(start_idx=707, length=1)
      2 # show_selected([119, 967, 713, 5224, 19681, 25371])

<ipython-input-6-1557c3d6a601> in show_all(start_idx, length)
      8         print(f'Image {cand_idx}:')
      9         img_path, boxes, labels, pred_labels, pred_scores, gt_rels, pred_rels, pred_rel_score, pred_rel_label = get_info_by_idx(cand_idx, detected_origin_result)
---> 10         draw_image(img_path=img_path, boxes=boxes, labels=labels, pred_labels=pred_labels, pred_scores=pred_scores, gt_rels=gt_rels, pred_rels=pred_rels, pred_rel_score=pred_rel_score, pred_rel_label=pred_rel_label, print_img=True)

<ipython-input-5-1fd503fdce41> in draw_image(img_path, boxes, labels, pred_labels, pred_scores, gt_rels, pred_rels, pred_rel_score, pred_rel_label, print_img)
     29         print_list('gt_rels', gt_rels, None)
     30         print('*' * 50)
---> 31     print_list('pred_labels', pred_labels, pred_rel_score)
     32     print('*' * 50)
     33     print_list('pred_rels', pred_rels, pred_rel_score)

<ipython-input-5-1fd503fdce41> in print_list(name, input_list, scores)
     10 def print_list(name, input_list, scores):
     11     for i, item in enumerate(input_list):
---> 12         if scores == None:
     13             print(name + ' ' + str(i) + ': ' + str(item))
     14         else:

TypeError: eq() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (!NoneType!)
 * (Number other)
      didn't match because some of the arguments have invalid types: (!NoneType!)

opened by ZhuYun97 5

can we test only one image by pre-trained model, not the whole test set？

❓ Questions and Help

Thanks for your wonderful work! i have a question. can we test only one image by pre-trained model, not the whole test set？ if so, what part of code should i change or what file should i prepare?

opened by forbiddenname 5

Training errors

❓ Questions and Help

Replace the backbone with ResNeSt and load pre-trained model, but loss is always nan. And after a long time, the program is corrupted.

.....
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 2.6727647100921956e-51
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.3363823550460978e-51
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 6.681911775230489e-52
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 3.3409558876152446e-52
2020-05-10 08:34:41,021 maskrcnn_benchmark INFO: eta: 1 day, 19:57:40  iter: 200  loss: nan (nan)  loss_classifier: nan (nan)  loss_box_reg: 0.0000 (nan)  loss_objectness: 0.6554 (3.2355)  loss_rpn_box_reg: 0.0783 (0.1020)  time: 1.6072 (1.5858)  data: 0.0132 (0.0184)  lr: 0.023000  max mem: 8743
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1.6704779438076223e-52
...
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 1e-323
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 5e-324
Gradient overflow.  Skipping step, loss scaler 0 reducing loss scale to 0.0

Traceback (most recent call last):
  File "tools/detector_pretrain_net.py", line 317, in <module>
    main()
  File "tools/detector_pretrain_net.py", line 310, in main
    model = train(cfg, args.local_rank, args.distributed, logger)
  File "tools/detector_pretrain_net.py", line 128, in train
    scaled_losses.backward()
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/contextlib.py", line 120, in __exit__
    next(self.gen)
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/site-packages/apex-0.1-py3.8-linux-x86_64.egg/apex/amp/handle.py", line 123, in scale_loss
    optimizer._post_amp_backward(loss_scaler)
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/site-packages/apex-0.1-py3.8-linux-x86_64.egg/apex/amp/_process_optimizer.py", line 249, in post_backward_no_master_weights
    post_backward_models_are_masters(scaler, params, stashed_grads)
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/site-packages/apex-0.1-py3.8-linux-x86_64.egg/apex/amp/_process_optimizer.py", line 131, in post_backward_models_are_masters
    scaler.unscale_with_stashed(
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/site-packages/apex-0.1-py3.8-linux-x86_64.egg/apex/amp/scaler.py", line 176, in unscale_with_stashed
    out_scale/grads_have_scale,   # 1./scale,
ZeroDivisionError: float division by zero
Traceback (most recent call last):
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/mist/anaconda3/envs/scene_graph_benchmark/lib/python3.8/site-packages/torch/distributed/launch.py", line 258, in main
    raise subprocess.CalledProcessError(returncode=process.returncode,
subprocess.CalledProcessError: Command '['/home/mist/anaconda3/envs/scene_graph_benchmark/bin/python', '-u', 'tools/detector_pretrain_net.py', '--local_rank=0', '--config-file', 'configs/e2e_det_ResNeSt_FPN.yaml', 'SOLVER.IMS_PER_BATCH', '4', 'TEST.IMS_PER_BATCH', '1', 'DTYPE', 'float16', 'SOLVER.MAX_ITER', '100000', 'SOLVER.STEPS', '(30000, 45000)', 'SOLVER.VAL_PERIOD', '2000', 'SOLVER.CHECKPOINT_PERIOD', '2000', 'MODEL.RELATION_ON', 'False', 'OUTPUT_DIR', '/home/mist/checkpoints/resnest', 'SOLVER.PRE_VAL', 'False', 'SOLVER.BASE_LR', '0.01', 'GLOVE_DIR', '/home/mist/glove', 'MODEL.WEIGHT', '/home/mist/checkpoints/resnest/resnest_det_new.pth']' returned non-zero exit status 1.

opened by ZhuYun97 5

CUDA error: device-side assert triggered

❓ Questions and Help

Hi, Thanks for your work. I use the VG100K dataset and your code. I am using the command of Training Example 2 : (SGCls, Causal, TDE, SUM Fusion, MOTIFS Model) to run the code.

For me, it is okay to run the "validation before training" part. SGG eval: A @ 20: 0.0000; A @ 50: 0.0000; A @ 100: 0.0000; for mode=sgcls, type=TopK Accuracy.

However, I met an error in training. File "./maskrcnn_benchmark/modeling/rpn/loss.py", line 106, in call sampled_pos_inds, sampled_neg_inds = self.fg_bg_sampler(labels) File "./maskrcnn_benchmark/modeling/balanced_positive_negative_sampler.py", line 53, in call neg_idx_per_image = negative[perm2] RuntimeError: CUDA error: device-side assert triggered index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.

I guess this probably means that your class labels are larger than the number of outputs from the model. But after checking, it seems fine.

I print the negative and perm2 and found that the elements inside the perm2 which is used as index are too large. Do you have this kind of error before? Hope to get your reply soom.

Thanks, Mengya

opened by XuMengyaAmy 4
cant use pretrained Faster R-CNN

❓ Questions and Help

Hello people, I got very strange issue. I pretrained Faster R-CNN with attributes on Visual genome using the following command: CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10001 --nproc_per_node=1 tools/detector_pretrain_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 2 TEST.IMS_PER_BATCH 1 DTYPE "float16" SOLVER.MAX_ITER 50000 SOLVER.STEPS "(30000, 45000)" SOLVER.VAL_PERIOD 20000 SOLVER.CHECKPOINT_PERIOD 20000 MODEL.RELATION_ON False OUTPUT_DIR /home/lkochiev/checkpoints/pretrained_faster_rcnn SOLVER.PRE_VAL False

So after it I obtained /home/lkochiev/checkpoints/pretrained_faster_rcnn/model_final.pth, with following performance: Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.111 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.251 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.085 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.047 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.095 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.132 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.206 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.332 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.343 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.234 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.332 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.340

After it I tried to train scene graph generator using this pretrained object detector and I ran the following command: CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10025 --nproc_per_node=1 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX False MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False SOLVER.IMS_PER_BATCH 2 TEST.IMS_PER_BATCH 1 DTYPE "float16" SOLVER.MAX_ITER 50000 SOLVER.VAL_PERIOD 2000 SOLVER.CHECKPOINT_PERIOD 2000 GLOVE_DIR /home/lkochiev/Documents/SFU/NSM/SGB/Scene-Graph-Benchmark.pytorch/ MODEL.PRETRAINED_DETECTOR_CKPT /home/lkochiev/checkpoints/pretrained_faster_rcnn/model_final.pth OUTPUT_DIR /home/lkochiev/checkpoints/motif-precls-exmp and strangely, I got a lot of NO-MATCHING of current module and REMATCHING! , but this is not the main problem. The main problem is evaluation. At first evaluation, before training I got very low performance:

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.006 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.007 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.006 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.007

Which is way different from the numbers which were outputted while training the detector. So after it I decided to evaluate my detector by running CUDA_VISIBLE_DEVICES=0 python -m torch.distributed.launch --master_port 10025 --nproc_per_node=1 tools/detector_pretest_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" MODEL.PRETRAINED_DETECTOR_CKPT /home/lkochiev/checkpoints/pretrained_faster_rcnn/model_final.pth. But somehow this script outputted me similar performance:

Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.006 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.007 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.006 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.007

like the model was never trained. I can't understand what I am doing wrong here. Thanks in advance for your guidance!

opened by nullkatar 4
Some questions about SGCls Predictions

❓ Questions and Help

SGCls: given bounding boxes and image, predicting labels and relationships.
It means the model needs to predict labels about bounding boxes. I print the predictions of the SGCls model, but the pred_scores are all 1. I am confused about why these values are not ranging from 0 to 1. Could you help me to understand it?

opened by ZhuYun97 4
Apply detectron2 in the SGDet

Hello, I found that the speed of running SGDet with the current maskrcnn benchmark is pretty slow. So, I want to integrate detectron2 with neural motif and run SGDet. Can anyone give me hint on which files should I need to change for changing the detector? I tried but I get stuck getting CUDA related errors. Thank you!

opened by bibekyess 0

AttributeError: module 'torchvision' has no attribute 'datasets'

❓ Questions and Help

hello, thank you for the great job. when i run the command bash train.sh Language_VG_Uniter a problem happened below:

Traceback (most recent call last):
  File "tools/relation_train_net.py", line 19, in <module>
    from maskrcnn_benchmark.data import make_data_loader
  File "/home/liujingwei/icme/sgg_from_nls/maskrcnn_benchmark/data/__init__.py", line 2, in <module>
    from .build import make_data_loader, get_dataset_statistics
  File "/home/liujingwei/icme/sgg_from_nls/maskrcnn_benchmark/data/build.py", line 14, in <module>
    from . import datasets as D
  File "/home/liujingwei/icme/sgg_from_nls/maskrcnn_benchmark/data/datasets/__init__.py", line 2, in <module>
    from .coco import COCODataset
  File "/home/liujingwei/icme/sgg_from_nls/maskrcnn_benchmark/data/datasets/coco.py", line 39, in <module>
    class COCODataset(torchvision.datasets.coco.CocoDetection):
AttributeError: module 'torchvision' has no attribute 'datasets'

I dont know if the torchvision is wrong or something happened.

opened by PrimoWW 1

AssertionError assert rois.size(0) > 0

❓ Questions and Help

Hi Kaihua, thanks for your great work. When I am using this codebase with batch_size=1 and SGCLS setting, the following assertion error would occur occasionally. I have checked the dataset while it seems that all image would have at least 1 bounding box, so I couldn't figure out the problem. Any idea? Thanks.

opened by yangzhaopeng3 0

custom_prediction.json file size is too large

❓ Questions and Help

During running in custom dataset... I got the error:

INIT SAVE DIR checkpoints/causal-motifs-sgdet
get_checkpoint_file checkpoints/causal-motifs-sgdet/last_checkpoint
last_saved checkpoints/causal-motifs-sgdet/model_0028000.pth
2022-09-13 19:49:41,654 maskrcnn_benchmark.utils.checkpoint INFO: Loading checkpoint from checkpoints/causal-motifs-sgdet/model_0028000.pth
100%|████████████████████████████████████████████████████████████████████████████████| 12195/12195 [03:49<00:00, 53.16it/s]
=====> output_dir/01/custom_data_info.json SAVED !
2022-09-13 19:53:50,023 maskrcnn_benchmark.inference INFO: Start evaluation on VG_stanford_filtered_with_attribute_test dataset(12195 images).
100%|██████████████████████████████████████████████████████████████████████████████| 12195/12195 [2:32:32<00:00,  1.33it/s]
2022-09-13 22:26:23,093 maskrcnn_benchmark.inference INFO: Total run time: 2:32:33.070335 (0.750559273055461 s / img per device, on 1 devices)
2022-09-13 22:26:23,093 maskrcnn_benchmark.inference INFO: Model inference time: 2:32:03.855223 (0.7481636098976652 s / img per device, on 1 devices)
Traceback (most recent call last):
  File "/home/ywjang/miniconda3/envs/usgg/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/ywjang/miniconda3/envs/usgg/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/ywjang/miniconda3/envs/usgg/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in <module>
    main()
  File "/home/ywjang/miniconda3/envs/usgg/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/ywjang/miniconda3/envs/usgg/bin/python', '-u', 'tools/relation_test_net.py', '--local_rank=0', '--config-file', 'configs/e2e_relation_X_101_32_8_FPN_1x.yaml', 'MODEL.ROI_RELATION_HEAD.USE_GT_BOX', 'False', 'MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL', 'False', 'MODEL.ROI_RELATION_HEAD.PREDICTOR', 'CausalAnalysisPredictor', 'MODEL.ROI_RELATION_HEAD.CAUSAL.EFFECT_TYPE', 'TDE', 'MODEL.ROI_RELATION_HEAD.CAUSAL.FUSION_TYPE', 'sum', 'MODEL.ROI_RELATION_HEAD.CAUSAL.CONTEXT_LAYER', 'motifs', 'TEST.IMS_PER_BATCH', '1', 'DTYPE', 'float16', 'GLOVE_DIR', '/data/glove', 'MODEL.PRETRAINED_DETECTOR_CKPT', 'checkpoints/causal-motifs-sgdet', 'OUTPUT_DIR', 'checkpoints/causal-motifs-sgdet', 'TEST.CUSTUM_EVAL', 'True', 'TEST.CUSTUM_PATH', '/data/AnotherMissOh/images_per_episode/01/', 'DETECTED_SGG_DIR', 'output_dir/01/']' died with <Signals.SIGKILL: 9>.

It died with <Signals.SIGKILL: 9>.

So, I run the tools/relation_test_net.py in only 10 images, it successfully executed, but the output file is too big:

ls output_dir/images_10/ -hl
total 60M
-rw-r--r-- 1 root root 2.5K  9월 12 16:50 custom_data_info.json
-rw-r--r-- 1 root root  60M  9월 12 16:51 custom_prediction.json

6 Megabytes per 1 image...

What's the problem? Should I edit some config files? where or what?

Thanks.

opened by greeksharifa 0

Understanding Training metrics

❓ Questions and Help

Hello! I have some general questions on understanding the metrics used in training. From the example below:

2022-08-13 19:12:26,520 maskrcnn_benchmark INFO: eta: 1:05:16 iter: 26000 loss: 0.0018 (0.0215) loss_refine_att: 0.0003 (0.0013) loss_refine_obj: 0.0014 (0.0201) loss_rel: 0.0000 (0.0002) time: 0.9628 (0.9792) data: 0.0123 (0.0197) lr: 0.000009 max mem: 8478 What is this general loss? And how is it different from the loss value inside parenthesis () and what about data? What does data here refer to? I would greatly appreciate your answer.

Thank you and have a good day!

opened by bibekyess 0
SGDet model can't be evaluated

❓ Questions and Help

I encounter another problem that SGDet model of Neural-Motifs (with and without attributes, tried both of them) can not be evaluated. It can be trained ,but failed in Starting validating(Starting evaluation on IMP_sgdet.txt motif_sgdet.txt IMP_sgdet.txt VG_stanford_filtered_with_attribute_val dataset(5000 images)).

there is two problem (1)OSError: [Errno 12] Cannot allocate memory .(2) RuntimeError: DataLoader worker (pid 48413) is killed by signal: Killed.When I searching the problem in google, they said that the proble may be not enough RAM memory(my RAM memory is 32 G, with 2 2080Ti,per gpu with 11G memory) when validation. Do you know how to reduce the memory occupation? I can not find the a command to solve the problem. the attachment is the runing log with motifs and IMP.

opened by Ceazy-Gentleman 1

PyTorch implementation of paper “Unbiased Scene Graph Generation from Biased Training”

Related tags

Overview

Scene Graph Benchmark in Pytorch

Recent Updates

Contents

Overview

Benefit from the up-to-date Faster R-CNN in maskrcnn-benchmark, this codebase achieves new state-of-the-art Recall@k on SGCls & SGGen (by 2020.2.16) through the reimplemented VCTree using two 1080ti GPUs and batch size 8:

The illustration of the Unbiased SGG from 'Unbiased Scene Graph Generation from Biased Training'

Installation

Dataset

Metrics and Results (IMPORTANT)

Pretrained Models

Faster R-CNN pre-training

Scene Graph Generation as RoI_Head

Perform training on Scene Graph Generation

Predefined Models

Customize Your Own Model

The proposed Causal TDE on Unbiased Scene Graph Generation from Biased Training

Examples of the Training Command

Evaluation

Examples of the Test Command

Examples of Pretrained Causal MOTIFS-SUM models

SGDet on Custom Images

Visualize Detected SGs of Custom Images

Other Options that May Improve the SGG

Tips and Tricks for any Unbiased TaskX from Biased Training

Frequently Asked Questions:

Citations

Comments

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

🐛 Bug

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

❓ Questions and Help

Owner

Kaihua Tang

Pytorch implementation of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

Pytorch implementation of the AAAI 2022 paper "Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification"

PyTorch code for ICLR 2021 paper Unbiased Teacher for Semi-Supervised Object Detection

Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

Official PyTorch implementation of "ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows"

Official Pytorch implementation of "Unbiased Classification Through Bias-Contrastive and Bias-Balanced Learning (NeurIPS 2021)

Simple Tensorflow implementation of Toward Spatially Unbiased Generative Models (ICCV 2021)

Toward Spatially Unbiased Generative Models (ICCV 2021)

[ICCV 2021] Released code for Causal Attention for Unbiased Visual Recognition

Allele-specific pipeline for unbiased read mapping(WIP), QTL discovery(WIP), and allelic-imbalance analysis

Code for CVPR 2021 oral paper "Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts"

Code for "Learning Canonical Representations for Scene Graph to Image Generation", Herzig & Bar et al., ECCV2020

image scene graph generation benchmark

Code for our paper "Graph Pre-training for AMR Parsing and Generation" in ACL2022

[TIP 2020] Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Neural Scene Graphs for Dynamic Scene (CVPR 2021)

Automatic number plate recognition using tech: Yolo, OCR, Scene text detection, scene text recognation, flask, torch

PyTorch implementation of paper "Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes", CVPR 2021

A pytorch implementation of the CVPR2021 paper "VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild"