Prerequisite
Task
I'm using the official example scripts/configs for the officially supported tasks/models/datasets.
Branch
master branch https://github.com/open-mmlab/mmrotate
Environment
(mmroate) ➜ mmrotate git:(qd) ✗ python mmrotate/utils/collect_env.py
sys.platform: linux
Python: 3.8.15 (default, Nov 24 2022, 15:19:38) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 2147483648
GPU 0: Tesla V100-SXM2-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.4, V11.4.152
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.10.0+cu111
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.11.0+cu111
OpenCV: 4.6.0
MMEngine: 0.4.0
MMRotate: 1.0.0rc0+3db9cc0
Reproduces the problem - code sample
python tools/train.py configs/rotated_rtmdet/rotated_rtmdet_l-3x-dota.py
Reproduces the problem - command or script
python tools/train.py configs/rotated_rtmdet/rotated_rtmdet_l-3x-dota.py
Reproduces the problem - error message
(mmroate) ➜ mmrotate git:(qd) ✗ python tools/train.py configs/rotated_rtmdet/rotated_rtmdet_l-3x-dota.py
01/02 00:48:06 - mmengine - WARNING - The "log_processor" registry in mmrotate did not set import location. Fallback to call `mmrotate.utils.register_all_modules` instead.
01/02 00:48:06 - mmengine - INFO -
------------------------------------------------------------
System environment:
sys.platform: linux
Python: 3.8.15 (default, Nov 24 2022, 15:19:38) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 1418043970
GPU 0: Tesla V100-SXM2-32GB
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.4, V11.4.152
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.10.0+cu111
PyTorch compiling details: PyTorch built with:
- GCC 7.3
- C++ Version: 201402
- Intel(R) Math Kernel Library Version 2020.0.0 Product Build 20191122 for Intel(R) 64 architecture applications
- Intel(R) MKL-DNN v2.2.3 (Git Hash 7336ca9f055cf1bfa13efb658fe15dc9b41f0740)
- OpenMP 201511 (a.k.a. OpenMP 4.5)
- LAPACK is enabled (usually provided by MKL)
- NNPACK is enabled
- CPU capability usage: AVX512
- CUDA Runtime 11.1
- NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86
- CuDNN 8.0.5
- Magma 2.5.2
- Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,
TorchVision: 0.11.0+cu111
OpenCV: 4.6.0
MMEngine: 0.4.0
Runtime environment:
cudnn_benchmark: False
mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
dist_cfg: {'backend': 'nccl'}
seed: None
Distributed launcher: none
Distributed training: False
GPU number: 1
------------------------------------------------------------
01/02 00:48:07 - mmengine - INFO - Config:
default_scope = 'mmrotate'
default_hooks = dict(
timer=dict(type='IterTimerHook'),
logger=dict(type='LoggerHook', interval=1),
param_scheduler=dict(type='ParamSchedulerHook'),
checkpoint=dict(type='CheckpointHook', interval=12, max_keep_ckpts=3),
sampler_seed=dict(type='DistSamplerSeedHook'),
visualization=dict(type='mmdet.DetVisualizationHook'))
env_cfg = dict(
cudnn_benchmark=False,
mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
dist_cfg=dict(backend='nccl'))
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
type='RotLocalVisualizer',
vis_backends=[dict(type='LocalVisBackend')],
name='visualizer')
log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True)
log_level = 'INFO'
load_from = None
resume = False
custom_hooks = [
dict(type='mmdet.NumClassCheckHook'),
dict(
type='EMAHook',
ema_type='mmdet.ExpMomentumEMA',
momentum=0.0002,
update_buffers=True,
priority=49)
]
max_epochs = 36
base_lr = 0.00025
interval = 12
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=36, val_interval=12)
val_cfg = dict(type='ValLoop')
test_cfg = dict(type='TestLoop')
param_scheduler = [
dict(
type='LinearLR', start_factor=1e-05, by_epoch=False, begin=0,
end=1000),
dict(
type='CosineAnnealingLR',
eta_min=1.25e-05,
begin=18,
end=36,
T_max=18,
by_epoch=True,
convert_to_iter_based=True)
]
optim_wrapper = dict(
type='OptimWrapper',
optimizer=dict(type='AdamW', lr=0.00025, weight_decay=0.05),
paramwise_cfg=dict(
norm_decay_mult=0, bias_decay_mult=0, bypass_duplicate=True))
dataset_type = 'DOTADataset'
data_root = 'data/icdar2019_tracka_modern_qbox/'
file_client_args = dict(backend='disk')
METAINFO = dict(classes=('table', ))
train_pipeline = [
dict(
type='mmdet.LoadImageFromFile', file_client_args=dict(backend='disk')),
dict(type='mmdet.LoadAnnotations', with_bbox=True, box_type='qbox'),
dict(type='ConvertBoxType', box_type_mapping=dict(gt_bboxes='rbox')),
dict(type='mmdet.Resize', scale=(1024, 1024), keep_ratio=True),
dict(
type='mmdet.RandomFlip',
prob=0.75,
direction=['horizontal', 'vertical', 'diagonal']),
dict(
type='RandomRotate',
prob=0.5,
angle_range=180,
rect_obj_labels=[9, 11]),
dict(
type='mmdet.Pad', size=(1024, 1024),
pad_val=dict(img=(114, 114, 114))),
dict(type='mmdet.PackDetInputs')
]
val_pipeline = [
dict(
type='mmdet.LoadImageFromFile', file_client_args=dict(backend='disk')),
dict(type='mmdet.Resize', scale=(1024, 1024), keep_ratio=True),
dict(type='mmdet.LoadAnnotations', with_bbox=True, box_type='qbox'),
dict(type='ConvertBoxType', box_type_mapping=dict(gt_bboxes='rbox')),
dict(
type='mmdet.Pad', size=(1024, 1024),
pad_val=dict(img=(114, 114, 114))),
dict(
type='mmdet.PackDetInputs',
meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]
test_pipeline = [
dict(
type='mmdet.LoadImageFromFile', file_client_args=dict(backend='disk')),
dict(type='mmdet.Resize', scale=(1024, 1024), keep_ratio=True),
dict(
type='mmdet.Pad', size=(1024, 1024),
pad_val=dict(img=(114, 114, 114))),
dict(
type='mmdet.PackDetInputs',
meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]
train_dataloader = dict(
batch_size=4,
num_workers=4,
persistent_workers=True,
sampler=dict(type='DefaultSampler', shuffle=True),
batch_sampler=None,
pin_memory=False,
dataset=dict(
type='DOTADataset',
metainfo=dict(classes=('table', )),
data_root='data/icdar2019_tracka_modern_qbox/',
ann_file='train_rotate_qbox/',
data_prefix=dict(img_path='train_rotate_img/'),
img_shape=(1024, 1024),
filter_cfg=dict(filter_empty_gt=True),
pipeline=[
dict(
type='mmdet.LoadImageFromFile',
file_client_args=dict(backend='disk')),
dict(
type='mmdet.LoadAnnotations', with_bbox=True, box_type='qbox'),
dict(
type='ConvertBoxType',
box_type_mapping=dict(gt_bboxes='rbox')),
dict(type='mmdet.Resize', scale=(1024, 1024), keep_ratio=True),
dict(
type='mmdet.RandomFlip',
prob=0.75,
direction=['horizontal', 'vertical', 'diagonal']),
dict(
type='RandomRotate',
prob=0.5,
angle_range=180,
rect_obj_labels=[9, 11]),
dict(
type='mmdet.Pad',
size=(1024, 1024),
pad_val=dict(img=(114, 114, 114))),
dict(type='mmdet.PackDetInputs')
]))
val_dataloader = dict(
batch_size=1,
num_workers=2,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False),
dataset=dict(
type='DOTADataset',
data_root='data/icdar2019_tracka_modern_qbox/',
metainfo=dict(classes=('table', )),
ann_file='test_rotate_qbox/',
data_prefix=dict(img_path='test_rotate_img/'),
img_shape=(1024, 1024),
test_mode=True,
pipeline=[
dict(
type='mmdet.LoadImageFromFile',
file_client_args=dict(backend='disk')),
dict(type='mmdet.Resize', scale=(1024, 1024), keep_ratio=True),
dict(
type='mmdet.LoadAnnotations', with_bbox=True, box_type='qbox'),
dict(
type='ConvertBoxType',
box_type_mapping=dict(gt_bboxes='rbox')),
dict(
type='mmdet.Pad',
size=(1024, 1024),
pad_val=dict(img=(114, 114, 114))),
dict(
type='mmdet.PackDetInputs',
meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]))
test_dataloader = dict(
batch_size=1,
num_workers=2,
persistent_workers=True,
drop_last=False,
sampler=dict(type='DefaultSampler', shuffle=False),
dataset=dict(
type='DOTADataset',
data_root='data/icdar2019_tracka_modern_qbox/',
metainfo=dict(classes=('table', )),
ann_file='test_rotate_qbox/',
data_prefix=dict(img_path='test_rotate_img/'),
img_shape=(1024, 1024),
test_mode=True,
pipeline=[
dict(
type='mmdet.LoadImageFromFile',
file_client_args=dict(backend='disk')),
dict(type='mmdet.Resize', scale=(1024, 1024), keep_ratio=True),
dict(
type='mmdet.LoadAnnotations', with_bbox=True, box_type='qbox'),
dict(
type='ConvertBoxType',
box_type_mapping=dict(gt_bboxes='rbox')),
dict(
type='mmdet.Pad',
size=(1024, 1024),
pad_val=dict(img=(114, 114, 114))),
dict(
type='mmdet.PackDetInputs',
meta_keys=('img_id', 'img_path', 'ori_shape', 'img_shape',
'scale_factor'))
]))
val_evaluator = dict(type='DOTAMetric', metric='mAP')
test_evaluator = dict(type='DOTAMetric', metric='mAP')
checkpoint = 'https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-l_8xb256-rsb-a1-600e_in1k-6a760974.pth'
angle_version = 'le90'
model = dict(
type='mmdet.RTMDet',
data_preprocessor=dict(
type='mmdet.DetDataPreprocessor',
mean=[103.53, 116.28, 123.675],
std=[57.375, 57.12, 58.395],
bgr_to_rgb=False,
boxtype2tensor=False,
batch_augments=None),
backbone=dict(
type='mmdet.CSPNeXt',
arch='P5',
expand_ratio=0.5,
deepen_factor=1,
widen_factor=1,
channel_attention=True,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU'),
init_cfg=dict(
type='Pretrained',
prefix='backbone.',
checkpoint=
'https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-l_8xb256-rsb-a1-600e_in1k-6a760974.pth'
)),
neck=dict(
type='mmdet.CSPNeXtPAFPN',
in_channels=[256, 512, 1024],
out_channels=256,
num_csp_blocks=3,
expand_ratio=0.5,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU')),
bbox_head=dict(
type='RotatedRTMDetSepBNHead',
num_classes=1,
in_channels=256,
stacked_convs=2,
feat_channels=256,
angle_version='le90',
anchor_generator=dict(
type='mmdet.MlvlPointGenerator', offset=0, strides=[8, 16, 32]),
bbox_coder=dict(type='DistanceAnglePointCoder', angle_version='le90'),
loss_cls=dict(
type='mmdet.QualityFocalLoss',
use_sigmoid=True,
beta=2.0,
loss_weight=1.0),
loss_bbox=dict(type='RotatedIoULoss', mode='linear', loss_weight=2.0),
with_objectness=False,
exp_on_reg=True,
share_conv=True,
pred_kernel_size=1,
use_hbbox_loss=False,
scale_angle=False,
loss_angle=None,
norm_cfg=dict(type='SyncBN'),
act_cfg=dict(type='SiLU')),
train_cfg=dict(
assigner=dict(
type='mmdet.DynamicSoftLabelAssigner',
iou_calculator=dict(type='RBboxOverlaps2D'),
topk=13),
allowed_border=-1,
pos_weight=-1,
debug=False),
test_cfg=dict(
nms_pre=2000,
min_bbox_size=0,
score_thr=0.05,
nms=dict(type='nms_rotated', iou_threshold=0.1),
max_per_img=2000))
launcher = 'none'
work_dir = './work_dirs/rotated_rtmdet_l-3x-dota'
01/02 00:48:14 - mmengine - WARNING - The "optimizer" registry in mmrotate did not set import location. Fallback to call `mmrotate.utils.register_all_modules` instead.
01/02 00:48:14 - mmengine - WARNING - Failed to search registry with scope "mmrotate" in the "optim_wrapper" registry tree. As a workaround, the current "optim_wrapper" registry in "mmengine" is used to build instance. This may cause unexpected failure when running the built modules. Please check whether "mmrotate" is a correct scope, or whether the registry is initialized.
01/02 00:48:14 - mmengine - WARNING - The "parameter scheduler" registry in mmrotate did not set import location. Fallback to call `mmrotate.utils.register_all_modules` instead.
01/02 00:48:14 - mmengine - WARNING - The "metric" registry in mmrotate did not set import location. Fallback to call `mmrotate.utils.register_all_modules` instead.
01/02 00:48:15 - mmengine - WARNING - The "weight initializer" registry in mmrotate did not set import location. Fallback to call `mmrotate.utils.register_all_modules` instead.
01/02 00:48:15 - mmengine - INFO - load backbone. in model from: https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-l_8xb256-rsb-a1-600e_in1k-6a760974.pth
Loads checkpoint by http backend from path: https://download.openmmlab.com/mmdetection/v3.0/rtmdet/cspnext_rsb_pretrain/cspnext-l_8xb256-rsb-a1-600e_in1k-6a760974.pth
01/02 00:48:15 - mmengine - INFO - Checkpoints will be saved to /home/ubuntu/mmroate-1.x/mmrotate/work_dirs/rotated_rtmdet_l-3x-dota.
/home/ubuntu/mmroate-1.x/mmrotate/mmrotate/structures/bbox/rotated_boxes.py:192: UserWarning: The `clip` function does nothing in `RotatedBoxes`.
warnings.warn('The `clip` function does nothing in `RotatedBoxes`.')
/home/ubuntu/mmroate-1.x/mmrotate/mmrotate/structures/bbox/rotated_boxes.py:192: UserWarning: The `clip` function does nothing in `RotatedBoxes`.
warnings.warn('The `clip` function does nothing in `RotatedBoxes`.')
/home/ubuntu/mmroate-1.x/mmrotate/mmrotate/structures/bbox/rotated_boxes.py:192: UserWarning: The `clip` function does nothing in `RotatedBoxes`.
warnings.warn('The `clip` function does nothing in `RotatedBoxes`.')
/home/ubuntu/mmroate-1.x/mmrotate/mmrotate/structures/bbox/rotated_boxes.py:192: UserWarning: The `clip` function does nothing in `RotatedBoxes`.
warnings.warn('The `clip` function does nothing in `RotatedBoxes`.')
/home/ubuntu/miniconda3/envs/mmroate/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
Traceback (most recent call last):
File "tools/train.py", line 122, in <module>
main()
File "tools/train.py", line 118, in main
runner.train()
File "/home/ubuntu/mmroate-1.x/mmengine/mmengine/runner/runner.py", line 1678, in train
model = self.train_loop.run() # type: ignore
File "/home/ubuntu/mmroate-1.x/mmengine/mmengine/runner/loops.py", line 90, in run
self.run_epoch()
File "/home/ubuntu/mmroate-1.x/mmengine/mmengine/runner/loops.py", line 106, in run_epoch
self.run_iter(idx, data_batch)
File "/home/ubuntu/mmroate-1.x/mmengine/mmengine/runner/loops.py", line 122, in run_iter
outputs = self.runner.model.train_step(
File "/home/ubuntu/mmroate-1.x/mmengine/mmengine/model/base_model/base_model.py", line 114, in train_step
losses = self._run_forward(data, mode='loss') # type: ignore
File "/home/ubuntu/mmroate-1.x/mmengine/mmengine/model/base_model/base_model.py", line 314, in _run_forward
results = self(**data, mode=mode)
File "/home/ubuntu/miniconda3/envs/mmroate/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/home/ubuntu/mmroate-1.x/mmdetection/mmdet/models/detectors/base.py", line 92, in forward
return self.loss(inputs, data_samples)
File "/home/ubuntu/mmroate-1.x/mmdetection/mmdet/models/detectors/single_stage.py", line 78, in loss
losses = self.bbox_head.loss(x, batch_data_samples)
File "/home/ubuntu/mmroate-1.x/mmdetection/mmdet/models/dense_heads/base_dense_head.py", line 123, in loss
losses = self.loss_by_feat(*loss_inputs)
File "/home/ubuntu/mmroate-1.x/mmrotate/mmrotate/models/dense_heads/rotated_rtmdet_head.py", line 299, in loss_by_feat
cls_reg_targets = self.get_targets(
File "/home/ubuntu/mmroate-1.x/mmdetection/mmdet/models/dense_heads/rtmdet_head.py", line 355, in get_targets
(all_anchors, all_labels, all_label_weights, all_bbox_targets,
ValueError: not enough values to unpack (expected 6, got 5)
### Additional information
```shell
python tools/train.py configs/rotated_rtmdet/rotated_rtmdet_l-3x-dota.py