While training the model on dota1.0, gpu memory keeps on increasing after every few iterations until it runs out of memory. I have tried both docker and without docker training. Any ideas on why this is happening?

I build the docker image with docker build -t dafne . After that I tried running the command from the readme (only --gpus 0 would work, or else it will return a unknown device error): ./tools/run.py --gpus 0 --config-file ./configs/dota-1.0/1024.yaml

This returned the following error:

Traceback:
  File "./tools/plain_train_net.py", line 597, in main
    do_train(cfg, model, resume=args.resume)
  File "./tools/plain_train_net.py", line 430, in do_train
    data_loader_train = build_train_loader(cfg)
  File "./tools/plain_train_net.py", line 274, in build_train_loader
    return build_detection_train_loader(cfg, mapper=mapper)
  File "/app/detectron2_repo/detectron2/config/config.py", line 207, in wrapped
    explicit_args = _get_args_from_config(from_config, *args, **kwargs)
  File "/app/detectron2_repo/detectron2/config/config.py", line 245, in _get_args_from_config
    ret = from_config_func(*args, **kwargs)
  File "/app/detectron2_repo/detectron2/data/build.py", line 337, in _train_loader_from_config
    dataset = get_detection_dataset_dicts(
  File "/app/detectron2_repo/detectron2/data/build.py", line 240, in get_detection_dataset_dicts
    dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names]
  File "/app/detectron2_repo/detectron2/data/build.py", line 240, in <listcomp>
    dataset_dicts = [DatasetCatalog.get(dataset_name) for dataset_name in names]
  File "/app/detectron2_repo/detectron2/data/catalog.py", line 58, in get
    return f()
  File "/app/dafne/dafne/data/datasets/dota.py", line 346, in <lambda>
    lambda: load_dota_json(
  File "/app/dafne/dafne/data/datasets/dota.py", line 89, in load_dota_json
    coco_api = COCO(json_file)
  File "/opt/conda/lib/python3.8/site-packages/pycocotools/coco.py", line 84, in __init__
    with open(annotation_file, 'r') as f:

Error: [Errno 2] No such file or directory: '/app/data/dota_1_split/train1024/DOTA1_train1024.json'

I'm not really familiar with docker, do I need to run this command from inside the docker image? I tried to run the command without docker from the readme: NVIDIA_VISIBLE_DEVICES=0 ./tools/plain_train_net.py --num-gpus 1 --config-file ./configs/dota-1.0/1024.yaml

But this just returns another error:

  File "./tools/plain_train_net.py", line 90
    def build_optimizer_custom(cfg: CfgNode, model: torch.nn.Module) -> torch.optim.Optimizer:
                                  ^
SyntaxError: invalid syntax

/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] /home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3190.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] Traceback: File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 601, in main do_train(cfg, model, resume=args.resume) File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 468, in do_train optimizer.step() File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper return wrapped(*args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad ret = func(self, *args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 151, in step sgd(params_with_grad, File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 202, in sgd func(params, File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 229, in _single_tensor_sgd d_p = d_p.add(param, alpha=weight_decay)

Error: add(): argument 'alpha' must be Number, not NoneType Traceback: File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 601, in main do_train(cfg, model, resume=args.resume) File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 468, in do_train optimizer.step() File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper return wrapped(*args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad ret = func(self, *args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 151, in step sgd(params_with_grad, File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 202, in sgd func(params, File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 229, in _single_tensor_sgd d_p = d_p.add(param, alpha=weight_decay)

Error: add(): argument 'alpha' must be Number, not NoneType Traceback (most recent call last): File "./tools/plain_train_net.py", line 664, in launch( File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/detectron2/engine/launch.py", line 67, in launch mp.spawn( File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 240, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method='spawn') File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 198, in start_processes while not context.join(): File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 160, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 1 terminated with the following error: Traceback (most recent call last): File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap fn(i, *args) File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/detectron2/engine/launch.py", line 126, in _distributed_worker main_func(*args) File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 655, in main raise e File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 601, in main do_train(cfg, model, resume=args.resume) File "/media/XXW/98A0C693A0C676F2/ubuntu-temp/DAFNe-master/tools/plain_train_net.py", line 468, in do_train optimizer.step() File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/lr_scheduler.py", line 68, in wrapper return wrapped(*args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad ret = func(self, *args, **kwargs) File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 151, in step sgd(params_with_grad, File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 202, in sgd func(params, File "/home/XXW/anaconda3/envs/DAFNe/lib/python3.8/site-packages/torch/optim/sgd.py", line 229, in _single_tensor_sgd d_p = d_p.add(param, alpha=weight_decay) TypeError: add(): argument 'alpha' must be Number, not NoneType

Can you make some suggestions？ Thank you very much!

Hi, could you please guide me on how to change the code to work with a custom dataset with new labels (and different number of categories)? I did see #3 but I couldn't find anything out about new categories. Thanks.

When running the command ./tools/run.py --gpus 0 --config-file ./configs/dota-1.0/1024.yaml I receive the following error:

[02/03 12:13:18 d2.data.build]: Removed 5478 images with no usable annotations. 10271 images left. [02/03 12:13:18 d2.data.build]: Distribution of instances among all 15 categories: | category | #instances | category | #instances | category | #instances | |:-------------:|:-------------|:-------------:|:-------------|:-------------:|:-------------| | plane | 16411 | baseball-di.. | 823 | bridge | 3632 | | ground-trac.. | 838 | small-vehicle | 49810 | large-vehicle | 36233 | | ship | 61019 | tennis-court | 4736 | basketball-.. | 1118 | | storage-tank | 9621 | soccer-ball.. | 849 | roundabout | 800 | | harbor | 13078 | swimming-pool | 3323 | helicopter | 1188 | | | | | | | | | total | 203479 | | | | | [02/03 12:13:18 d2.data.build]: Using training sampler RepeatFactorTrainingSampler [02/03 12:13:18 d2.data.common]: Serializing 10271 elements to byte tensors and concatenating them all ... [02/03 12:13:19 d2.data.common]: Serialized dataset takes 15.84 MiB [02/03 12:13:19 detectron2]: Starting training from iteration 0 /app/detectron2_repo/detectron2/structures/masks.py:368: UserWarning: This overload of nonzero is deprecated: nonzero() Consider using one of the following signatures instead: nonzero(, bool as_tuple) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:983.) item = item.nonzero().squeeze(1).cpu().numpy().tolist() /app/detectron2_repo/detectron2/structures/masks.py:368: UserWarning: This overload of nonzero is deprecated: nonzero() Consider using one of the following signatures instead: nonzero(, bool as_tuple) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:983.) item = item.nonzero().squeeze(1).cpu().numpy().tolist() /app/detectron2_repo/detectron2/structures/masks.py:368: UserWarning: This overload of nonzero is deprecated: nonzero() Consider using one of the following signatures instead: nonzero(, bool as_tuple) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:983.) item = item.nonzero().squeeze(1).cpu().numpy().tolist() /app/detectron2_repo/detectron2/structures/masks.py:368: UserWarning: This overload of nonzero is deprecated: nonzero() Consider using one of the following signatures instead: nonzero(, bool as_tuple) (Triggered internally at ../torch/csrc/utils/python_arg_parser.cpp:983.) item = item.nonzero().squeeze(1).cpu().numpy().tolist() Traceback: File "./tools/plain_train_net.py", line 597, in main do_train(cfg, model, resume=args.resume) File "./tools/plain_train_net.py", line 450, in do_train loss_dict = model(data) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 881, in _call_impl result = self.forward(*input, **kwargs) File "/app/dafne/dafne/modeling/one_stage_detector.py", line 47, in forward return super().forward(batched_inputs) File "/app/detectron2_repo/detectron2/modeling/meta_arch/rcnn.py", line 301, in forward images = ImageList.from_tensors(images, self.backbone.size_divisibility) File "/app/detectron2_repo/detectron2/structures/image_list.py", line 88, in from_tensors max_size = (max_size + (stride - 1)).div(stride, rounding_mode="floor") * stride

Error: div() got an unexpected keyword argument 'rounding_mode' Moving output directory from /app/results/dafne/22-02-03_13:12_default-1643890350 to /app/results/dafne/22-02-03_13:12_default-1643890350_error Traceback (most recent call last): File "./tools/plain_train_net.py", line 660, in launch( File "/app/detectron2_repo/detectron2/engine/launch.py", line 82, in launch main_func(*args) File "./tools/plain_train_net.py", line 651, in main raise e File "./tools/plain_train_net.py", line 597, in main do_train(cfg, model, resume=args.resume) File "./tools/plain_train_net.py", line 450, in do_train loss_dict = model(data) File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 881, in _call_impl result = self.forward(*input, **kwargs) File "/app/dafne/dafne/modeling/one_stage_detector.py", line 47, in forward return super().forward(batched_inputs) File "/app/detectron2_repo/detectron2/modeling/meta_arch/rcnn.py", line 301, in forward images = ImageList.from_tensors(images, self.backbone.size_divisibility) File "/app/detectron2_repo/detectron2/structures/image_list.py", line 88, in from_tensors max_size = (max_size + (stride - 1)).div(stride, rounding_mode="floor") * stride TypeError: div() got an unexpected keyword argument 'rounding_mode'

According to this detectron2 issue this seems to be a problem with the pytorch version. When trying to build the project with this command docker build -t dafne . I had multiple problems. I had to add the keyword --upgrade to the pip installation of tensorboard because I was getting this error:

ERROR: Exception: Traceback (most recent call last): File "/opt/conda/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 165, in exc_logging_wrapper status = run_func(*args) File "/opt/conda/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper return func(self, options, args) File "/opt/conda/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 389, in run to_install = resolver.get_installation_order(requirement_set) File "/opt/conda/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 188, in get_installation_order weights = get_topological_weights( File "/opt/conda/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 276, in get_topological_weights assert len(weights) == expected_node_count AssertionError The command '/bin/sh -c pip install tensorboard' returned a non-zero code: 2

I also tried a different pytorch version by changing this line in the docker file: FROM nvcr.io/nvidia/pytorch:21.02-py3 but this only lead to other errors. Did the detectron2 project change since this repository was last updated? Are there any packages without a specific version number involved in the build process that might have changed?

Hello, i cloned the repo, followed the readme, installed requirements but am getting an error when i run >python plain_train_net.py

The error is being thrown from nms/nms.py imports: from poly_nms import poly_gpu_nms

I gather these are part of the DOTA devkit. i tried copying the contents of the poly_nms_gpu folder from the devkit into the nms folder but that did not fix the error. Are the imports for poly_nms and poly_overlaps meant to import the cython .pyx files? How do i set up things up for these imports to work correctly?

Thanks,

file "dafne.modeling.nms.nms.py" line 6 ->from poly_nms import poly_gpu_nms where can I find this poly_nms module? look forward to your answer

I download the DOTA v1.5 ，How should the data preprocessing be done? Whether the image is transmitted to the network is the original image or the cut image. Error: [Errno 2] No such file or directory:'/app/data/dota_1_split/train1024/DOTA1_train1024.json' What is this json file, is it automatically generated in your code or I need to write code to generate this json file