I changed nothing for the config file but only got 39.9 mAP when training res50 by myself. And I tested with your provided weight file and only got 31 mAP for res50 and 36 for res101.

Hi, thanks for your great work on OTA. I have tried to incorporate OTA into my model and found a huge difference in training time for dynamic k estimation vs fixed k settings. I am using 4 Tesla P100 to train. It takes about 2.5 days for fixed k while it takes more than 6 days to train for dynamic k estimation. I am wondering what your training time is like on both settings. Do you face similar issues? Thanks a lot!

I used pods_train --num-gpus 1 , but I got error: Traceback (most recent call last): File "/home/***/anaconda3/envs/torch1.8/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/***/anaconda3/envs/torch1.8/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/*/.vscode/extensions/ms-python.python-2021.5.926500501/pythonFiles/lib/python/debugpy/main.py", line 45, in cli.main() File "/home//.vscode/extensions/ms-python.python-2021.5.926500501/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main run() File "/home//.vscode/extensions/ms-python.python-2021.5.926500501/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file runpy.run_path(target_as_str, run_name=compat.force_str("main")) File "/home//anaconda3/envs/torch1.8/lib/python3.7/runpy.py", line 263, in run_path pkg_name=pkg_name, script_name=fname) File "/home//anaconda3/envs/torch1.8/lib/python3.7/runpy.py", line 96, in _run_module_code mod_name, mod_spec, pkg_name, script_name) File "/home//anaconda3/envs/torch1.8/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/media//D/05_source_code/01_torch/detection/cvpods/tools/train_net.py", line 124, in args=(args,), File "/media//D/05_source_code/01_torch/detection/cvpods/cvpods/engine/launch.py", line 56, in launch main_func(args) File "/media//D/05_source_code/01_torch/detection/cvpods/tools/train_net.py", line 110, in main runner.train() File "/media//D/05_source_code/01_torch/detection/cvpods/cvpods/engine/runner.py", line 271, in train super().train(self.start_iter, self.start_epoch, self.max_iter) File "/media//D/05_source_code/01_torch/detection/cvpods/cvpods/engine/base_runner.py", line 84, in train self.run_step() File "/media//D/05_source_code/01_torch/detection/cvpods/cvpods/engine/base_runner.py", line 185, in run_step loss_dict = self.model(data) File "/home//anaconda3/envs/torch1.8/lib/python3.7/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/media//D/05_source_code/01_torch/detection/cvpods/playground/detection/coco/ota.res50.fpn.coco.800size.1x/fcos.py", line 218, in forward gt_ious, box_cls, box_delta, box_iou) File "/media//D/05_source_code/01_torch/detection/cvpods/playground/detection/coco/ota.res50.fpn.coco.800size.1x/fcos.py", line 383, in losses dist.all_reduce(num_foreground) File "/home//anaconda3/envs/torch1.8/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1168, in all_reduce default_pg = _get_default_group() File "/home//anaconda3/envs/torch1.8/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 347, in _get_default_group raise RuntimeError("Default process group has not been initialized, " RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

how to train with one GPU? Thank you

Hi Thanks for providing wonderful work! I have a question about the sinkhorn distances implementation.

In the forward function of sinkhorn distances (below), you use log scale updating rule which is not same as original algorithm, and add v_t,u_t to update v_t+1,u_t+1. Whats the reason for that? https://github.com/Megvii-BaseDetection/OTA/blob/2c85b4d0f9031396854aae969330dde2ab5eacbd/playground/detection/coco/ota.x101.dcnv2.fpn.coco.800size.1x/fcos.py#L693-L699

I met a unexpected error ,shown as : 2022-08-31 17:30:53 | ERROR | cvpods.engine.launch:64 - An error has been caught in function 'launch', process 'MainProcess' (1731), thread 'MainThread' (140323764021056): Traceback (most recent call last):

File "/root/cvpods/tools/train_net.py", line 157, in launch( └ <function launch at 0x7f9d54537040>

File "/root/cvpods/cvpods/engine/launch.py", line 64, in launch main_func(*args) │ └ (Namespace(clearml=False, dir='atss/atss.res50.fpn.coco.800size.1x/', dist_url='tcp://127.0.0.1:49152', eval_only=False, mach... └ <function main at 0x7f9d54553b80>

File "/root/cvpods/tools/train_net.py", line 89, in main runner.resume_or_load(resume=args.resume) │ │ │ └ False │ │ └ Namespace(clearml=False, dir='atss/atss.res50.fpn.coco.800size.1x/', dist_url='tcp://127.0.0.1:49152', eval_only=False, machi... │ └ <function DefaultRunner.resume_or_load at 0x7f9d545531f0> └ <cvpods.engine.runner.DefaultRunner object at 0x7f9d545a77c0>

File "/root/cvpods/cvpods/engine/runner.py", line 172, in resume_or_load self.start_iter = (self.checkpointer.resume_or_load( │ │ │ │ └ <function Checkpointer.resume_or_load at 0x7f9e113c78b0> │ │ │ └ <cvpods.checkpoint.checkpoint.DefaultCheckpointer object at 0x7f9cc6676730> │ │ └ <cvpods.engine.runner.DefaultRunner object at 0x7f9d545a77c0> │ └ 0 └ <cvpods.engine.runner.DefaultRunner object at 0x7f9d545a77c0>

File "/root/cvpods/cvpods/checkpoint/checkpoint.py", line 177, in resume_or_load return self.load(path) │ │ └ 'detectron2://ImageNetPretrained/MSRA/R-50.pkl' │ └ <function Checkpointer.load at 0x7f9e113c7670> └ <cvpods.checkpoint.checkpoint.DefaultCheckpointer object at 0x7f9cc6676730>

File "/root/cvpods/cvpods/checkpoint/checkpoint.py", line 112, in load assert megfile.smart_isfile(path), "Checkpoint {} not found!".format(path) │ │ │ └ 'detectron2://ImageNetPretrained/MSRA/R-50.pkl' │ │ └ 'detectron2://ImageNetPretrained/MSRA/R-50.pkl' │ └ <function smart_isfile at 0x7f9e114143a0> └ <module 'megfile' from '/usr/local/lib/python3.8/dist-packages/megfile/init.py'>

File "/usr/local/lib/python3.8/dist-packages/megfile/smart.py", line 101, in smart_isfile return SmartPath(path).is_file(followlinks=followlinks) │ │ └ False │ └ 'detectron2://ImageNetPretrained/MSRA/R-50.pkl' └ <class 'megfile.smart_path.SmartPath'>

File "/usr/local/lib/python3.8/dist-packages/megfile/smart_path.py", line 14, in smart_method return getattr(self.pathlike, name)(*args, **kwargs) │ │ │ │ └ {'followlinks': False} │ │ │ └ () │ │ └ 'is_file' │ └ Detectron2Path('ImageNetPretrained/MSRA/R-50.pkl') └ SmartPath('detectron2://ImageNetPretrained/MSRA/R-50.pkl')

TypeError: is_file() got an unexpected keyword argument 'followlinks'

2022-03-15 15:37:57.573 | INFO | main::157 - Create soft link to /data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x 2022-03-15 15:37:57.607 | ERROR | cvpods.engine.launch:launch:64 - An error has been caught in function 'launch', process 'MainProcess' (15181), thread 'MainThread' (140337650262208): Traceback (most recent call last):

File "/icislab/volume1/wjm/cvpods/tools/train_net.py", line 165, in args=(args, config, build_model), │ │ └ <function build_model at 0x7fa13a3dd290> │ └ ╒═════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════╕ │ │ confi... └ Namespace(clearml=False, dir=None, dist_url='tcp://127.0.0.1:50206', eval_only=False, machine_rank=0, num_gpus=1, num_machine...

File "/icislab/volume1/wjm/cvpods/cvpods/engine/launch.py", line 64, in launch main_func(*args) │ └ (Namespace(clearml=False, dir=None, dist_url='tcp://127.0.0.1:50206', eval_only=False, machine_rank=0, num_gpus=1, num_machin... └ <function main at 0x7fa2e67a73b0>

File "/icislab/volume1/wjm/cvpods/tools/train_net.py", line 82, in main cfg = default_setup(config, args) │ │ └ Namespace(clearml=False, dir=None, dist_url='tcp://127.0.0.1:50206', eval_only=False, machine_rank=0, num_gpus=1, num_machine... │ └ ╒═════════════════╤═════════════════════════════════════════════════════════════════════════════════════════════════╕ │ │ confi... └ <function default_setup at 0x7fa13a3d05f0>

File "/icislab/volume1/wjm/cvpods/cvpods/engine/setup.py", line 131, in default_setup ensure_dir(output_dir) │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x' └ <function ensure_dir at 0x7fa1577545f0>

File "/icislab/volume1/wjm/cvpods/cvpods/utils/file/file_io.py", line 22, in ensure_dir megfile.smart_makedirs(path, exist_ok=True) │ │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x' │ └ <function smart_makedirs at 0x7fa157752c20> └ <module 'megfile' from '/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/init.py'>

File "/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/smart.py", line 443, in smart_makedirs SmartPath(path).makedirs(exist_ok) │ │ └ True │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x' └ <class 'megfile.smart_path.SmartPath'> File "/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/pathlike.py", line 245, in makedirs self.mkdir(exist_ok=exist_ok) │ │ └ True │ └ <function _bind_function..smart_method at 0x7fa157cd3320> └ SmartPath('/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x') File "/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/smart_path.py", line 14, in smart_method return getattr(self.pathlike, name)(*args, **kwargs) │ │ │ │ └ {'exist_ok': True} │ │ │ └ () │ │ └ 'makedirs' │ └ FSPath('/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x') └ SmartPath('/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x') File "/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/pathlike.py", line 245, in makedirs self.mkdir(exist_ok=exist_ok) │ │ └ True │ └ <function fs_makedirs at 0x7fa157c58710> └ FSPath('/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x') File "/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/fs_path.py", line 18, in fs_method return getattr(fs, name)(self.path_without_protocol, *args, **kwargs) │ │ │ │ │ └ {'exist_ok': True} │ │ │ │ └ () │ │ │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x' │ │ └ FSPath('/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x') │ └ 'fs_makedirs' └ <module 'megfile.fs' from '/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/fs.py'> File "/home/weijinmeng/.local/lib/python3.7/site-packages/megfile/fs.py", line 228, in fs_makedirs return os.makedirs(path, exist_ok=exist_ok) │ │ │ └ True │ │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos/fcos.res50.fpn.coco.800size.1x' │ └ <function makedirs at 0x7fa2e7b76ef0> └ <module 'os' from '/icislab/volume1/wjm/Anaconda/envs/open-mmlab/lib/python3.7/os.py'> File "/icislab/volume1/wjm/Anaconda/envs/open-mmlab/lib/python3.7/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) │ │ └ True │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco/fcos' └ <function makedirs at 0x7fa2e7b76ef0> File "/icislab/volume1/wjm/Anaconda/envs/open-mmlab/lib/python3.7/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) │ │ └ True │ └ '/data/Outputs/model_logs/cvpods_playground/detection/coco' └ <function makedirs at 0x7fa2e7b76ef0> File "/icislab/volume1/wjm/Anaconda/envs/open-mmlab/lib/python3.7/os.py", line 213, in makedirs makedirs(head, exist_ok=exist_ok) │ │ └ True │ └ '/data/Outputs/model_logs/cvpods_playground/detection' └ <function makedirs at 0x7fa2e7b76ef0> [Previous line repeated 4 more times] File "/icislab/volume1/wjm/Anaconda/envs/open-mmlab/lib/python3.7/os.py", line 223, in makedirs mkdir(name, mode) │ │ └ 511 │ └ '/data' └

PermissionError: [Errno 13] Permission denied: '/data'

ERROR: Command errored out with exit status 1: command: /home/anaconda3/envs/OTA/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/thomas/cvpods/setup.py'"'"'; file='"'"'/home/thomas/cvpods/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps cwd: /home/thomas/cvpods/ Complete output (82 lines): which: no hipcc in (/home/anaconda3/envs/OTA/bin:/home/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) running develop running egg_info writing cvpods.egg-info/PKG-INFO writing dependency_links to cvpods.egg-info/dependency_links.txt writing requirements to cvpods.egg-info/requires.txt writing top-level names to cvpods.egg-info/top_level.txt reading manifest file 'cvpods.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'cvpods.egg-info/SOURCES.txt' running build_ext building 'cvpods._C' extension Emitting ninja build file /home/thomas/cvpods/build/temp.linux-x86_64-3.7/build.ninja... Compiling objects... Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N) [1/1] /home/anaconda3/envs/OTA/bin/x86_64-conda-linux-gnu-c++ -MMD -MF /home/thomas/cvpods/build/temp.linux-x86_64-3.7/home/thomas/cvpods/cvpods/layers/csrc/vision.o.d -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/anaconda3/envs/OTA/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/anaconda3/envs/OTA/include -fPIC -I/home/thomas/cvpods/cvpods/layers/csrc -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/TH -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/THC -I/home/anaconda3/envs/OTA/include/python3.7m -c -c /home/thomas/cvpods/cvpods/layers/csrc/vision.cpp -o /home/thomas/cvpods/build/temp.linux-x86_64-3.7/home/thomas/cvpods/cvpods/layers/csrc/vision.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14 FAILED: /home/thomas/cvpods/build/temp.linux-x86_64-3.7/home/thomas/cvpods/cvpods/layers/csrc/vision.o /home/anaconda3/envs/OTA/bin/x86_64-conda-linux-gnu-c++ -MMD -MF /home/thomas/cvpods/build/temp.linux-x86_64-3.7/home/thomas/cvpods/cvpods/layers/csrc/vision.o.d -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/anaconda3/envs/OTA/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/anaconda3/envs/OTA/include -fPIC -I/home/thomas/cvpods/cvpods/layers/csrc -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/torch/csrc/api/include -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/TH -I/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/THC -I/home/anaconda3/envs/OTA/include/python3.7m -c -c /home/thomas/cvpods/cvpods/layers/csrc/vision.cpp -o /home/thomas/cvpods/build/temp.linux-x86_64-3.7/home/thomas/cvpods/cvpods/layers/csrc/vision.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=_C -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++14 cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ In file included from /home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/THC/THCGeneral.h:7, from /home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/THC/THC.h:4, from /home/thomas/cvpods/cvpods/layers/csrc/PSROIPool/psroi_pool_cuda.h:4, from /home/thomas/cvpods/cvpods/layers/csrc/vision.cpp:15: /home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/include/c10/cuda/CUDAStream.h:6:10: fatal error: cuda_runtime_api.h: No such file or directory 6 | #include <cuda_runtime_api.h> | ^~~~~~~~~~~~~~~~~~~~ compilation terminated. ninja: build stopped: subcommand failed. Traceback (most recent call last): File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1400, in _run_ninja_build check=True) File "/home/anaconda3/envs/OTA/lib/python3.7/subprocess.py", line 512, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/thomas/cvpods/setup.py", line 192, in <module>
    "tools/pods_debug",
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/setuptools/command/develop.py", line 34, in run
    self.install_for_development()
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/setuptools/command/develop.py", line 114, in install_for_development
    self.run_command('build_ext')
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 79, in run
    _build_ext.run(self)
  File "/root/.local/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 580, in build_extensions
    build_ext.build_extensions(self)
  File "/root/.local/lib/python3.7/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/setuptools/command/build_ext.py", line 202, in build_extension
    _build_ext.build_extension(self, ext)
  File "/home/anaconda3/envs/OTA/lib/python3.7/distutils/command/build_ext.py", line 534, in build_extension
    depends=ext.depends)
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 423, in unix_wrap_ninja_compile
    with_cuda=with_cuda)
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
    error_prefix='Error compiling objects for extension')
  File "/home/anaconda3/envs/OTA/lib/python3.7/site-packages/torch/utils/cpp_extension.py", line 1413, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
----------------------------------------

ERROR: Command errored out with exit status 1: /home/anaconda3/envs/OTA/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/thomas/cvpods/setup.py'"'"'; file='"'"'/home/thomas/cvpods/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output. @zengarden 麻烦问下这是什么原因啊，折磨了好久了

Good code! I am revising the code for sing gpu. and ask the of 3 lines of code in losses function(fcos.py).

if self.norm_sync:
   dist.all_reduce(num_foreground)
   num_foreground /= dist.get_world_size()

What's meaning of the 3 lines of code? Can I delete these? Thanks.