Pytorch implementation of Feature Pyramid Network (FPN) for Object Detection

I am training my own dataset using fpn, i set the num_workers > 0 like =4, and then it collapses after the first training iter, but it will be ok if num_workers = 0, my device is following： OS: Ubuntu 16.04 PyTorch version: pytorch 0.3.1 Python version: python 2.7 CUDA version: 8.0 GPU models : Tesla P40

[session 3][epoch  1][iter    0] loss: 4.3830, lr: 1.00e-03
			fg/bg=(115/397), time cost: 5.939724
			rpn_cls: 0.6945, rpn_box: 1.2795, rcnn_cls: 2.3964, rcnn_box 0.0127
Traceback (most recent call last):
  File "trainval_net.py", line 339, in <module>
    data = data_iter.next()
  File "/root/anaconda2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 204, in __next__
    idx, batch = self.data_queue.get()
  File "/root/anaconda2/lib/python2.7/multiprocessing/queues.py", line 378, in get
    return recv()
  File "/root/anaconda2/lib/python2.7/site-packages/torch/multiprocessing/queue.py", line 22, in recv
    return pickle.loads(buf)
  File "/root/anaconda2/lib/python2.7/pickle.py", line 1388, in loads
    return Unpickler(file).load()
  File "/root/anaconda2/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/root/anaconda2/lib/python2.7/pickle.py", line 1139, in load_reduce
    value = func(*args)
  File "/root/anaconda2/lib/python2.7/site-packages/torch/multiprocessing/reductions.py", line 68, in rebuild_storage_fd
    fd = multiprocessing.reduction.rebuild_handle(df)
  File "/root/anaconda2/lib/python2.7/multiprocessing/reduction.py", line 155, in rebuild_handle
    conn = Client(address, authkey=current_process().authkey)
  File "/root/anaconda2/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/root/anaconda2/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
    s.connect(address)
  File "/root/anaconda2/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 111] Connection refused

i saw the same problem https://github.com/pytorch/pytorch/issues/1355, but his python verison is 3.x, and i tried that solution, it does not works. @jwyang

When running make.sh, I'm getting a no such file or directory error for the file nms_cuda.c. This appears to be missing from the lib/model/nms/src directory.

running build_ext
building '_nms' extension
creating home
creating home/thomasbalestri
creating home/thomasbalestri/PycharmProjects
creating home/thomasbalestri/PycharmProjects/fpn.pytorch
creating home/thomasbalestri/PycharmProjects/fpn.pytorch/lib
creating home/thomasbalestri/PycharmProjects/fpn.pytorch/lib/model
creating home/thomasbalestri/PycharmProjects/fpn.pytorch/lib/model/nms
creating home/thomasbalestri/PycharmProjects/fpn.pytorch/lib/model/nms/src
gcc -pthread -B /home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/include/python2.7 -c _nms.c -o ./_nms.o
gcc -pthread -B /home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/compiler_compat -Wl,--sysroot=/ -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DWITH_CUDA -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/TH -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/../../lib/include/THC -I/usr/local/cuda/include -I/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/include/python2.7 -c /home/thomasbalestri/PycharmProjects/fpn.pytorch/lib/model/nms/src/nms_cuda.c -o ./home/thomasbalestri/PycharmProjects/fpn.pytorch/lib/model/nms/src/nms_cuda.o
gcc: error: /home/thomasbalestri/PycharmProjects/fpn.pytorch/lib/model/nms/src/nms_cuda.c: No such file or directory
Traceback (most recent call last):
  File "build.py", line 36, in <module>
    ffi.build()
  File "/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 167, in build
    _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
  File "/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 103, in _build_extension
    ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
  File "/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/cffi/api.py", line 690, in compile
    compiler_verbose=verbose, debug=debug, **kwds)
  File "/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/cffi/recompiler.py", line 1513, in recompile
    compiler_verbose, debug)
  File "/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/cffi/ffiplatform.py", line 22, in compile
    outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
  File "/home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/lib/python2.7/site-packages/cffi/ffiplatform.py", line 58, in _build
    raise VerificationError('%s: %s' % (e.__class__.__name__, e))
cffi.error.VerificationError: CompileError: command 'gcc' failed with exit status 1

I'm experiencing a CUDA error when running the scripts. There errors occurs here. in lib/model/nms/src/nms_cuda_kernel.cu. The error is: CUDA Error: invalid argument, at line 147 CUDA Error: invalid argument, at line 154 which I assuming is referring to the line in the nms cuda kernel.

When executing nms in this line, of lib/model/rpn/proposal_layer_fpn.py nms(torch.cat((proposals_single, scores_single), 1), nms_thresh).

as a check, I have ensured in the make.sh file I changed compute to my gtx1080. I'm using pytorch 0.4 with python 2.7.

I haven't worked with cuda files before so I'm not too sure how to solve this.

The default cnn network is res101. However, on the fpn.py line 184, the size of p2, p3, p4, p5, p6 are 1/4,1/8,1/16,1/16,1/32 of the input image's size, so the FPN_FEAT_STRIDES should be [4,8,16,16,32]. But on the trainval_net.py line 173, default FPN_FEAT_STRIDES is [4, 8, 16, 32, 64], the config and code don't match. I not sure whether the result will be affected, but i think you can modify it. @jwyang

    if cfg.POOLING_MODE == "crop":

        grid_xy = _affine_grid_gen(rois, base_feat.size()[2:], self.grid_size)
        grid_yx = torch.stack([grid_xy.data[:,:,:,1], grid_xy.data[:,:,:,0]], 3).contiguous()
        roi_pool_feat = self.RCNN_roi_crop(base_feat, Variable(grid_yx).detach())
        if cfg.CROP_RESIZE_WITH_MAX_POOL:
            roi_pool_feat = F.max_pool2d(roi_pool_feat, 2, 2)
    else :
        roi_pool_feats = []
        box_to_levels = []
        for i, l in enumerate(range(2, 6)):
            if (roi_level == l).sum() == 0:
                continue
            idx_l = (roi_level == l).nonzero().squeeze()
            box_to_levels.append(idx_l)
            #scale = feat_maps[i].size(2) / im_info[0][0]
            if cfg.POOLING_MODE == 'align':
                feat = self.RCNN_roi_align(feat_maps[i], rois[idx_l])
            elif cfg.POOLING_MODE == 'pool':
                feat = self.RCNN_roi_pool(feat_maps[i], rois[idx_l])
            roi_pool_feats.append(feat)
 
     
        roi_pool_feat = torch.cat(roi_pool_feats, 0)       
        box_to_level = torch.cat(box_to_levels, 0)
        idx_sorted, order = torch.sort(box_to_level)
        roi_pool_feat = roi_pool_feat[order]
        
    return roi_pool_feat

in _PyramidRoI_Feat function of fpn.py when i porting test_net code to change torch version 0.4 to 1,
occured "runtime error : zero-dimensional tensor(at position 3) cannot be concatenated " because 724 box_to_levle[3]'s dimesion is 1 error data shape : tensor(271,device='cuda:0') other data shape : tensor([3, 10, 13, 14, .... 282],device='cuda:0')

i change some code

1. test_net.py ( line : 277 ) before : cls_dets = torch.cat(cls_boxes, cls_scores, 1) after : cls_dets = torch.cat((cls_boxes, cls_scores.unsqueeze(1)), 1)
2. test_net.py ( line : 279 ) before : keep = nms(cls_dets, cfg.TEST.NMS) after : keep = nms(cls_boxes[order, :], cls_scores[order], cfg.TEST.NMS)

my enviorment : based project is your faster - rcnn , which support version torch 1.0 anaconda pytorch1.0, cuda version is 10 python version is 2.7 dataset is voc2007 ( occured error in 001433 data, test data 724th of vocdata2007)

i wondering where i shold debug. i should not get a sense.

/home/zhuyuhe/.conda/envs/dwzpy/lib/python3.5/site-packages/torch/nn/functional.py:1749: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)) /home/zhuyuhe/duwangzhe/fpn_pytorch/lib/model/rpn/rpn_fpn.py:79: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. rpn_cls_prob_reshape = F.softmax(rpn_cls_score_reshape) Traceback (most recent call last): File "trainval_net.py", line 332, in roi_labels = FPN(im_data, im_info, gt_boxes, num_boxes) File "/home/zhuyuhe/.conda/envs/dwzpy/lib/python3.5/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, **kwargs) File "/home/zhuyuhe/duwangzhe/fpn_pytorch/lib/model/fpn/fpn.py", line 246, in forward roi_pool_feat = self._PyramidRoI_Feat(mrcnn_feature_maps, rois, im_info) File "/home/zhuyuhe/duwangzhe/fpn_pytorch/lib/model/fpn/fpn.py", line 160, in _PyramidRoI_Feat feat = self.RCNN_roi_pool(feat_maps[i], rois[idx_l], scale) File "/home/zhuyuhe/.conda/envs/dwzpy/lib/python3.5/site-packages/torch/nn/modules/module.py", line 491, in call result = self.forward(*input, **kwargs) File "/home/zhuyuhe/duwangzhe/fpn_pytorch/lib/model/roi_pooling/modules/roi_pool.py", line 14, in forward return RoIPoolFunction(self.pooled_height, self.pooled_width, scale)(features, rois) File "/home/zhuyuhe/duwangzhe/fpn_pytorch/lib/model/roi_pooling/functions/roi_pool.py", line 26, in forward features, rois, output, ctx.argmax) File "/home/zhuyuhe/.conda/envs/dwzpy/lib/python3.5/site-packages/torch/utils/ffi/init.py", line 197, in safe_call result = torch._C._safe_call(*args, **kwargs) TypeError: float() not supported on cdata 'void *' Anyone can help me? Thx!

When there is a problem with demo.py, we first use the demo.py given by the author @jwyang and report an error import error: no module named fast_ rcnn.faster_ rcnn_ cascade。 It was found that the demo of the project was not updated for FPN, so it was replaced by the demo.py provided by @Zongxianlee. My environment is Python 2.7.18, CUDA 9.0.176, torch 0.4.0 torch vision 0.2.1 cudatoolkit 9.0. It is confirmed that the CUDA version is consistent with the CUDA version of PyTorch, but running the updated demo.py still reports an error: undefined symbol:__ cudaPopCallConfiguration。Any idea about the problem? Thanks！

demo.py出现问题，最开始使用作者@jwyang给出的demo.py，报错ImportError: No module named faster_rcnn.faster_rcnn_cascade。发现该项目的demo并没有针对FPN更新，于是换成了@ZongxianLee 提供的demo.py. 我的环境是：python 2.7.18 、cuda 9.0.176 、torch 0.4.0 torchvision 0.2.1 cudatoolkit 9.0. 确定了cuda的版本和pytorch 的cuda版本是一致的，但运行更新后的demo.py依旧报错：undefined symbol: __cudaPopCallConfiguration。有什么建议吗？谢谢！

进入代码目录 到lib，首先要编译c代码 sh make.sh,但是我得到了以下错误 fpn.pytorch/lib/model/roi_crop/src/roi_crop.c:10:30: error: dereferencing pointer to incomplete type int batchsize = inputImages->size[0]; ^ /fpn.pytorch/lib/model/roi_crop/src/roi_crop.c:11:39: error: dereferencing pointer to incomplete type int inputImages_height = inputImages->size[1];

I have run this code, based on resnet101, the best epoch result is 70.1, which is much less compared to faster-rcnn ---74.X, I want to know why?Can you tell me?Thank you!

grid_xy = _affine_grid_gen(rois, base_feat.size()[2:], self.grid_size)

NameError: global name 'base_feat' is not defined how to solve this problem？Is there someone meet this problem？Hope your request，thanks！