Thanks for the good work!

Just wanted to mention that I have tried the two currently most stared EfficientDet PyTorch repos on Github, and neither reproduce the paper results on COCO, not even close.

They mostly claim they train on custom data and port weights from official TF checkpoints, but fail to train from scratch on COCO.

I tried their implementations with 200+ kinds of hyper-parameter tuning sets & settings - yes 200+ jobs!

Very keen to see your completed training on COCO. Looking forward to that.

Cheers

Thank you for sharing this code with us.

Can I train on my own dataset with only 11 classes? I am overriding config.num_classes with 11 and overriding config.image_size with 512 but I get very bad results, it is almost like the model is not even aware of the image..

I made sure to put in the boxes as yxyx in the dataset and also made sure the classes start from 1 as I think it is needed due to how fast_collate is working.

Would appreciate your kind help, thank you.

I am curious to know your view on efficientnetLiteB0 as a backbone. Google has officially not released but do you think switching backbone from efficientnetB0 to efficientnetLiteB0 will make coco AP degrade less than 5% of original or it will degrade more?

In the current code base, if I call efficientnetLiteB0 from your timm model collection, will it work?

Hi @rwightman

Thank you so much for making this repo.

I am currently experiencing an issue calling batch_label_anchors when ground truth bounding box list only has 1 bbox. Not sure what might caused this issue, I am wondering if you can take a look. Thanks in advance :)

Issue: anchor_labeler.batch_label_anchors () has index out of range error. Trace attached

Setup anchor:

model_config = get_efficientdet_config('tf_efficientdet_d0')
model = EfficientDet(model_config,pretrained_backbone=True)
model_config.num_classes = 1
model_config.image_size = 512

anchors = Anchors(
    model_config.min_level,model_config.max_level,
    model_config.num_scales, model_config.aspect_ratios,
    model_config.anchor_scale, model_config.image_size
    )


anchor_labeler = AnchorLabeler(anchors,model_config.num_classes,match_threshold=0.5)

Reproduce:

tb = torch.tensor([[468.,353.,52.,386.5]])
tb = tb.int().float()
tlbl = torch.tensor([1.])
cls_targets, box_targets,num_positives = anchor_labeler.batch_label_anchors(1,[tb],[tlbl])

Trace:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-45-e8bceaf11fb2> in <module>
----> 1 cls_targets, box_targets,num_positives = anchor_labeler.batch_label_anchors(1,[tb],[tlbl])

/opt/conda/envs/fastai2/lib/python3.7/site-packages/effdet/anchors.py in batch_label_anchors(self, batch_size, gt_boxes, gt_classes)
    394             # cls_weights, box_weights are not used
    395             cls_targets, _, box_targets, _, matches = self.target_assigner.assign(
--> 396                 anchor_box_list, BoxList(gt_boxes[i]), gt_classes[i])
    397 
    398             # class labels start from 1 and the background class = -1

/opt/conda/envs/fastai2/lib/python3.7/site-packages/effdet/object_detection/target_assigner.py in assign(self, anchors, groundtruth_boxes, groundtruth_labels, groundtruth_weights)
    144         match_quality_matrix = self._similarity_calc.compare(groundtruth_boxes, anchors)
    145         match = self._matcher.match(match_quality_matrix)
--> 146         reg_targets = self._create_regression_targets(anchors, groundtruth_boxes, match)
    147         cls_targets = self._create_classification_targets(groundtruth_labels, match)
    148         reg_weights = self._create_regression_weights(match, groundtruth_weights)

/opt/conda/envs/fastai2/lib/python3.7/site-packages/effdet/object_detection/target_assigner.py in _create_regression_targets(self, anchors, groundtruth_boxes, match)
    167         zero_box = torch.zeros(4, device=device)
    168         matched_gt_boxes = match.gather_based_on_match(
--> 169             groundtruth_boxes.boxes(), unmatched_value=zero_box, ignored_value=zero_box)
    170         matched_gt_boxlist = box_list.BoxList(matched_gt_boxes)
    171         if groundtruth_boxes.has_field(self._keypoints_field_name):

/opt/conda/envs/fastai2/lib/python3.7/site-packages/effdet/object_detection/matcher.py in gather_based_on_match(self, input_tensor, unmatched_value, ignored_value)
    171         input_tensor = torch.cat([ss, input_tensor], dim=0)
    172         gather_indices = torch.clamp(self.match_results + 2, min=0)
--> 173         gathered_tensor = torch.index_select(input_tensor, 0, gather_indices)
    174         return gathered_tensor

IndexError: index out of range in self

Here are some values:
ipdb> p input_tensor
tensor([[  0.,   0.,   0.,   0.],
        [  0.,   0.,   0.,   0.],
        [468., 353.,  52., 386.]])

ipdb> p gather_indices.shape
torch.Size([49104])

ipdb> p gather_indices
tensor([    1,     1,     1,  ...,     1,     1, 24554])

ipdb> p self.match_results
tensor([   -1,    -1,    -1,  ...,    -1,    -1, 24552])

I am trying to get this model (trained on custom dataset) to run under opencv so it will perform better on jetson nano. However, conversion to ONNX does not go well. this is the code i am using:

from effdet import get_efficientdet_config, EfficientDet                                                                                                                 
from effdet.bench import DetBenchPredict                                                                                                                                 
from effdet.efficientdet import HeadNet     

def load_net(checkpoint_path):                                                                                                                                           
    config = get_efficientdet_config('tf_efficientdet_d1')                                                                                                               
    net = EfficientDet(config, pretrained_backbone=False)                                                                                                                
                                                                                                                                                                         
    config.num_classes = 1                                                                                                                                               
    config.image_size=512                                                                                                                                                
    net.class_net = HeadNet(config, num_outputs=config.num_classes, norm_kwargs=dict(eps=.001, momentum=.01))                                                            
                                                                                                                                                                         
    checkpoint = torch.load(checkpoint_path)                                                                                                                             
    net.load_state_dict(checkpoint['model_state_dict'])                                                                                                                  
    net = DetBenchPredict(net, config)                                                                                                                                   
    net.eval();                                                                                                                                                          
    return net.cuda()  

import torch
from functools import partial

def load_model_weight(model, model_path):
    model=torch.load(model_path)
    model = model.eval()
    return model

def export_onnx_model(model, input_shape, onnx_path, input_names=None, output_names=None, dynamic_axes=None):
    inputs = torch.ones(*input_shape)
    origin_forward = model.forward
    model.forward = partial(
        model.forward, img_scales=torch.tensor([1.0], dtype=torch.float).cuda(), img_size=torch.tensor([input_shape[-2:]], dtype=torch.float).cuda())
    #model(inputs)
    torch.onnx.export(model, (inputs), onnx_path, input_names=input_names, output_names=output_names, dynamic_axes=dynamic_axes)

model_path = "best-checkpoint-013epoch.bin"
model = load_net(model_path)
input_shape = (1, 3, 512, 512)
onnx_path = "test.onnx"
export_onnx_model(model, input_shape, onnx_path)

but it gives:

     17 # Calculate asymmetric TensorFlow-like 'SAME' padding for a convolution
     18 def get_same_padding(x: int, k: int, s: int, d: int):
---> 19     return max((math.ceil(x / s) - 1) * s + (k - 1) * d + 1 - x, 0)
     20 
     21 

RuntimeError: Integer division of tensors using div or / is no longer supported, and in a future release div will perform true division as in Python 3. Use true_divide or floor_divide (// in Python) instead.

Hey @rwightman , I am trying to train EfficientDet D0 model on COCO from scratch, it works perfectly and converges when I use your settings: ./distributed_train.sh 4 /mscoco --model efficientdet_d0 -b 22 --amp --lr .12 --sync-bn --opt fusedmomentum --warmup-epochs 5 --lr-noise 0.4 0.9 --model-ema --model-ema-decay 0.9999

But when I use a Larger Batch size setting like the following, loss becomes nan:

1. Number of GPU=8 and Per GPU batch size=30 and lr=0.32 (Single Node)
2. Number of GPU=32 and Per GPU batch size=30 and lr=1.39 (Multi Node) Both FP32 and AMP
3. Number of GPU=32 and Per GPU batch size=30 and lr=0.86 (Multi Node) LR I chose midway between linear scaling and square-root scaling. Both FP32 and AMP.

In case of AMP there is a cascade of loss scaling and then the loss becomes nan, And this does not necessarily happen in the first few epochs.

Strangely enough the TF1 Google automl code base has no problem scaling linearly. Do you know what might be the problem here? Do I need to change the recipe when using a large batch size?

Hi @rwightman - super excited that you have made this! Can you describe best method for training on custom datasets? (I can hack it up but prefer if you have have advice up front). Thanks!

torch.exp(neg_logits) will give a NaN for a large negative value of neg_logits. Prevent this by replacing torch.log1p(torch.exp(neg_logits)) with torch.logaddexp(neg_logits, 0).

logaddexp(x, y) is log(exp(x) + exp(y)), so logaddexp(x, 0) is log(exp(x) + 1). This is equivalent to log1p(exp(x)) which is just log(exp(x) + 1).

The implementation of logaddexp does not return a NaN if the operands are very large. It is equivalent to max(x, y) + log(1 + exp(-1 * abs(x - y))). This forces the exp to be called with a large negative value instead of a large positive value.

I'm not clear on how this changes the back-propagation behaviour. Also, I think logaddexp was only available from pytorch 1.6.0.

Thanks for your good work.

https://github.com/google/automl/tree/master/efficientdet

They now report 34.3 val mAP for EffDet-D0. Not sure what exactly they did, but seems like they made some changes in their model definition etc. It'd be great if you could have a look at their changes, and potentially sync-up your repo so it reproduces those numbers.

Thanks a million :)

Have you tried ONNX export for tf_efficientdet_lite0? I tried with exporting only backbone+BiFPN by defining forward_dummy similar to used in mmdetect. I got very complicated ONNX graph (Scatter/Gather/Div/Squeeze etc..) Whereas efficientNetLite models you exported using gen-efficientnet-pytorch/blob/master/onnx_export.py are very clean.

Is there a way to get clean ONNX graph for tf_efficientdet_lite0?

Thanks a lot for all your help.

If I am training on a custom dataset and preparing a dataloader on my own should I input the box in the format [yxyx] or [xyxy]. (I see in the ConfigParser the bbox_yxyx: bool = True) Right now, I input in the form of [yxyx].

For the detections, like it is stated in the READme that the GPU NMS Pytorch supports XYXY. I am a bit confused if I will have to input the box in [xyxy] or [yxyx].

Also, in the DetbenchTrain and Predict I don't see any flag converting into [XYXY] by default. However, in the evaluator PascalEvaluator if pred_yxyx is False then img_dets[:, [1,0,3,2]] is stated . This means it supports img_pred_xyxy as True which indiretly puts the detected boxes in [xyxy]. Eventually, the iou(boxes1, boxes2) need bboxes in [yxyx]. So, I cannot debug the conversion from [xyxy] to [yxyx] here.

So I am a bit confused if how the output['detections'][:4] i.e, the generate_detections from anchors.py gives which output format?

Since the decode_box_output function in the anchors.py takes [yxyx] as an input I had input the input target bbox as [yxyx]. Did I do it right?

TIA

Describe the bug Unable to save a model after successfully torch-scripting it.

To Reproduce

The following snippet:

from effdet import EfficientDet
from effdet.config.model_config import get_efficientdet_config
import torch

architecture = "tf_efficientdet_d1"
config = get_efficientdet_config(architecture)
model = EfficientDet(config, pretrained_backbone=True)
model.toggle_head_bn_level_first()
ts_model = torch.jit.script(model)
torch.jit.save(ts_model, "model.pt")

produces the following output:

(venv) maxime@Morphling: python failing_effdet.py 
Traceback (most recent call last):
  File "failing_effdet.py", line 10, in <module>
    torch.jit.save(ts_model, "model.pt")
  File "/home/maxime/mambaforge/envs/venv/lib/python3.8/site-packages/torch/jit/_serialization.py", line 81, in save
    m.save(f, _extra_files=_extra_files)
  File "/home/maxime/mambaforge/envs/venv/lib/python3.8/site-packages/torch/jit/_script.py", line 714, in save
    return self._c.save(str(f), **kwargs)
RuntimeError: 
Could not export Python function call '_forward'. Remove calls to Python functions before export. Did you forget to add @script or @script_method annotation? If this is a nn.ModuleList, add it to __constants__:
  File "/home/maxime/mambaforge/envs/venv/lib/python3.8/site-packages/effdet/efficientdet.py", line 440
            return self._forward_level_first(x)
        else:
            return self._forward(x)
                   ~~~~~~~~~~~~~ <--- HERE

Expected behavior Using model.toggle_head_bn_level_first() should have made torchscript happy, because the _forward method, which depends on it, is illegal for torchscript. However, it seems that the function is still checked before saving.

Scripting + saving works fine with torchvision and pytorch lightning models.

Desktop:

• OS: Ubuntu using WSL2 with VSCode
• python==3.8
• effdet==0.3.0
• torch==1.12.0
• CPU only (not attempted on GPU yet)

Found this commit, but did not check it. Looks like as bug fix: https://github.com/ceshine/efficientdet-pytorch/commit/b8ccb5247533645db523d17a8dec4ae9ad612fe9

Hi! Thanks for your repo, it's very useful. But i met with some problem when tried to create model from pretrained checkpoint. I'm using effdet==0.3.0 package. When i run the folowing code: model = create_model("tf_efficientdet_d5", bench_task="train", num_classes=3, image_size=(512, 512), bench_labeler=True, pretrained=False, checkpoint_path="tf_efficientdet_d5_51-c79f9be6.pth" ) I get the following error: 'size mismatch for class_net.predict.conv_pw.weight' on line 'load_checkpoint(model, checkpoint_path, use_ema=checkpoint_ema)' of the file 'effdet/factory.py"'

After that in the file 'effdet/factory.py' i placed code : ' if checkpoint_path: load_checkpoint(model, checkpoint_path, use_ema=checkpoint_ema)' before code: if num_classes is not None and num_classes != config.num_classes: model.reset_head(num_classes=num_classes) and error is gone. Did you met with such problem?

As per your given TODOs in the README, I thought it would be nice to add initial mosaic augmentation to this codebase. Original implementation is from YoloV5 repo, which I have slightly tweaked to fit here.

I currently do not have the hardware to train with this implementation, but I still would like to contribute. Thats how some of them look visualized from the train script in img, target = create_dataset(args) .

Sometimes the bboxes look off as well. If you give me any pointers to improve the implementation, or to test (on limited hardware) whether this augmentation works here, please let me know. I am willing to make changes if needed, as I am just starting in this deep learning world.

It does add an opencv dependency as per the original repo, so dont know whether that is a problem. Thanks!

Hi Ross, thanks for sharing this repo! Curious if you have any plans on releasing a model trained on OpenImages?

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Have access to a pretrained model trained on OpenImages

Describe alternatives you've considered The official repo (see here), but their model is outdated and it's also in tensorflow, which is inconvenient

Additional context N/A

Hi! I'm trying to train tf_efficientdet_d3 with your code on custom coco-like data. I'm training on Google Colab. !python3 train.py '/content/drive/MyDrive/dataset' --model tf_efficientdet_d3 --num-classes 60 --pretrained -b 1 --save-images --log-interval 100 --epochs 10 --output '/content/drive/MyDrive/Colab Notebooks/PyTorch training/' After the first epoch, I faced with this type of error:

Traceback (most recent call last):
  File "train.py", line 656, in <module>
    main()
  File "train.py", line 435, in main
    best_metric, best_epoch = saver.save_checkpoint(epoch=epoch, metric=eval_metrics[eval_metric])
  File "/usr/local/lib/python3.7/dist-packages/timm-0.4.5-py3.7.egg/timm/utils/checkpoint_saver.py", line 78, in save_checkpoint
    os.link(last_save_path, save_path)
OSError: [Errno 38] Function not implemented: '/content/drive/MyDrive/Colab Notebooks/PyTorch training/train/20210311-161812-tf_efficientdet_d3/last.pth.tar' -> '/content/drive/MyDrive/Colab Notebooks/PyTorch training/train/20210311-161812-tf_efficientdet_d3/checkpoint-0.pth.tar'

Something went wrong during the process of saving the checkpoint. I'll be grateful to recieve the solution of this problem or some tips that may help me to solve it. Thanks!