Hi, Thanks for your sharing this nice work! I would like to confirm the following question:

• The image encoder in CLIP is not used here, instead by a zeroshot classifier?
• The text embeddings are precomputed by CLIP text encoder saved as metadata/lvis_v1_clip_a+cname.npy for LVIS val set, so text encoder can be regarded as fixed in training?
• In zeroshot classifier forward process, what x and classifier represent for?

Hello, I am trying to prepare the data for zero-shot COCO.

When I run get_cc_tags.py, the code breaks because 'categories' is not in cc_data, here: https://github.com/facebookresearch/Detic/blob/main/tools/get_cc_tags.py#L135

If I try to use --cat_path, the code again breaks because 'categories' is not in cc_data, here: https://github.com/facebookresearch/Detic/blob/main/tools/get_cc_tags.py#L128

Could you help me out with this?

Thanks so much!

Hi @xingyizhou

Thank you for sharing the great work.

I am trying to reproduce the Detic results on LVIS. Firstly Box Supervised ViLD baseline model has been successfully trained.

Now using its weights, I am trying to run Detic configuration using Detic_ViLD_200e.py config file, but after running the lazy_train_net.py, it prints initial logs and get stuck when iteration 0 begins to start. I am attaching screenshots below, after the last warning, the training does not get started, while the utilization from all GPUs instantly reaches 100% and get stuck on 100% Even after quitting the python run, the memory usage still get stuck to 100%, I have to manually kill the python process in-order to reset the GPU utilization.

Can you please help me in running the Detic training for LVIS. I am using the same environment, workstation etc on which the Box Supervised ViLD baseline is trained.

Waiting for your kind response. Thank you.

I am currently trying to use this software for realtime domain like robotics, where faster inference is often preferred to high accuracy. https://github.com/HiroIshida/detic_ros

So it would be great if someone share tips or config settings to make get inference faster at the sacrifice of something (accuracy, class number, etc..).

A simple attempt to do that end may changing the input image size. However, the inference time varying input image size is like follows, which means input image size is not the primal factor. 300x300 0.19 100x100 0.17 50x50 0.165 (Using Geforce GTX-1080ti and the model is used as same as the demo described in README)

Finally, thank you very much for releasing such a great software.

Hi,

Thank you for the great work. I am exporting torch script model but during inference I run into shapes error for model corresponding to the config : "Detic_LI21k_CLIP_SwinB_896b32_4x_ft4x_max-size.yaml"

export script example :

# Prepare inputs 

im = cv2.imread('input.jpg')
image = aug.get_transform(im).apply_image(im)
image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))
inputs = [{"image": image}]

## Export to torchscipt
if isinstance(model, GeneralizedRCNN):
    def inference(model, inputs):
        # use do_postprocess=False so it returns ROI mask
        inst = model.inference(inputs, do_postprocess=False)[0]
        return [{"instances": inst}]
else:
    inference = None

traceable_model = TracingAdapter(model, inputs, inference)
traceable_model.eval()
_# 1. option : pass image array as a tuple_
ts_model = torch.jit.trace(traceable_model, (image,))
_# or 2nd option_ 
# ts_model = torch.jit.trace(traceable_model, image, strict=False)
d = {"shape": image.shape}
extra_files = {'exported_config.txt': json.dumps(d)}

with open("models/detic_model_2.ts", "wb") as f:
    torch.jit.save(ts_model, f)

I am curious about how the bbox labels are predicted since the data are split into base and novel. Taking LVIS as example, when calculating the AP_novel, are the labels obtained by taking the maximum of the 1203 scores of all categories or only the maximum of the 337 novel categories?

Hi. Fixed links:

• to 21K model and LVIS API licence in README.md
• to UniDet repository in datasets/REAMDE.md
• to Detic_DeformDETR_R50_2x config in docs/MODEL_ZOO.md

HI, thanks for making Detic available; there's some fantastic functionality in here.

My issue is quite mundane: How can I get the predictor function to output the prediction classes as text strings with an associated confidence level? I'd like to iterate across a large number of images and record the results in a pandas dataframe or similar. Having them overlaid on an image is less useful for this.

Thanks!

Hi, I have some questions about using CLIP embeddings as classification weights

1. I think there is no other code which load CLIP embeddings as classification weights except load model. Aren’t the provided weights trained? (Are CLIP embeddings loaded only?)

2. I think it may work if FC layer of a classification network is used as classification weights of detector. Why are you choose to use CLIP embedding instead of pre-trained classification network?

Thank you.

Hello.

I was able to get box_features from a previous question.

The dimension of these box_features was 1024 as shown in the config file in the picture below.

I'm trying to use this box_features in another pre-trained model, but the dimension doesn't match. This is because the dimension of other pre-trained models is 1280.

I was wondering if I need to re-train the model from scratch to get box_features of 1280 dimension or how to get it.

Or, I was wondering if it is a good way to extend the dimension with Autoencoder or PCA method to convert from 1024 dimension to 1280 dimension.

Thank you for your time to read this question!

I have added a few features to the demo application - one is to enable a screen-shot video source so that it is possible to run on screen as video input. I also added an additional argument for specifying CPU only as I do not have a GPU on my MAC. Is this of interest - should I prepare a PR with this?

Screenshots like this can be run in the demo application:

Hi, in your repo, it seems there are many implementations for other variants of losses but can not find the implementation for DLWL which uses ADMM algorithm for label assignment. It would be grateful if you provide that one since the author of DLWL does not release the code. Thanks

Hello Xingyi Zhou, Thank you for your brilliant work.

As said in datasets/README.md, I follow the instructions from UniDet.

We followed the instructions in UniDet to convert the metadata for OpenImages.

But there are some problems.

1. run $python tools/convert_datasets/convert_oid.py -p datasets/oid/ --subsets train NameError: name 'image_label_sourcefile' is not defined
2. run python tools/convert_datasets/convert_oid.py -p datasets/oid/ --subsets val --expand_label FileNotFoundError: [Errno 2] No such file or directory: 'datasets/oid/annotations/challenge-2019-validation-detection-human-imagelabels_expanded.csv'
3. For pre-processed annotation files url, there is no any annotation files. even the old url in commit history.

Our pre-processed annotation files can be directly downloaded here.

Anyone helps?

similiar issue #https://github.com/xingyizhou/UniDet/issues/2

Hello @facebookresearch, Thank you for your fantastic work. I am a student, and I wonder which is the best confidence score for the inference phase. (both for COCO and Imagenet datasets).

Thank you for your help, this value will benefit further research.

Best, TIn

The new version of Cog improves the Python API, along with several other changes. Particularly pydantic is now used for Predictor and the previous version will be deprecated.

This PR upgrades the Replicate demo and API to Cog version >= 0.1. I have already pushed this to Replicate, so you don't need to do anything for the demo to keep working :) https://replicate.com/facebookresearch/detic

Hi, Xinyi!

I loaded the file "lvis_v1_train_cat_info.json", it seems to contain image_count for rare classes. It may lead to data leak in the open-vocabulary setting when using the fed loss.