Hi, Thanks for your work, i have an question about extracting features. Could you please tell me how to extract features by using Omnivore on epic100.

I note a forward_seg function in the class BasicLayer() of swin_transformer_3d.

What does this function work for? Should I switch the forward function to the forward_seg function when I fine-tune for the semantic segmentation task?

Thanks!

have been getting urllib.error.HTTPError: HTTP Error 403: Forbidden

using

model = torch.hub.load("facebookresearch/omnivore:main", model=model_name)

I tried running the inference notebook. It breaks at model = torch.hub.load("facebookresearch/omnivore:main", model=model_name) complaining:

FileNotFoundError: [Errno 2] No such file or directory: '/root/.cache/torch/hub/facebookresearch_omnivore_main/hubconf.py'

I am using Google Colab and the torch version is: 1.11.0+cu113. I have tried with the following too but it doesn't help:

model_name = "omnivore_swinB"
model = torch.hub.load("facebookresearch/omnivore", model=model_name)

@imisra @mannatsingh

Dear author, I use the given pretrain model on the epic-kitchen dataset to inference on the validation dataset, however, I get a very poor result: {"action_top1_acc": "35.19", "action_top5_acc": "57.09"}. Could you please tell me where I have a mistake?

This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

This also means we can make a web page where other people can try out your model! View it here: https://replicate.com/facebookresearch/omnivore. We enable selecting different models for inference, and you can find the docker file under the tab ‘run model with docker’.

We have added some examples to the page, but do claim the page so you can own the page, customise the Example gallery as you like, push any future update to the web demo, and we'll feature it on our website and tweet about it too.

In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

Hi

I'm curious whether the SUN RGB-D data with disparity can be released or the code to covert the depth to disparity for the SUN RGB-D can be released.

Thanks

Hello,

Thanks for the amazing work. I am curious about the performance differences between Tab 3 and Tab 7 on NYUv2 segmentation.

Can you confirm that the performance difference is due to the different datasets used in pre-training? In Tab3, is Omnivore Swin-B pre-trained on IN1K ? In Tab 7, is Omnivore Swin-B pre-trained on IN21K, IN1K, K400, and SUN ?

Thanks

Hi,

Thanks for the really cool work, and for sharing the repository! I was wondering if you can provide more details on obtaining the 19 scene classification labels for SUN RGB-D dataset. When I downloaded directly from https://rgbd.cs.princeton.edu/ (SUN RGBD V1), and I looked in the scene.txt file for each image, it seemed like there were more than 19 scene labels (I saw 44 different scene labels).

Thanks!

Hello author, I am very interested in your code! I would like to ask if it is possible to publish the code of the loss function for the training period, and I would like to know if the labels of the 3 datasets are constrained separately or together. Or is it computing 3 classification tasks or one classification task.

I got an error said: ERROR: Could not find a version that satisfies the requirement pytorchvideo (from versions: none) ERROR: No matching distribution found for pytorchvideo

Hello Omnivore!

Thank you very much for everything you have done and provided through this repository.

I'm interested in building-upon your work for egocentric action recognition, and my first step was to replicate the action recognition results quoted in your paper for the EK100 dataset. If I understand correctly, using the provided omnivore_swimB_epic checkpoint, I should be able to obtain an action recognition accuracy close to 50% on the validation subset of the data. However, trying, I only got an accuracy of 35.91% (for top1 action).

This is a similar problem as the one listed in #20 . However, that issue was closed without a clear resolution. I hope you can help me find what I might be missing. Here are more details about what I am doing:

Loading the data

I adapted the epic kitchens data loader found here to do the following:

For each action listed in the Epic-Kitchens validation csv:

• find the center frame
• load 32 sequential frames (separated with a stride of 2), starting at the center frame. The resulting shape is [1, 3, 256, 456]
• divide all loaded frame values by 255
• subtract the omnivore `mean = [0.485, 0.456, 0.406] from each channel
• divide by the omnivore std = [0.229, 0.224, 0.225]
• crop the center [256x256] pixels in the image to yield the shape [1,3,256,256]

I'm tracking my work in this repository: https://github.com/iranroman/ego_actrecog_analysis

Any hints will be greatly appreciated. Thank you very much in advance!

Hi. Thank you for your amazing code. May I ask how to use the omnimae model to extract the features of the videos instead of output the keywords. Thank you very much.

Hi.

• fixed ModuleNotFoundError while running inference_tutorial.ipynb notebook
• removed unnecessary imports os, torch.nn.functional, matplotlib.image;
• fixed referenced to CONTRIBUTING, LICENCE, NOTICE, CODE_OF_CONDUCT inside {omnivore,omnimae}/README.md

P.S: during this import from torchvision.transforms._transforms_video import NormalizeVideo a UserWarning raised: so probably it's better to fix this import for future.

P.S.S: tested in colab environment, "Run all cells" without any errors.

Hi.

Thank you very much for your excellent work and for sharing the repository!

I was wondering if you could provide more details on the hyperparameter of fine-tuning on SUN RGBD and Kinetics400 (in Table 2).

I think I would use the ImageNet-1k pre-trained model (ImageSwin) and fine-tune the parameters according to Supplement A., right? Also, is the performance of the Omnivore model in Table 2 without using a pre-trained model?