command line: python -m ego4d.cli.cli --yes --datasets clips --benchmarks FHO --metadata --output_directory ~/ego4d_data/ when I run this command,I met this problem(showing in figures)

Hi,

When I try to download the dataset by following the instructions, I have encountered this problem:

botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Any idea about how to fix this? Thanks in advance!

Hi,

When I download benchmark EM or FHO using "python -m ego4d.cli.cli --output_directory="I:/Ego4D dataset/ego4d_data" --datasets clips --benchmarks FHO",

I keep getting the error: "datasets: ['clips'] Download Path: I:\Ego4D dataset\ego4d_data\v1

Ego4D Metadata: I:\Ego4D dataset\ego4d_data\ego4d.json Checking requested datasets and versions... Created download directory for version: 'v1' of dataset: 'clips' at: I:\Ego4D dataset\ego4d_data\v1\clips

Filtering by benchmarks: ['fho'] Retrieving object metadata from S3... 100%|█████████████████████████████████████████████████████████████████████████| 1723/1723 [00:02<00:00, 733.31object/s] Checking if latest file versions are already downloaded... 0%| | 1/1723 [01:33<44:57:28, 93.99s/file] WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com WARNING:urllib3.connectionpool:Connection pool is full, discarding connection: ego4d-unict.s3.eu-central-1.amazonaws.com Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connection.py", line 159, in _new_conn conn = connection.create_connection( File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\connection.py", line 84, in create_connection raise err File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\connection.py", line 74, in create_connection sock.connect(sa) TimeoutError: [WinError 10060] 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败。

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\httpsession.py", line 443, in send urllib_response = conn.urlopen( File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 724, in urlopen retries = retries.increment( File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py", line 379, in increment raise six.reraise(type(error), error, _stacktrace) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\packages\six.py", line 735, in reraise raise value File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 670, in urlopen httplib_response = self._make_request( File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 381, in _make_request self._validate_conn(conn) File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py", line 976, in _validate_conn conn.connect() File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connection.py", line 308, in connect conn = self._new_conn() File "C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connection.py", line 171, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <botocore.awsrequest.AWSHTTPSConnection object at 0x000001C2D8468E50>: Failed to establish a new connection: [WinError 10060] 由于连接方在一段时间后没有正确答复或连接的主机没有反应，连接尝试失败。

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\ProgramData\Anaconda3\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\ProgramData\Anaconda3\lib\site-packages\ego4d\cli\cli.py", line 220, in main(config) File "C:\ProgramData\Anaconda3\lib\site-packages\ego4d\cli\cli.py", line 136, in main active_downloads = filter_already_downloaded( File "C:\ProgramData\Anaconda3\lib\site-packages\ego4d\cli\download.py", line 241, in filter_already_downloaded to_download = list( File "C:\ProgramData\Anaconda3\lib\site-packages\tqdm\std.py", line 1129, in iter for obj in iterable: File "C:\ProgramData\Anaconda3\lib\concurrent\futures_base.py", line 611, in result_iterator yield fs.pop().result() File "C:\ProgramData\Anaconda3\lib\concurrent\futures_base.py", line 439, in result return self.__get_result() File "C:\ProgramData\Anaconda3\lib\concurrent\futures_base.py", line 388, in __get_result raise self._exception File "C:\ProgramData\Anaconda3\lib\concurrent\futures\thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\ego4d\cli\download.py", line 244, in lambda x: x.s3_object and not already_downloaded(x) and x.s3_exists, File "C:\ProgramData\Anaconda3\lib\site-packages\ego4d\cli\download.py", line 193, in already_downloaded download.s3_exists = download.exists() File "C:\ProgramData\Anaconda3\lib\site-packages\ego4d\cli\download.py", line 65, in exists self.s3_object.load() File "C:\ProgramData\Anaconda3\lib\site-packages\boto3\resources\factory.py", line 564, in do_action response = action(self, *args, **kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\boto3\resources\action.py", line 88, in call response = getattr(parent.meta.client, operation_name)(*args, **params) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\client.py", line 894, in _make_api_call http, parsed_response = self._make_request( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\client.py", line 917, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 116, in make_request return self._send_request(request_dict, operation_model) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 199, in _send_request while self._needs_retry( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 351, in _needs_retry responses = self._event_emitter.emit( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\hooks.py", line 256, in emit return self._emit(event_name, kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\hooks.py", line 239, in _emit response = handler(**kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\utils.py", line 1579, in redirect_from_error new_region = self.get_bucket_region(bucket, response) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\utils.py", line 1638, in get_bucket_region response = self._client.head_bucket(Bucket=bucket) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\client.py", line 508, in _api_call return self._make_api_call(operation_name, kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\client.py", line 894, in _make_api_call http, parsed_response = self._make_request( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\client.py", line 917, in _make_request return self._endpoint.make_request(operation_model, request_dict) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 116, in make_request return self._send_request(request_dict, operation_model) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 199, in _send_request while self._needs_retry( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 351, in _needs_retry responses = self._event_emitter.emit( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\hooks.py", line 256, in emit return self._emit(event_name, kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\hooks.py", line 239, in _emit response = handler(**kwargs) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\retryhandler.py", line 207, in call if self._checker(**checker_kwargs): File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\retryhandler.py", line 284, in call should_retry = self._should_retry( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\retryhandler.py", line 320, in _should_retry return self._checker(attempt_number, response, caught_exception) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\retryhandler.py", line 363, in call checker_response = checker( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\retryhandler.py", line 247, in call return self._check_caught_exception( File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\retryhandler.py", line 416, in _check_caught_exception raise caught_exception File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 278, in _do_get_response http_response = self._send(request) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\endpoint.py", line 374, in _send return self.http_session.send(request) File "C:\ProgramData\Anaconda3\lib\site-packages\botocore\httpsession.py", line 472, in send raise EndpointConnectionError(endpoint_url=request.url, error=e) botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://ego4d-unict.s3.eu-central-1.amazonaws.com/"

But I can download benchmark AV successfully.

Has anyone else received the problem? How can I solve this problem? Thank you so much!

Additionally update reqiurements.txt

Changes:

• Refactor such that we get audio inputs and can optionally normalize them.
• Adds mel spectrogram "model", which is very simple
• Add flag to exclude videos without audio

Hi, an error occurred when I ran this command: "ego4d --output_directory="~/ego4d_data" --datasets full_scale annotations --metadata". Could you tell me how to solve it? Thank you!

Hi, I keep getting the error "An error occurred (403) when calling the HeadObject operation: Forbidden" when I run the CLI command to download the dataset. I am already an IAM user with root privileges and added the key & the id for AWS cli.

Has anyone else received the problem? Thanks

Hi,

Thanks for the code release. I noticed that the filter_already downloaded function seems to have 2 issues:

• it doesn't current operate as expected since the files seem to be missing a version which gets triggered as a mismatch. As a result, all files are redownloaded as noted by #48
• Even if we comment out the version checks in cli/download.py (L207-212, L219-225). The check for number of downloads is incorrect since to_download at that point refers to an array of booleans which will always have the same length and is only used to filter the statements in the return statement using list(compress(downloads, to_download)). This only affects the log as the function returns the correctly filtered set.

I can submit a PR to remove the version checks and fix the to_download, however, I wasn't sure if there were plans to use version checking.

Thanks, Mohamed

Downloading Ego4D metadata json.. Traceback (most recent call last): File "E:\SoftWare\Anaconda\lib\runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "E:\SoftWare\Anaconda\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "E:\Project\project\Ego4d\ego4d\cli\cli.py", line 216, in main(config) File "E:\Project\project\Ego4d\ego4d\cli\cli.py", line 63, in main metadata_path = download_metadata( File "E:\Project\project\Ego4d\ego4d\cli\manifest.py", line 189, in download_metadata _metadata_object(version, s3).download_file(str(download_path)) File "E:\SoftWare\Anaconda\lib\site-packages\boto3\s3\inject.py", line 319, in object_download_file return self.meta.client.download_file( File "E:\SoftWare\Anaconda\lib\site-packages\boto3\s3\inject.py", line 173, in download_file return transfer.download_file( File "E:\SoftWare\Anaconda\lib\site-packages\boto3\s3\transfer.py", line 315, in download_file future.result() File "E:\SoftWare\Anaconda\lib\site-packages\s3transfer\futures.py", line 103, in result return self._coordinator.result() File "E:\SoftWare\Anaconda\lib\site-packages\s3transfer\futures.py", line 266, in result raise self._exception File "E:\SoftWare\Anaconda\lib\site-packages\s3transfer\tasks.py", line 269, in _main self._submit(transfer_future=transfer_future, **kwargs) File "E:\SoftWare\Anaconda\lib\site-packages\s3transfer\download.py", line 354, in _submit response = client.head_object( File "E:\SoftWare\Anaconda\lib\site-packages\botocore\client.py", line 391, in _api_call return self._make_api_call(operation_name, kwargs) File "E:\SoftWare\Anaconda\lib\site-packages\botocore\client.py", line 719, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Running python -m ego4d.cli.cli --aws_profile_name="ego4d" --output_directory="./ego4d_data" --datasets full_scale annotations --metadata gives a bunch of warnings that some mp4s are missing on S3:

Boto 403 Exception For exists: a0b8bc0a-9ed2-489b-a31c-48e1afac1bc1 | a0b8bc0a-9ed2-489b-a31c-48e1afac1bc1.mp4
Missing s3 object (ignored for download): a0b8bc0a-9ed2-489b-a31c-48e1afac1bc1 | a0b8bc0a-9ed2-489b-a31c-48e1afac1bc1.mp4

Boto 403 Exception For exists: eaa560d5-6432-4030-a03a-b6ba512e621d | eaa560d5-6432-4030-a03a-b6ba512e621d.mp4
Missing s3 object (ignored for download): eaa560d5-6432-4030-a03a-b6ba512e621d | eaa560d5-6432-4030-a03a-b6ba512e621d.mp4

Boto 403 Exception For exists: 19a7cce8-5771-4ec9-bdd0-189cc34c082d | 19a7cce8-5771-4ec9-bdd0-189cc34c082d.mp4
Missing s3 object (ignored for download): 19a7cce8-5771-4ec9-bdd0-189cc34c082d | 19a7cce8-5771-4ec9-bdd0-189cc34c082d.mp4
...

No existing videos to filter.
ERROR:root:1515/9670 missing S3 downloads will be ignored
Downloading 8155/9670..
Expected size of downloaded files is 5448.6 GB. Do you want to start the download? (y/n)

Is this expectable behavior?

name: ego4d
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - ca-certificates=2021.10.26=h06a4308_2
  - certifi=2021.10.8=py38h06a4308_2
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - ncurses=6.3=h7f8727e_2
  - openssl=1.1.1m=h7f8727e_0
  - pip=21.2.4=py38h06a4308_0
  - python=3.8.12=h12debd9_0
  - readline=8.1.2=h7f8727e_1
  - setuptools=58.0.4=py38h06a4308_0
  - sqlite=3.37.2=hc218d9a_0
  - tk=8.6.11=h1ccaba5_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7f8727e_4
  - pip:
    - boto3==1.21.3
    - botocore==1.24.3
    - ego4d==1.0
    - jmespath==0.10.0
    - python-dateutil==2.8.2
    - s3transfer==0.5.1
    - six==1.16.0
    - tqdm==4.62.3
    - urllib3==1.26.8

Refer to the README in ego4d/research/clep/README.md for usage instructions

Code structure:

• ego4d/research with a utility dataloader for Features and associated function for preprocessing them to HDF5 (using h5py)
• ego4d/research/clep, CLIP-based training on Ego4D/narrations akin to EgoVLP. Presented at CVPR. Structured as follows:
  • dataset.py: associated datasets/data loader utilities
  • config.py: configuration dataclasses for training, preprocessing, etc.
  • train.py: a LightningLite training script
  • val.py: associated evaluation code
  • configs: hydra YAML config files
  • run_preprocess.py: script to preprocess data for training/validation
  • preprocess/<data>.py
    • associated preprocess script for specific dataset, i.e. Ego4D, CC, Ego-Charades, Kinetics
• A notebook covering a basic high-level overview of how it works

Hi, Some videos encountered strange bugs like below video = av.open(path) video.seek(0) # PermissionError: [Errno 1] Operation not permitted I've checked out the buggy video and everything seems normal.

So, could I remove the Line65 self.vid._container.seek(0) in the features/dataset.py ?

In your paper, You commented about making mini-sets in "3.6 Dataset Accessibility" How can i get them? Is in Ego4d Download API?

And last, Is there annotations composed of only verb/noun ?

We've been looking carefully at the IMU data, and sometimes it seems like there are videos where the IMU/video syncing gets off a bit. In other words, just looking at the raw data, it looks like the timestamps of the IMU datapoints (which are reported relative to a start time for the video stream) are sometimes -- not always, but for portions of some videos -- misaligned to the video by at least 500ms or maybe even as much as 1s.

This may not seem like a lot in absolute terms, but it's actually quite an issue, since this will often mean that an action, such as the viewer turning their head, will appear in the video stream meaningfully before the IMU stream registers the action, breaking the temporal causality in the ordering between the streams.

I've attached a list of four videos with timestamps / frame ranges where we see the issue, but happy to give more examples if that would be helpful. It happens pretty frequently (like maybe half the times where there is IMU data to start with).

It does seem like this situation mostly (or maybe exclusively) occurs when another known problem in the data happens -- namely a non-monotonicity of the IMU data timestamps. (We read in the data documentation about this non monotonicity being a known problem.) It seems to us that whatever caused the non-monotonicity might sometimes coincide with an event that causes the IMU timestamping to get out of sync with the video timestamps.

So our questions are:

(1) Have you folks noticed this issue?

(2) Do you have idea how to fix the problem? The data documentation suggests "sorting" IMU data according to the timestamps whenever the non-monotonicity arises, but it doesn't seem like doing that would resolve the issue we're reporting. Any suggestions?

Here is a list of some of a few examples (with video ID, time, and frame number) where we saw the issue, but there are many more like this.

video_id     time frame b148c2bd-349a-4c3d-af5c-6791c300bea9    01:52-01:57    3360-3510 1246d6ec-5620-4f71-8b4b-d823775f58c2    00:04-00:05    120-150 b5410470-6cb6-43ea-8233-4824ba6a27b0    10:01-10:02    18030-18060 fe058f9c-d653-488f-897e-125040d12cf6    05:27-05:29    9810-9870

Hi there, thanks a lot for your work!

I would like to get the action labels matching the point of no return annotations. That is, given an annotation from fho_oscc-pnr_train.json, I would like to get the corresponding action segment, in order to get the verb and noun labels corresponding to the OSCC annotation.

Quoting from the Hand & Object Interactions docs

We select the data to annotate based on activities that are likely to involve hand-object interactions (e.g., knitting, carpentry, baking, etc.). We start by labeling each narrated hand-object interaction. For each, we label three moments in time (pre, PNR, post) and the bounding boxes for the hands, tools, and objects in each of the three frames. We also annotate the state change types (remove, burn, etc.), action verbs, and nouns for the objects.

it seems that indeed it should be possible to get verbs and nouns (the action labels I'm after) for each PNR annotation. However, I could not find this information.

In fho_scod_train.json I found object labels corresponding to each PNR, however I could not find the verb label.

I tried matching the PNR in fho_oscc-pnr_train.json to the action segments in fho_lta_train.json, however I wonder if there is a better way of achieving this goal.

Thanks in advance!

Successful command:

ego4d --output_directory="F:/" --datasets annotations
ego4d --output_directory="G:/" --datasets video_540ss

Failed Command:

ego4d --output_directory="F:/" --datasets annotations_540ss

Error:

Warning: Non-standard Dataset Specfied (Allowed, will attempt download): ['annotations_540ss']
Datasets to download: {'annotations_540ss'}
Download Path: F:\v1
Ego4D Metadata: F:\ego4d.json
Checking requested datasets and versions...
Created download directory for version 'v1' of dataset: 'annotations_540ss' at: F:\v1\annotations_540ss
Traceback (most recent call last):
  File "C:\Anaconda\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Anaconda\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Anaconda\Scripts\ego4d.exe\__main__.py", line 7, in <module>
  File "C:\Anaconda\lib\site-packages\ego4d\cli\cli.py", line 251, in main
    main_cfg(config)
  File "C:\Anaconda\lib\site-packages\ego4d\cli\cli.py", line 119, in main_cfg
    manifest_path = download_manifest_for_version(
  File "C:\Anaconda\lib\site-packages\ego4d\cli\manifest.py", line 159, in download_manifest_for_version
    _manifest_object(version, dataset, s3).download_file(str(download_path))
  File "C:\Anaconda\lib\site-packages\boto3\s3\inject.py", line 359, in object_download_file
    return self.meta.client.download_file(
  File "C:\Anaconda\lib\site-packages\boto3\s3\inject.py", line 190, in download_file
    return transfer.download_file(
  File "C:\Anaconda\lib\site-packages\boto3\s3\transfer.py", line 320, in download_file
    future.result()
  File "C:\Anaconda\lib\site-packages\s3transfer\futures.py", line 103, in result
    return self._coordinator.result()
  File "C:\Anaconda\lib\site-packages\s3transfer\futures.py", line 266, in result
    raise self._exception
  File "C:\Anaconda\lib\site-packages\s3transfer\tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "C:\Anaconda\lib\site-packages\s3transfer\download.py", line 354, in _submit
    response = client.head_object(
  File "C:\Anaconda\lib\site-packages\botocore\client.py", line 507, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Anaconda\lib\site-packages\botocore\client.py", line 943, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

Hello,

'SSLError' ocurred when I ran the command 'ego4d --output_directory="G:/Folder2/ego4d_data" --datasets clips' to download clips.

Could you tell me how to solve it? Thanks in advance!