A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

Related tags

Deep Learning webdataset-lightning
Overview

A small demonstration of using WebDataset with ImageNet and PyTorch Lightning

This is a small repo illustrating how to use WebDataset on ImageNet. using the PyTorch Lightning framework.

First, create the virtualenv:

$ ./run venv  # make virtualenv

Next, you need to shard the ImageNet data:

$ ln -s /some/imagenet/directory data
$ mkdir shards
$ ./run makeshards  # create shards

Run the training script:

$ ./run train -b 128 --gpus 2 # run the training jobs using PyTorch lightning

Of course, for local data, there is no need to go through this trouble. However, you can now easily train remotely, for example by putting the data on a webserver:

$ rsync -av shards webserver:/var/www/html/shards
$ ./run train --gpus 2 --bucket http://webserver/shards

The AIStore server is a high performance S3-compatible storage server (and web server) that works very with WebDataset.

Comments
  • When using ddp with lighting it stuck while training

    When using ddp with lighting it stuck while training

    I followed the sample this repo provided. set NCCL_IB_DISABLE=1 and NCCL_SOCKET_IFNAME=eth0, it will stuck while start training.

    When I switch normal dataloader, it works on ddp accelerator. It is clearly because of webdataset not support distributed sampling. Would you guys fix it?

    initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
NCCL version 2.7.8+cuda11.0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 0 sendbuff 0x7f4fec200000 recvbuff 0x7f4fec200000 count 1 datatype 1 op 0 root 0 comm 0x7f4fe0000e00 [nranks=2] stream 0x56123d2c4000
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 0 sendbuff 0x7fecd2200000 recvbuff 0x7fecd2200000 count 1 datatype 1 op 0 root 0 comm 0x7feca4000e00 [nranks=2] stream 0x55f74bed1ee0
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All DDP processes registered. Starting ddp with 2 processes
----------------------------------------------------------------------------------------------------

6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 1 sendbuff 0x7fecd2200000 recvbuff 0x7fecd2200000 count 1 datatype 1 op 0 root 0 comm 0x7feca4000e00 [nranks=2] stream 0x55f74bed1ee0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 1 sendbuff 0x7f4fec200000 recvbuff 0x7f4fec200000 count 1 datatype 1 op 0 root 0 comm 0x7f4fe0000e00 [nranks=2] stream 0x56123d2c4000
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 2 sendbuff 0x7fecd2200000 recvbuff 0x7fecd2200000 count 1 datatype 1 op 0 root 0 comm 0x7feca4000e00 [nranks=2] stream 0x55f74bed1ee0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 2 sendbuff 0x7f4fec200000 recvbuff 0x7f4fec200000 count 1 datatype 1 op 0 root 0 comm 0x7f4fe0000e00 [nranks=2] stream 0x56123d2c4000
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3,4,5,6,7]
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 3 sendbuff 0x7fecd2200000 recvbuff 0x7fecd2200000 count 1 datatype 1 op 0 root 0 comm 0x7feca4000e00 [nranks=2] stream 0x55f74bed1ee0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 3 sendbuff 0x7f4fec200000 recvbuff 0x7f4fec200000 count 1 datatype 1 op 0 root 0 comm 0x7f4fe0000e00 [nranks=2] stream 0x56123d2c4000
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO AllGather: opCount 0 sendbuff 0x7fec33600000 recvbuff 0x7fec33600600 count 8 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO AllGather: opCount 0 sendbuff 0x7f4f73600000 recvbuff 0x7f4f73600600 count 8 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 1 sendbuff 0x7fec33600000 recvbuff 0x7fec33600000 count 8 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 1 sendbuff 0x7f4f73600000 recvbuff 0x7f4f73600000 count 8 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 2 sendbuff 0x7f4f73600200 recvbuff 0x7f4f73600200 count 31 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 2 sendbuff 0x7fec33600200 recvbuff 0x7fec33600200 count 31 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 3 sendbuff 0x7f4f3a000000 recvbuff 0x7f4f3a000000 count 39082372 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 3 sendbuff 0x7fec00000000 recvbuff 0x7fec00000000 count 39082372 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 4 sendbuff 0x7f4f73746000 recvbuff 0x7f4f73746000 count 2416 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 4 sendbuff 0x7fec33746000 recvbuff 0x7fec33746000 count 2416 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 5 sendbuff 0x7fec33746200 recvbuff 0x7fec33746200 count 8 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 5 sendbuff 0x7f4f73746200 recvbuff 0x7f4f73746200 count 8 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 6 sendbuff 0x7f4f73746400 recvbuff 0x7f4f73746400 count 68 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 6 sendbuff 0x7fec33746400 recvbuff 0x7fec33746400 count 68 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 0 sendbuff 0x7f4fec200000 recvbuff 0x7f4fec200000 count 1 datatype 1 op 0 root 0 comm 0x7f4f34000e00 [nranks=4] stream 0x56129c6a94c0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO AllReduce: opCount 0 sendbuff 0x7f4f73746200 recvbuff 0x7f4f73746200 count 1 datatype 1 op 0 root 0 comm 0x7f4f2c000e00 [nranks=4] stream 0x5612b830be50
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 0 sendbuff 0x7fecd2200000 recvbuff 0x7fecd2200000 count 1 datatype 1 op 0 root 0 comm 0x7fec2c12a000 [nranks=4] stream 0x55f7ac1717f0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO AllReduce: opCount 0 sendbuff 0x7fec33746200 recvbuff 0x7fec33746200 count 1 datatype 1 op 0 root 0 comm 0x7febf8000e00 [nranks=4] stream 0x55f7c9489e90
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 1 sendbuff 0x7f4ea9600000 recvbuff 0x7f4ea9600000 count 1 datatype 1 op 0 root 0 comm 0x7f4f34000e00 [nranks=4] stream 0x56129c6a94c0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO AllReduce: opCount 1 sendbuff 0x7f4f73746200 recvbuff 0x7f4f73746200 count 1 datatype 1 op 0 root 0 comm 0x7f4f2c000e00 [nranks=4] stream 0x5612b830be50
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 1 sendbuff 0x7feb75000000 recvbuff 0x7feb75000000 count 1 datatype 1 op 0 root 0 comm 0x7fec2c12a000 [nranks=4] stream 0x55f7ac1717f0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO AllReduce: opCount 1 sendbuff 0x7fec33746200 recvbuff 0x7fec33746200 count 1 datatype 1 op 0 root 0 comm 0x7febf8000e00 [nranks=4] stream 0x55f7c9489e90
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 7 sendbuff 0x7f4f73746200 recvbuff 0x7f4f73746200 count 8 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO Broadcast: opCount 8 sendbuff 0x7f4f73746400 recvbuff 0x7f4f73746400 count 80 datatype 0 op 0 root 0 comm 0x7f4f64000e00 [nranks=2] stream 0x56129cdd4b80
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 7 sendbuff 0x7fec33746200 recvbuff 0x7fec33746200 count 8 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO Broadcast: opCount 8 sendbuff 0x7fec33746400 recvbuff 0x7fec33746400 count 80 datatype 0 op 0 root 0 comm 0x7fec2c000e00 [nranks=2] stream 0x55f7c95060a0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [1] NCCL INFO AllReduce: opCount 2 sendbuff 0x7feb75000000 recvbuff 0x7feb75000000 count 1 datatype 1 op 0 root 0 comm 0x7fec2c12a000 [nranks=4] stream 0x55f7ac1717f0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44748:44748 [4] NCCL INFO AllReduce: opCount 2 sendbuff 0x7fec33746200 recvbuff 0x7fec33746200 count 1 datatype 1 op 0 root 0 comm 0x7febf8000e00 [nranks=4] stream 0x55f7c9489e90

   | Name                         | Type       | Params
-------------------------------------------------------------
0  | atom_encoder                 | Linear     | 22.0 K
1  | edge_encoder                 | Linear     | 64    
2  | spatial_pos_encoder          | Embedding  | 16.4 K
3  | degree_encoder               | Embedding  | 262 K 
4  | input_dropout                | Dropout    | 0     
5  | layers                       | ModuleList | 9.5 M 
6  | final_ln                     | LayerNorm  | 1.0 K 
7  | out_proj                     | Linear     | 513   
8  | graph_token                  | Embedding  | 512   
9  | graph_token_virtual_distance | Embedding  | 32    
10 | loss_fn                      | MSELoss    | 0     
-------------------------------------------------------------
9.8 M     Trainable params
0         Non-trainable params
9.8 M     Total params
39.082    Total estimated model params size (MB)
Validation sanity check: 0it [00:00, ?it/s]6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 2 sendbuff 0x7f4ea9600000 recvbuff 0x7f4ea9600000 count 1 datatype 1 op 0 root 0 comm 0x7f4f34000e00 [nranks=4] stream 0x56129c6a94c0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO AllReduce: opCount 2 sendbuff 0x7f4f73746200 recvbuff 0x7f4f73746200 count 1 datatype 1 op 0 root 0 comm 0x7f4f2c000e00 [nranks=4] stream 0x5612b830be50
Validation sanity check:  50%|████████████████████████████████████████████████████████████████████████████████████████████████████████▌                                                                                                        | 1/2 [00:03<00:03,  3.06s/it]At validation_step, global_step= 0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [0] NCCL INFO AllReduce: opCount 3 sendbuff 0x7f4ea9600000 recvbuff 0x7f4ea9600000 count 1 datatype 1 op 0 root 0 comm 0x7f4f34000e00 [nranks=4] stream 0x56129c6a94c0
6d4c7ddf-e133-482e-ac7d-237da037cd0f:44670:44670 [3] NCCL INFO AllReduce: opCount 3 sendbuff 0x7f4f73746c00 recvbuff 0x7f4f73746c00 count 1 datatype 1 op 0 root 0 comm 0x7f4f2c000e00 [nranks=4] stream 0x5612b830be50
    opened by lkfo415579 1
  • [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning 
    tensorboard 2.1.0 requires protobuf, which is not installed.
ray 0.8.0 requires faulthandler, which is not installed.
ray 0.8.0 requires jsonschema, which is not installed.
ray 0.8.0 requires protobuf, which is not installed.

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- medium severity | 551/1000
    Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-WHEEL-3180413 | wheel:
    0.30.0 -> 0.38.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.

    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Regular Expression Denial of Service (ReDoS)

    opened by tmbdev 0
  • [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning 
    tensorboard 2.1.0 requires protobuf, which is not installed.
tensorboard 2.1.0 requires grpcio, which is not installed.
tensorboard 2.1.0 has requirement setuptools>=41.0.0, but you have setuptools 39.0.1.
google-auth 1.35.0 has requirement setuptools>=40.3.0, but you have setuptools 39.0.1.

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- low severity | 441/1000
    Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
    39.0.1 -> 65.5.1
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.

    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Regular Expression Denial of Service (ReDoS)

    opened by tmbdev 0
  • [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning 
    tensorboard 2.1.0 requires protobuf, which is not installed.
tensorboard 2.1.0 requires grpcio, which is not installed.
ray 0.8.0 requires protobuf, which is not installed.
ray 0.8.0 requires jsonschema, which is not installed.

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- medium severity | 551/1000
    Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-WHEEL-3092128 | wheel:
    0.30.0 -> 0.38.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.

    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Regular Expression Denial of Service (ReDoS)

    opened by tmbdev 0
  • [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning 
    tensorboard 2.1.0 requires protobuf, which is not installed.
tensorboard 2.1.0 requires grpcio, which is not installed.
ray 0.8.0 requires protobuf, which is not installed.
ray 0.8.0 requires jsonschema, which is not installed.

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- medium severity | 551/1000
    Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-WHEEL-3092128 | wheel:
    0.30.0 -> 0.38.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.

    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Regular Expression Denial of Service (ReDoS)

    opened by tmbdev 0
  • [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    [Snyk] Security upgrade wheel from 0.30.0 to 0.38.0

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning 
    tensorboard 2.1.0 requires protobuf, which is not installed.
tensorboard 2.1.0 requires grpcio, which is not installed.
ray 0.8.0 requires protobuf, which is not installed.
ray 0.8.0 requires jsonschema, which is not installed.

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- medium severity | 551/1000
    Why? Recently disclosed, Has a fix available, CVSS 5.3 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-WHEEL-3092128 | wheel:
    0.30.0 -> 0.38.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.

    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Regular Expression Denial of Service (ReDoS)

    opened by tmbdev 0
  • [Snyk] Security upgrade ray from 1.0.1.post1 to 1.8.0

    [Snyk] Security upgrade ray from 1.0.1.post1 to 1.8.0

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning 
    tensorboard 2.1.0 requires protobuf, which is not installed.
ray 0.8.0 requires protobuf, which is not installed.
ray 0.8.0 requires jsonschema, which is not installed.

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- medium severity | 616/1000
    Why? Proof of Concept exploit, Has a fix available, CVSS 5.9 | Race Condition
    SNYK-PYTHON-RAY-2308017 | ray:
    1.0.1.post1 -> 1.8.0
    | No | Proof of Concept

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.

    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Learn about vulnerability in an interactive lesson of Snyk Learn.

    opened by tmbdev 0
Owner
High Performance I/O for Large Scale Deep Learning
null
GitHub
Predicting path with preference based on user demonstration using Maximum Entropy Deep Inverse Reinforcement Learning in a continuous environment

Preference-Planning-Deep-IRL Introduction Check my portfolio post Dependencies Gym stable-baselines3 PyTorch Usage Take Demonstration python3 record.

Tianyu Li 9 Oct 26, 2022
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 11 Jul 28, 2022
A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.

Object Pose Estimation Demo This tutorial will go through the steps necessary to perform pose estimation with a UR3 robotic arm in Unity. You’ll gain

Unity Technologies 187 Dec 24, 2022
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pytorch Lightning 1.4k Jan 1, 2023
Public implementation of "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression" from CoRL'21

Self-Supervised Reward Regression (SSRR) Codebase for CoRL 2021 paper "Learning from Suboptimal Demonstration via Self-Supervised Reward Regression "

null 19 Dec 12, 2022
robomimic: A Modular Framework for Robot Learning from Demonstration

robomimic [Homepage]   [Documentation]   [Study Paper]   [Study Website]   [ARISE Initiative] Latest Updates [08/09/2021] v0.1.0: Initial code and pap

ARISE Initiative 178 Jan 5, 2023
Demonstration of the Model Training as a CI/CD System in Vertex AI

Model Training as a CI/CD System This project demonstrates the machine model training as a CI/CD system in GCP platform. You will see more detailed wo

Chansung Park 19 Dec 28, 2022
Modified fork of Xuebin Qin's U-2-Net Repository. Used for demonstration purposes.

U^2-Net (U square net) Modified version of U2Net used for demonstation purposes. Paper: U^2-Net: Going Deeper with Nested U-Structure for Salient Obje

Shreyas Bhat Kera 13 Aug 28, 2022
A TensorFlow implementation of Neural Program Synthesis from Diverse Demonstration Videos

ViZDoom http://vizdoom.cs.put.edu.pl ViZDoom allows developing AI bots that play Doom using only the visual information (the screen buffer). It is pri

Hyeonwoo Noh 1 Aug 19, 2020
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022
Spiking Neural Network for Computer Vision using SpikingJelly framework and Pytorch-Lightning

Spiking Neural Network for Computer Vision using SpikingJelly framework and Pytorch-Lightning

Sami BARCHID 2 Oct 20, 2022
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
A general framework for deep learning experiments under PyTorch based on pytorch-lightning

torchx Torchx is a general framework for deep learning experiments under PyTorch based on pytorch-lightning. TODO list gan-like training wrapper text

Yingtian Liu 6 Mar 17, 2022
A PyTorch re-implementation of the paper 'Exploring Simple Siamese Representation Learning'. Reproduced the 67.8% Top1 Acc on ImageNet.

Exploring simple siamese representation learning This is a PyTorch re-implementation of the SimSiam paper on ImageNet dataset. The results match that

Taojiannan Yang 72 Nov 9, 2022
(ImageNet pretrained models) The official pytorch implemention of the TPAMI paper "Res2Net: A New Multi-scale Backbone Architecture"

Res2Net The official pytorch implemention of the paper "Res2Net: A New Multi-scale Backbone Architecture" Our paper is accepted by IEEE Transactions o

Res2Net Applications 928 Dec 29, 2022
Pytorch implementation of "Training a 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet"

Token Labeling: Training an 85.4% Top-1 Accuracy Vision Transformer with 56M Parameters on ImageNet (arxiv) This is a Pytorch implementation of our te

蒋子航 383 Dec 27, 2022
Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(2021) paper

ImageNet-21K Pretraining for the Masses Paper | Pretrained models Official PyTorch Implementation Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, Lihi Zelni

null 574 Jan 2, 2023
PyTorch implementation of PNASNet-5 on ImageNet

PNASNet.pytorch PyTorch implementation of PNASNet-5. Specifically, PyTorch code from this repository is adapted to completely match both my implemetat

Chenxi Liu 314 Nov 25, 2022
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

null 32 Dec 26, 2022