[CVPR'2020] DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data

Overview

DeepDeform (CVPR'2020)

DeepDeform is an RGB-D video dataset containing over 390,000 RGB-D frames in 400 videos, with 5,533 optical and scene flow images and 4,479 foreground object masks. We also provide 149,228 sparse match annotations and 63,512 occlusion point annotations.

Download Data

If you would like to download the DeepDeform data, please fill out this google form and, once accepted, we will send you the link to download the data.

Online Benchmark

If you want to participate in the benchmark(s), you can submit your results at DeepDeform Benchmark website.

Currently we provide benchmarks for the following tasks:

By uploading your results on the test set to the DeepDeform Benchmark website the performance of you method is automatically evaluated on the hidden test labels, and compared to other already evaluated methods. You can decide if you want to make the evaluation results public or not.

If you want to evaluate on validation set, we provide code that is used for evaluation of specific benchmarks in directory evaluation/. To evaluate optical flow or non-rigid reconstruction, you need to adapt FLOW_RESULTS_DIR or RECONSTRUCTION_RESULTS_DIR in config.py to correspond to your results directory (that would be in the same format as for the online submission, described here).

In order to evaluate reconstruction, you need to compile additional C++ modules.

  • Install necessary dependencies:
pip install pybind11
pip install Pillow
pip install plyfile
pip install tqdm
pip install scikit-image
  • Inside the evaluation/csrc adapt includes.py to point to your Eigen include directory.

  • Compile the code by executing the following in evaluation/csrc:

python setup.py install

Data Organization

Data is organized into 3 subsets, train, val, and test directories, using 340-30-30 sequence split. In every subset each RGB-D sequence is stored in a directory <sequence_id>, which follows the following format:

<sequence_id>
|-- <color>: color images for every frame (`%06d.jpg`)
|-- <depth>: depth images for every frame (`%06d.png`)
|-- <mask>: mask images for a few frames (`%06d.png`)
|-- <optical_flow>: optical flow images for a few frame pairs (`<object_id>_<source_id>_<target_id>.oflow` or `%s_%06d_%06d.oflow`)
|-- <scene_flow>: scene flow images for a few frame pairs (`<object_id>_<source_id>_<target_id>.sflow` or `%s_%06d_%06d.sflow`)
|-- <intrinsics.txt>: 4x4 intrinsics matrix

All labels are provided in .json files in root dataset r directory:

  • train_matches.json and val_matches.json:
    Manually annotated sparse matches.
  • train_dense.json and val_dense.json:
    Densely aligned optical and scene flow images with the use of sparse matches as a guidance.
  • train_selfsupervised.json and val_selfsupervised.json:
    Densely aligned optical and scene flow images using self-supervision (DynamicFusion pipeline) for a few sequences. - train_selfsupervised.json and `val_skaldir
  • train_masks.json and val_masks.json:
    Dynamic object annotations for a few frames per sequence.
  • train_occlusions.json and val_occlusions.json:
    Manually annotated sparse occlusions.

Data Formats

We recommend you to test out scripts in demo/ directory in order to check out loading of different file types.

RGB-D Data: 3D data is provided as RGB-D video sequences, where color and depth images are already aligned. Color images are provided as 8-bit RGB .jpg, and depth images as 16-bit .png (divide by 1000 to obtain depth in meters).

Camera Parameters: A 4x4 intrinsic matrix is given for every sequence (because different cameras were used for data capture, every sequence can have different intrinsic matrix). Since the color and depth images are aligned, no extrinsic transformation is necessary.

Optical Flow Data: Dense optical flow data is provided as custom binary image of resolution 640x480 with extension .oflow. Every pixel contains two values for flow in x and y direction, in pixels. Helper function to load/store binary flow images is provided in utils.py.

Scene Flow Data: Dense scene flow data is provided as custom binary image of resolution 640x480 with extension .sflow. Every pixel contains 3 values for flow in x, y and z direction, in meters. Helper function to load/store binary flow images is provided in utils.py.

Object Mask Data: A few frames per sequences also include foreground dynamic object annotation. The mask image is given as 16-bit .png image (1 for object, 0 for background).

Sparse Match Annotations: We provide manual sparse match annotations for a few frame pairs for every sequence. They are stored in .json format, with paths to corresponding source and target RGB-D frames, as a list of source and target pixels.

Sparse Occlusion Annotations: We provide manual sparse occlusion annotations for a few frame pairs for every sequence. They are stored in .json format, with paths to corresponding source and target RGB-D frames, as a list of occluded pixels in source frame.

Citation

If you use DeepDeform data or code please cite:

@inproceedings{bozic2020deepdeform, 
    title={DeepDeform: Learning Non-rigid RGB-D Reconstruction with Semi-supervised Data}, 
    author={Bo{\v{z}}i{\v{c}}, Alja{\v{z}} and Zollh{\"o}fer, Michael and Theobalt, Christian and Nie{\ss}ner, Matthias}, 
    journal={Conference on Computer Vision and Pattern Recognition (CVPR)}, 
    year={2020}
}

Help

If you have any questions, please contact us at [email protected], or open an issue at Github.

License

The data is released under DeepDeform Terms of Use, and the code is release under a non-comercial creative commons license.

Comments
  • Generating mask for every frame

    Generating mask for every frame

    Hello! Thanks for the dataset and toolbox! I am wondering is there a way to generate object mask for every single frame, so that I can have a continuous scanned sequence of a particular object? Thanks!

    opened by zhan-xu 4
  • Some depth data missing?

    Some depth data missing?

    Thank you for making the dataset available! I believe I downloaded the data correctly, but it appears that one sequence is missing depth data-- Train sequence 000 is missing 1138 depth frames. I'm pretty sure my download went fine, but I give details below. Should we just throw out sequence 000 or is the full depth data available?

    Here's what 7z says about the archive:

    Scanning the drive for archives:
    1 file, 108827217362 bytes (102 GiB)
    
    Listing archive: deepdeform_v1.7z
    
    --
    Path = deepdeform_v1.7z
    Type = 7z
    Physical Size = 108827217362
    Headers Size = 5450645
    Method = LZMA2:24
    Solid = +
    Blocks = 86
    

    Get the file list:

    7z l deepdeform_v1.7z > deepdeform_v1.7z.filelist.txt
    

    Here's how I poked around and found the discrepancy... all the other sequences look good 👍

    lines = open('deepdeform_v1.7z.filelist.txt').readlines() 
    import re
    from collections import defaultdict
    for line in lines: 
      m = re.search('(train|test)\/seq(\d+)\/(color|depth)\/.*', line) 
      if m: 
        counts['-'.join(m.group(i) for i in (1, 2, 3))] += 1 
    import pprint
    pprint.pprint(counts)                                                                                                                    
    >>> defaultdict(<class 'int'>,
                {'test-000-color': 421,
                 'test-000-depth': 421,
                 'test-001-color': 979,
                 'test-001-depth': 979,
                 'test-002-color': 629,
                 'test-002-depth': 629,
                 'test-003-color': 432,
                 'test-003-depth': 432,
                 'test-004-color': 549,
                 'test-004-depth': 549,
                 'test-005-color': 790,
                 'test-005-depth': 790,
                 'test-006-color': 856,
                 'test-006-depth': 856,
                 'test-007-color': 763,
                 'test-007-depth': 763,
                 'test-008-color': 894,
                 'test-008-depth': 894,
                 'test-009-color': 1023,
                 'test-009-depth': 1023,
                 'test-010-color': 1179,
                 'test-010-depth': 1179,
                 'test-011-color': 1064,
                 'test-011-depth': 1064,
                 'test-012-color': 962,
                 'test-012-depth': 962,
                 'test-013-color': 1026,
                 'test-013-depth': 1026,
                 'test-014-color': 869,
                 'test-014-depth': 869,
                 'test-015-color': 1029,
                 'test-015-depth': 1029,
                 'test-016-color': 1036,
                 'test-016-depth': 1036,
                 'test-017-color': 886,
                 'test-017-depth': 886,
                 'test-018-color': 902,
                 'test-018-depth': 902,
                 'test-019-color': 952,
                 'test-019-depth': 952,
                 'test-020-color': 781,
                 'test-020-depth': 781,
                 'test-021-color': 1017,
                 'test-021-depth': 1017,
                 'test-022-color': 996,
                 'test-022-depth': 996,
                 'test-023-color': 1185,
                 'test-023-depth': 1185,
                 'test-024-color': 1130,
                 'test-024-depth': 1130,
                 'test-025-color': 1193,
                 'test-025-depth': 1193,
                 'test-026-color': 1254,
                 'test-026-depth': 1254,
                 'test-027-color': 967,
                 'test-027-depth': 967,
                 'test-028-color': 1049,
                 'test-028-depth': 1049,
                 'test-029-color': 1060,
                 'test-029-depth': 1060,
                 'train-000-color': 2632,
                 'train-000-depth': 1494,
                 'train-001-color': 4377,
                 'train-001-depth': 4377,
                 'train-002-color': 2241,
                 'train-002-depth': 2241,
                 'train-003-color': 5688,
                 'train-003-depth': 5688,
                 'train-004-color': 1610,
                 'train-004-depth': 1610,
                 'train-005-color': 1203,
                 'train-005-depth': 1203,
                 'train-006-color': 6960,
                 'train-006-depth': 6960,
                 'train-007-color': 1660,
                 'train-007-depth': 1660,
                 'train-008-color': 9685,
                 'train-008-depth': 9685,
                 'train-009-color': 15797,
                 'train-009-depth': 15797,
                 'train-010-color': 1370,
                 'train-010-depth': 1370,
                 'train-011-color': 7791,
                 'train-011-depth': 7791,
                 'train-012-color': 6421,
                 'train-012-depth': 6421,
                 'train-013-color': 6867,
                 'train-013-depth': 6867,
                 'train-014-color': 6949,
                 'train-014-depth': 6949,
                 'train-015-color': 674,
                 'train-015-depth': 674,
                 'train-016-color': 801,
                 'train-016-depth': 801,
                 'train-017-color': 719,
                 'train-017-depth': 719,
                 'train-018-color': 843,
                 'train-018-depth': 843,
                 'train-019-color': 1089,
                 'train-019-depth': 1089,
                 'train-020-color': 885,
                 'train-020-depth': 885,
                 'train-021-color': 714,
                 'train-021-depth': 714,
                 'train-022-color': 787,
                 'train-022-depth': 787,
                 'train-023-color': 837,
                 'train-023-depth': 837,
                 'train-024-color': 788,
                 'train-024-depth': 788,
                 'train-025-color': 733,
                 'train-025-depth': 733,
                 'train-026-color': 551,
                 'train-026-depth': 551,
                 'train-027-color': 1272,
                 'train-027-depth': 1272,
                 'train-028-color': 380,
                 'train-028-depth': 380,
                 'train-029-color': 342,
                 'train-029-depth': 342,
                 'train-030-color': 255,
                 'train-030-depth': 255,
                 'train-031-color': 253,
                 'train-031-depth': 253,
                 'train-032-color': 259,
                 'train-032-depth': 259,
                 'train-033-color': 236,
                 'train-033-depth': 236,
                 'train-034-color': 563,
                 'train-034-depth': 563,
                 'train-035-color': 602,
                 'train-035-depth': 602,
                 'train-036-color': 453,
                 'train-036-depth': 453,
                 'train-037-color': 333,
                 'train-037-depth': 333,
                 'train-038-color': 492,
                 'train-038-depth': 492,
                 'train-039-color': 852,
                 'train-039-depth': 852,
                 'train-040-color': 613,
                 'train-040-depth': 613,
                 'train-041-color': 856,
                 'train-041-depth': 856,
                 'train-042-color': 709,
                 'train-042-depth': 709,
                 'train-043-color': 521,
                 'train-043-depth': 521,
                 'train-044-color': 643,
                 'train-044-depth': 643,
                 'train-045-color': 678,
                 'train-045-depth': 678,
                 'train-046-color': 520,
                 'train-046-depth': 520,
                 'train-047-color': 1051,
                 'train-047-depth': 1051,
                 'train-048-color': 682,
                 'train-048-depth': 682,
                 'train-049-color': 297,
                 'train-049-depth': 297,
                 'train-050-color': 331,
                 'train-050-depth': 331,
                 'train-051-color': 260,
                 'train-051-depth': 260,
                 'train-052-color': 226,
                 'train-052-depth': 226,
                 'train-053-color': 277,
                 'train-053-depth': 277,
                 'train-054-color': 249,
                 'train-054-depth': 249,
                 'train-055-color': 294,
                 'train-055-depth': 294,
                 'train-056-color': 225,
                 'train-056-depth': 225,
                 'train-057-color': 857,
                 'train-057-depth': 857,
                 'train-058-color': 935,
                 'train-058-depth': 935,
                 'train-059-color': 760,
                 'train-059-depth': 760,
                 'train-060-color': 833,
                 'train-060-depth': 833,
                 'train-061-color': 827,
                 'train-061-depth': 827,
                 'train-062-color': 540,
                 'train-062-depth': 540,
                 'train-063-color': 603,
                 'train-063-depth': 603,
                 'train-064-color': 803,
                 'train-064-depth': 803,
                 'train-065-color': 785,
                 'train-065-depth': 785,
                 'train-066-color': 908,
                 'train-066-depth': 908,
                 'train-067-color': 771,
                 'train-067-depth': 771,
                 'train-068-color': 674,
                 'train-068-depth': 674,
                 'train-069-color': 659,
                 'train-069-depth': 659,
                 'train-070-color': 783,
                 'train-070-depth': 783,
                 'train-071-color': 761,
                 'train-071-depth': 761,
                 'train-072-color': 750,
                 'train-072-depth': 750,
                 'train-073-color': 1018,
                 'train-073-depth': 1018,
                 'train-074-color': 768,
                 'train-074-depth': 768,
                 'train-075-color': 469,
                 'train-075-depth': 469,
                 'train-076-color': 553,
                 'train-076-depth': 553,
                 'train-077-color': 813,
                 'train-077-depth': 813,
                 'train-078-color': 779,
                 'train-078-depth': 779,
                 'train-079-color': 841,
                 'train-079-depth': 841,
                 'train-080-color': 987,
                 'train-080-depth': 987,
                 'train-081-color': 684,
                 'train-081-depth': 684,
                 'train-082-color': 909,
                 'train-082-depth': 909,
                 'train-083-color': 1149,
                 'train-083-depth': 1149,
                 'train-084-color': 1150,
                 'train-084-depth': 1150,
                 'train-085-color': 1096,
                 'train-085-depth': 1096,
                 'train-086-color': 578,
                 'train-086-depth': 578,
                 'train-087-color': 527,
                 'train-087-depth': 527,
                 'train-088-color': 249,
                 'train-088-depth': 249,
                 'train-089-color': 778,
                 'train-089-depth': 778,
                 'train-090-color': 336,
                 'train-090-depth': 336,
                 'train-091-color': 256,
                 'train-091-depth': 256,
                 'train-092-color': 540,
                 'train-092-depth': 540,
                 'train-093-color': 400,
                 'train-093-depth': 400,
                 'train-094-color': 916,
                 'train-094-depth': 916,
                 'train-095-color': 914,
                 'train-095-depth': 914,
                 'train-096-color': 878,
                 'train-096-depth': 878,
                 'train-097-color': 755,
                 'train-097-depth': 755,
                 'train-098-color': 966,
                 'train-098-depth': 966,
                 'train-099-color': 800,
                 'train-099-depth': 800,
                 'train-100-color': 831,
                 'train-100-depth': 831,
                 'train-101-color': 184,
                 'train-101-depth': 184,
                 'train-102-color': 762,
                 'train-102-depth': 762,
                 'train-103-color': 287,
                 'train-103-depth': 287,
                 'train-104-color': 232,
                 'train-104-depth': 232,
                 'train-105-color': 334,
                 'train-105-depth': 334,
                 'train-106-color': 501,
                 'train-106-depth': 501,
                 'train-107-color': 778,
                 'train-107-depth': 778,
                 'train-108-color': 919,
                 'train-108-depth': 919,
                 'train-109-color': 723,
                 'train-109-depth': 723,
                 'train-110-color': 298,
                 'train-110-depth': 298,
                 'train-111-color': 838,
                 'train-111-depth': 838,
                 'train-112-color': 164,
                 'train-112-depth': 164,
                 'train-113-color': 181,
                 'train-113-depth': 181,
                 'train-114-color': 169,
                 'train-114-depth': 169,
                 'train-115-color': 417,
                 'train-115-depth': 417,
                 'train-116-color': 735,
                 'train-116-depth': 735,
                 'train-117-color': 296,
                 'train-117-depth': 296,
                 'train-118-color': 378,
                 'train-118-depth': 378,
                 'train-119-color': 376,
                 'train-119-depth': 376,
                 'train-120-color': 282,
                 'train-120-depth': 282,
                 'train-121-color': 299,
                 'train-121-depth': 299,
                 'train-122-color': 239,
                 'train-122-depth': 239,
                 'train-123-color': 180,
                 'train-123-depth': 180,
                 'train-124-color': 479,
                 'train-124-depth': 479,
                 'train-125-color': 605,
                 'train-125-depth': 605,
                 'train-126-color': 860,
                 'train-126-depth': 860,
                 'train-127-color': 686,
                 'train-127-depth': 686,
                 'train-128-color': 789,
                 'train-128-depth': 789,
                 'train-129-color': 768,
                 'train-129-depth': 768,
                 'train-130-color': 251,
                 'train-130-depth': 251,
                 'train-131-color': 321,
                 'train-131-depth': 321,
                 'train-132-color': 679,
                 'train-132-depth': 679,
                 'train-133-color': 618,
                 'train-133-depth': 618,
                 'train-134-color': 669,
                 'train-134-depth': 669,
                 'train-135-color': 581,
                 'train-135-depth': 581,
                 'train-136-color': 574,
                 'train-136-depth': 574,
                 'train-137-color': 581,
                 'train-137-depth': 581,
                 'train-138-color': 843,
                 'train-138-depth': 843,
                 'train-139-color': 779,
                 'train-139-depth': 779,
                 'train-140-color': 625,
                 'train-140-depth': 625,
                 'train-141-color': 681,
                 'train-141-depth': 681,
                 'train-142-color': 785,
                 'train-142-depth': 785,
                 'train-143-color': 834,
                 'train-143-depth': 834,
                 'train-144-color': 942,
                 'train-144-depth': 942,
                 'train-145-color': 574,
                 'train-145-depth': 574,
                 'train-146-color': 795,
                 'train-146-depth': 795,
                 'train-147-color': 661,
                 'train-147-depth': 661,
                 'train-148-color': 825,
                 'train-148-depth': 825,
                 'train-149-color': 1022,
                 'train-149-depth': 1022,
                 'train-150-color': 612,
                 'train-150-depth': 612,
                 'train-151-color': 509,
                 'train-151-depth': 509,
                 'train-152-color': 1154,
                 'train-152-depth': 1154,
                 'train-153-color': 854,
                 'train-153-depth': 854,
                 'train-154-color': 731,
                 'train-154-depth': 731,
                 'train-155-color': 1035,
                 'train-155-depth': 1035,
                 'train-156-color': 1029,
                 'train-156-depth': 1029,
                 'train-157-color': 1060,
                 'train-157-depth': 1060,
                 'train-158-color': 1369,
                 'train-158-depth': 1369,
                 'train-159-color': 1389,
                 'train-159-depth': 1389,
                 'train-160-color': 1137,
                 'train-160-depth': 1137,
                 'train-161-color': 866,
                 'train-161-depth': 866,
                 'train-162-color': 897,
                 'train-162-depth': 897,
                 'train-163-color': 1134,
                 'train-163-depth': 1134,
                 'train-164-color': 1016,
                 'train-164-depth': 1016,
                 'train-165-color': 1176,
                 'train-165-depth': 1176,
                 'train-166-color': 793,
                 'train-166-depth': 793,
                 'train-167-color': 1319,
                 'train-167-depth': 1319,
                 'train-168-color': 885,
                 'train-168-depth': 885,
                 'train-169-color': 1098,
                 'train-169-depth': 1098,
                 'train-170-color': 1207,
                 'train-170-depth': 1207,
                 'train-171-color': 1103,
                 'train-171-depth': 1103,
                 'train-172-color': 1493,
                 'train-172-depth': 1493,
                 'train-173-color': 1019,
                 'train-173-depth': 1019,
                 'train-174-color': 989,
                 'train-174-depth': 989,
                 'train-175-color': 999,
                 'train-175-depth': 999,
                 'train-176-color': 840,
                 'train-176-depth': 840,
                 'train-177-color': 1007,
                 'train-177-depth': 1007,
                 'train-178-color': 1006,
                 'train-178-depth': 1006,
                 'train-179-color': 893,
                 'train-179-depth': 893,
                 'train-180-color': 946,
                 'train-180-depth': 946,
                 'train-181-color': 936,
                 'train-181-depth': 936,
                 'train-182-color': 955,
                 'train-182-depth': 955,
                 'train-183-color': 1067,
                 'train-183-depth': 1067,
                 'train-184-color': 1954,
                 'train-184-depth': 1954,
                 'train-185-color': 1315,
                 'train-185-depth': 1315,
                 'train-186-color': 1347,
                 'train-186-depth': 1347,
                 'train-187-color': 1229,
                 'train-187-depth': 1229,
                 'train-188-color': 1018,
                 'train-188-depth': 1018,
                 'train-189-color': 1044,
                 'train-189-depth': 1044,
                 'train-190-color': 1110,
                 'train-190-depth': 1110,
                 'train-191-color': 1335,
                 'train-191-depth': 1335,
                 'train-192-color': 1222,
                 'train-192-depth': 1222,
                 'train-193-color': 1252,
                 'train-193-depth': 1252,
                 'train-194-color': 801,
                 'train-194-depth': 801,
                 'train-195-color': 1427,
                 'train-195-depth': 1427,
                 'train-196-color': 1272,
                 'train-196-depth': 1272,
                 'train-197-color': 1149,
                 'train-197-depth': 1149,
                 'train-198-color': 1369,
                 'train-198-depth': 1369,
                 'train-199-color': 1459,
                 'train-199-depth': 1459,
                 'train-200-color': 1116,
                 'train-200-depth': 1116,
                 'train-201-color': 1047,
                 'train-201-depth': 1047,
                 'train-202-color': 1434,
                 'train-202-depth': 1434,
                 'train-203-color': 1254,
                 'train-203-depth': 1254,
                 'train-204-color': 1194,
                 'train-204-depth': 1194,
                 'train-205-color': 1389,
                 'train-205-depth': 1389,
                 'train-206-color': 1379,
                 'train-206-depth': 1379,
                 'train-207-color': 1438,
                 'train-207-depth': 1438,
                 'train-208-color': 1775,
                 'train-208-depth': 1775,
                 'train-209-color': 1269,
                 'train-209-depth': 1269,
                 'train-210-color': 1267,
                 'train-210-depth': 1267,
                 'train-211-color': 1258,
                 'train-211-depth': 1258,
                 'train-212-color': 1133,
                 'train-212-depth': 1133,
                 'train-213-color': 1279,
                 'train-213-depth': 1279,
                 'train-214-color': 1149,
                 'train-214-depth': 1149,
                 'train-215-color': 1697,
                 'train-215-depth': 1697,
                 'train-216-color': 1404,
                 'train-216-depth': 1404,
                 'train-217-color': 991,
                 'train-217-depth': 991,
                 'train-218-color': 1200,
                 'train-218-depth': 1200,
                 'train-219-color': 1188,
                 'train-219-depth': 1188,
                 'train-220-color': 1515,
                 'train-220-depth': 1515,
                 'train-221-color': 1374,
                 'train-221-depth': 1374,
                 'train-222-color': 1198,
                 'train-222-depth': 1198,
                 'train-223-color': 1077,
                 'train-223-depth': 1077,
                 'train-224-color': 999,
                 'train-224-depth': 999,
                 'train-225-color': 1062,
                 'train-225-depth': 1062,
                 'train-226-color': 1052,
                 'train-226-depth': 1052,
                 'train-227-color': 1014,
                 'train-227-depth': 1014,
                 'train-228-color': 981,
                 'train-228-depth': 981,
                 'train-229-color': 920,
                 'train-229-depth': 920,
                 'train-230-color': 901,
                 'train-230-depth': 901,
                 'train-231-color': 980,
                 'train-231-depth': 980,
                 'train-232-color': 945,
                 'train-232-depth': 945,
                 'train-233-color': 878,
                 'train-233-depth': 878,
                 'train-234-color': 933,
                 'train-234-depth': 933,
                 'train-235-color': 965,
                 'train-235-depth': 965,
                 'train-236-color': 962,
                 'train-236-depth': 962,
                 'train-237-color': 877,
                 'train-237-depth': 877,
                 'train-238-color': 898,
                 'train-238-depth': 898,
                 'train-239-color': 844,
                 'train-239-depth': 844,
                 'train-240-color': 1099,
                 'train-240-depth': 1099,
                 'train-241-color': 1009,
                 'train-241-depth': 1009,
                 'train-242-color': 963,
                 'train-242-depth': 963,
                 'train-243-color': 967,
                 'train-243-depth': 967,
                 'train-244-color': 846,
                 'train-244-depth': 846,
                 'train-245-color': 838,
                 'train-245-depth': 838,
                 'train-246-color': 813,
                 'train-246-depth': 813,
                 'train-247-color': 832,
                 'train-247-depth': 832,
                 'train-248-color': 820,
                 'train-248-depth': 820,
                 'train-249-color': 1351,
                 'train-249-depth': 1351,
                 'train-250-color': 1155,
                 'train-250-depth': 1155,
                 'train-251-color': 1116,
                 'train-251-depth': 1116,
                 'train-252-color': 1074,
                 'train-252-depth': 1074,
                 'train-253-color': 1187,
                 'train-253-depth': 1187,
                 'train-254-color': 1049,
                 'train-254-depth': 1049,
                 'train-255-color': 1080,
                 'train-255-depth': 1080,
                 'train-256-color': 1142,
                 'train-256-depth': 1142,
                 'train-257-color': 1065,
                 'train-257-depth': 1065,
                 'train-258-color': 1017,
                 'train-258-depth': 1017,
                 'train-259-color': 1097,
                 'train-259-depth': 1097,
                 'train-260-color': 1098,
                 'train-260-depth': 1098,
                 'train-261-color': 1045,
                 'train-261-depth': 1045,
                 'train-262-color': 1016,
                 'train-262-depth': 1016,
                 'train-263-color': 653,
                 'train-263-depth': 653,
                 'train-264-color': 713,
                 'train-264-depth': 713,
                 'train-265-color': 631,
                 'train-265-depth': 631,
                 'train-266-color': 795,
                 'train-266-depth': 795,
                 'train-267-color': 809,
                 'train-267-depth': 809,
                 'train-268-color': 991,
                 'train-268-depth': 991,
                 'train-269-color': 1528,
                 'train-269-depth': 1528,
                 'train-270-color': 904,
                 'train-270-depth': 904,
                 'train-271-color': 1171,
                 'train-271-depth': 1171,
                 'train-272-color': 983,
                 'train-272-depth': 983,
                 'train-273-color': 771,
                 'train-273-depth': 771,
                 'train-274-color': 981,
                 'train-274-depth': 981,
                 'train-275-color': 1140,
                 'train-275-depth': 1140,
                 'train-276-color': 1113,
                 'train-276-depth': 1113,
                 'train-277-color': 935,
                 'train-277-depth': 935,
                 'train-278-color': 1073,
                 'train-278-depth': 1073,
                 'train-279-color': 917,
                 'train-279-depth': 917,
                 'train-280-color': 1353,
                 'train-280-depth': 1353,
                 'train-281-color': 1690,
                 'train-281-depth': 1690,
                 'train-282-color': 1292,
                 'train-282-depth': 1292,
                 'train-283-color': 1511,
                 'train-283-depth': 1511,
                 'train-284-color': 449,
                 'train-284-depth': 449,
                 'train-285-color': 259,
                 'train-285-depth': 259,
                 'train-286-color': 290,
                 'train-286-depth': 290,
                 'train-287-color': 226,
                 'train-287-depth': 226,
                 'train-288-color': 272,
                 'train-288-depth': 272,
                 'train-289-color': 290,
                 'train-289-depth': 290,
                 'train-290-color': 285,
                 'train-290-depth': 285,
                 'train-291-color': 259,
                 'train-291-depth': 259,
                 'train-292-color': 368,
                 'train-292-depth': 368,
                 'train-293-color': 368,
                 'train-293-depth': 368,
                 'train-294-color': 218,
                 'train-294-depth': 218,
                 'train-295-color': 197,
                 'train-295-depth': 197,
                 'train-296-color': 155,
                 'train-296-depth': 155,
                 'train-297-color': 265,
                 'train-297-depth': 265,
                 'train-298-color': 198,
                 'train-298-depth': 198,
                 'train-299-color': 414,
                 'train-299-depth': 414,
                 'train-300-color': 346,
                 'train-300-depth': 346,
                 'train-301-color': 488,
                 'train-301-depth': 488,
                 'train-302-color': 496,
                 'train-302-depth': 496,
                 'train-303-color': 869,
                 'train-303-depth': 869,
                 'train-304-color': 792,
                 'train-304-depth': 792,
                 'train-305-color': 793,
                 'train-305-depth': 793,
                 'train-306-color': 452,
                 'train-306-depth': 452,
                 'train-307-color': 565,
                 'train-307-depth': 565,
                 'train-308-color': 753,
                 'train-308-depth': 753,
                 'train-309-color': 670,
                 'train-309-depth': 670,
                 'train-310-color': 642,
                 'train-310-depth': 642,
                 'train-311-color': 789,
                 'train-311-depth': 789,
                 'train-312-color': 813,
                 'train-312-depth': 813,
                 'train-313-color': 653,
                 'train-313-depth': 653,
                 'train-314-color': 553,
                 'train-314-depth': 553,
                 'train-315-color': 677,
                 'train-315-depth': 677,
                 'train-316-color': 566,
                 'train-316-depth': 566,
                 'train-317-color': 378,
                 'train-317-depth': 378,
                 'train-318-color': 804,
                 'train-318-depth': 804,
                 'train-319-color': 667,
                 'train-319-depth': 667,
                 'train-320-color': 646,
                 'train-320-depth': 646,
                 'train-321-color': 856,
                 'train-321-depth': 856,
                 'train-322-color': 456,
                 'train-322-depth': 456,
                 'train-323-color': 508,
                 'train-323-depth': 508,
                 'train-324-color': 481,
                 'train-324-depth': 481,
                 'train-325-color': 585,
                 'train-325-depth': 585,
                 'train-326-color': 508,
                 'train-326-depth': 508,
                 'train-327-color': 613,
                 'train-327-depth': 613,
                 'train-328-color': 627,
                 'train-328-depth': 627,
                 'train-329-color': 870,
                 'train-329-depth': 870,
                 'train-330-color': 725,
                 'train-330-depth': 725,
                 'train-331-color': 734,
                 'train-331-depth': 734,
                 'train-332-color': 754,
                 'train-332-depth': 754,
                 'train-333-color': 634,
                 'train-333-depth': 634,
                 'train-334-color': 385,
                 'train-334-depth': 385,
                 'train-335-color': 473,
                 'train-335-depth': 473,
                 'train-336-color': 459,
                 'train-336-depth': 459,
                 'train-337-color': 470,
                 'train-337-depth': 470,
                 'train-338-color': 317,
                 'train-338-depth': 317,
                 'train-339-color': 445,
                 'train-339-depth': 445})
    
    opened by pwais 4
  • How to control the number of vertices?

    How to control the number of vertices?

    Thank you for public this repo. In the code I see that we need to make sure the number of vertices the same between the source and the target mesh. However, for normal fusion process, the number of vertices will be changed. So I am wondering how will we control the number of vertices?

    opened by phamtrongthang123 2
  • Dense correspondence between point clouds?

    Dense correspondence between point clouds?

    Hello,

    Thanks for this amazing work. I was wondering whether there is dense ground truth information available between two point clouds? I see that there is a per-pixel displacement from optical flow. Is it possible to use this for establishing dense point-point map between two point clouds? Thank you!

    opened by Sentient07 1
  • Masks of certain objects are missing?

    Masks of certain objects are missing?

    Thank you for publishing this fantastic dataset!

    The paper said that all dynamic objects that are found in certain frames are given unique instance ids and their masks are annotated. But it seems that some object masks are missing, "train_masks.json" says: { "color": "train/seq000/color/000200.jpg", "depth": "train/seq000/depth/000200.png", "mask": "train/seq000/mask/000200.png", "seq_id": "seq000", "frame_id": "000200", "object_id": "blackdog" }, and { "color": "train/seq000/color/000200.jpg", "depth": "train/seq000/depth/000200.png", "mask": "train/seq000/mask/000200.png", "seq_id": "seq000", "frame_id": "000200", "object_id": "lady" }, , these two dictionaries refer to the same mask file while the file only illustrates the mask for the lady: image

    Is the mask for blackdog missing or I got something wrong here?

    opened by dwxrycb123 1
  • wrong loop range?

    wrong loop range?

    https://github.com/AljazBozic/DeepDeform/blob/6572f4e0df633dba342c1957b5cfe4b88985275d/evaluation/csrc/cpu/image_proc.cpp#L101 Is this line supposed to be for (int x = 1; x < width - 1; ++x) {

    opened by wenbin-lin 1
  • Dataset requirement

    Dataset requirement

    I want to test this project and need the dataset. When I answered all the question, I found no button can be clicked to submit. image Could you directly send me the data link?

    opened by ShiwenH 1
Owner
Aljaz Bozic
PhD Student at Visual Computing Group
Aljaz Bozic
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

null 117 Dec 28, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 274 Jan 5, 2023
A variational Bayesian method for similarity learning in non-rigid image registration (CVPR 2022)

A variational Bayesian method for similarity learning in non-rigid image registration We provide the source code and the trained models used in the re

daniel grzech 14 Nov 21, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 235 Dec 26, 2022
GPU Accelerated Non-rigid ICP for surface registration

GPU Accelerated Non-rigid ICP for surface registration Introduction Preivous Non-rigid ICP algorithm is usually implemented on CPU, and needs to solve

Haozhe Wu 144 Jan 4, 2023
Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Project looking into use of autoencoder for semi-supervised learning and comparing data requirements compared to supervised learning.

Tom-R.T.Kvalvaag 2 Dec 17, 2021
DSAC* for Visual Camera Re-Localization (RGB or RGB-D)

DSAC* for Visual Camera Re-Localization (RGB or RGB-D) Introduction Installation Data Structure Supported Datasets 7Scenes 12Scenes Cambridge Landmark

Visual Learning Lab 143 Dec 22, 2022
3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

3DMV 3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans. This work is based on our ECCV'18 p

Владислав Молодцов 0 Feb 6, 2022
Weakly Supervised Learning of Rigid 3D Scene Flow

Weakly Supervised Learning of Rigid 3D Scene Flow This repository provides code and data to train and evaluate a weakly supervised method for rigid 3D

Zan Gojcic 124 Dec 27, 2022
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Felix Wimbauer 494 Jan 6, 2023
Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image

NonCuboidRoom Paper Learning to Reconstruct 3D Non-Cuboid Room Layout from a Single RGB Image Cheng Yang*, Jia Zheng*, Xili Dai, Rui Tang, Yi Ma, Xiao

null 67 Dec 15, 2022
code for paper "Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning" by Zhongzheng Ren*, Raymond A. Yeh*, Alexander G. Schwing.

Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learning Overview This code is for paper: Not All Unlabeled Data are Equa

Jason Ren 22 Nov 23, 2022
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Google Research 80 Dec 25, 2022
Towards uncontrained hand-object reconstruction from RGB videos

Towards uncontrained hand-object reconstruction from RGB videos Yana Hasson, Gül Varol, Ivan Laptev and Cordelia Schmid Project page Paper Table of Co

Yana 69 Dec 27, 2022
OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

OcclusionFusion (CVPR'2022) Project Page | Paper | Video Overview This repository contains the code for the CVPR 2022 paper OcclusionFusion, where we

Wenbin Lin 193 Dec 15, 2022
UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

dddzg 49 Jan 2, 2023
Hybrid CenterNet - Hybrid-supervised object detection / Weakly semi-supervised object detection

Hybrid-Supervised Object Detection System Object detection system trained by hybrid-supervision/weakly semi-supervision (HSOD/WSSOD): This project is

null 5 Dec 10, 2022
A non-linear, non-parametric Machine Learning method capable of modeling complex datasets

Fast Symbolic Regression Symbolic Regression is a non-linear, non-parametric Machine Learning method capable of modeling complex data sets. fastsr aim

VAMSHI CHOWDARY 3 Jun 22, 2022
This is an implementation for the CVPR2020 paper "Learning Invariant Representation for Unsupervised Image Restoration"

Learning Invariant Representation for Unsupervised Image Restoration (CVPR 2020) Introduction This is an implementation for the paper "Learning Invari

GarField 88 Nov 7, 2022