The first dataset on shadow generation for the foreground object in real-world scenes.

Related tags

Deep Learning image-composition shadow-generation
Overview

Object-Shadow-Generation-Dataset-DESOBA

Object Shadow Generation is to deal with the shadow inconsistency between the foreground object and the background in a composite image, that is, generating shadow for the foreground object according to background information, to make the composite image more realistic.

Our dataset DESOBA is a synthesized dataset for Object Shadow Generation. We build our dataset on the basis of Shadow-OBject Association dataset SOBA, which collects real-world images in complex scenes and provides annotated masks for object-shadow pairs. Based on SOBA dataset, we remove all the shadows to construct our DEshadowed Shadow-OBject Association(DESOBA) dataset, which can be used for shadow generation task and other shadow-related tasks as well. We illustrate the process of our DESOBA dataset construction based on SOBA dataset in the figure below.

Illustration of DESOBA dataset construction: The green arrows illustrate the process of acquiring paired data for training and evaluation. Given a ground-truth target image Ig, we manually remove all shadows to produce a deshadowed image Id. Then, we randomly select a foreground object in Ig, and replace its shadow area with the counterpart in Id to synthesize a composite image Ic without foreground shadow. Ic and Ig form a pair of input composite image and ground-truth target image. The red arrow illustrates our shadow generation task. Given Ic and its foreground mask Mfo, we aim to generate the target image Ig with foreground shadow.

Our DESOBA dataset contains 840 training images with totally 2,999 object-shadow pairs and 160 test images with totally 624 object-shadow pairs. The DESOBA dataset is provided in Baidu Cloud (access code: sipx), or Google Drive.

Prerequisites

  • Python
  • Pytorch
  • PIL

Getting Started

Installation

  • Clone this repo:
git clone https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBA.git
cd Object-Shadow-Generation-Dataset-DESOBA

  • Download the DESOBA dataset.

  • We provide the code of obtaining training/testing tuples, each tuple contains foreground object mask, foreground shadow mask, background object mask, background shadow mask, shadow image, and synthetic composite image without foreground shadow mask. The dataloader is available in /data_processing/data/DesobaSyntheticImageGeneration_dataset.py, which can be used as dataloader in training phase or testing phase.

  • We also provide the code of visualization of training/testing tuple, run:

python Vis_Desoba_Dataset.py

Vis_Desoba_Dataset.py is available in /data_processing/.

  • We show some examples of training/testing tuples in below:

from left to right: synthetic composite image without foreground shadow, target image with foreground shadow, foreground object mask, foreground shadow mask, background object mask, and background shadow mask.

Bibtex

If you find this work is useful for your research, please cite our paper using the following BibTeX [arxiv]:

@article{hong2021shadow,
  title={Shadow Generation for Composite Image in Real-world Scenes},
  author={Hong, Yan and Niu, Li and Zhang, Jianfu and Zhang, Liqing},
  journal={arXiv preprint arXiv:2104.10338},
  year={2021}
}
Comments
  • What is the difference between new_mask and shadow_mask in the training process?

    What is the difference between new_mask and shadow_mask in the training process?

    Dear authors,

    Could you please let me know the difference between the new_mask and shadow_mask in the training process?

    --bg_shadow_path $bg_shadow_path\
--new_mask_path $new_mask_path\

    Thank you very much for your work!

    Best regards,

    Sky

    opened by mxz2013 2
  • Google drive link missing for CompositionShadowGeneration-100

    Google drive link missing for CompositionShadowGeneration-100

    Hello authors, Please share the google drive link for the 100 real composite images (CompositionShadowGeneration-100). https://github.com/bcmi/Object-Shadow-Generation-Dataset-DESOBA#visualization-results-on-real-composite-images Thank you.

    opened by albertmundu 2
  • 关于requirement.txt 文件输出格式是否有一些问题

    关于requirement.txt 文件输出格式是否有一些问题

    PyWavelets @ file:///tmp/build/80754af9/pywavelets_1601658315997/work imageio @ file:///tmp/build/80754af9/imageio_1594161405741/work ... 我安装时候出现 ERROR: Could not install packages due to an OSError: [Errno 2] 没有那个文件或目录: '/tmp/build/80754af9/dask-core_1602083700509/work'

    这些有特定版本要求吗

    opened by zhanghongyong123456 2
  • How do you calculate linear transformation parameters?

    How do you calculate linear transformation parameters?

    Hi, @ustcnewly .

    I read your paper and think that your work can bring the future direction to shadow generation research!

    I have a question about your method about

    Given paired images {Ic, Ig}, the ground-truth shadow parameter {w_{dark}, b_{dark}} for the foreground shadow can be easily calculated by using linear regression.

    How do you calculate a linear regression? Would you explain this question?

    Best regards.

    opened by UdonDa 2
  • EOFError：Ran out of input

    EOFError：Ran out of input

    Hello, sorry to bother you. When I run pix2pix_RealComposite. Sh, EOFError: Ran out of input appears. I hope you can tell me how to solve this problem, thank you!

    opened by xjsxjs 1
  • Explain lambda parameters

    Explain lambda parameters

    Hi, Please explain below parameters in SGRNet options, not able understand with the name.

    lambda_GAN: 0.01 lambda_GAN_mask: 0.5 lambda_I1: 10.0 lambda_L1: 0.0
    lambda_M1: 10.0
    lambda_P1: 1
    lambda_REF1: 1
    lambda_STN1: 1
    lambda_TV1: 1
    lambda_bd: 0.0 finetuning: False
    finetuning_dir: None
    finetuning_epoch: None
    finetuning_name: None

    opened by sumantha-NTS 1
  • Custom training

    Custom training

    Hi,

    Not able to train model with DESOBA dataset. Training script able to load selected options, but not able to create model file.

    Below is the log file of train script ----------------- Options --------------- adain_mask_generator: 0
    angle: 60
    batch_size: 8 [default: 1] beta1: 0.5
    bg_instance_path: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET//InstanceMask [default: None] bg_instance_path1: None
    bg_maskattention: 0
    bg_selfattention: 0
    bg_shadow_path: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET//ShadowMask [default: None] bg_shadow_path1: None
    checkpoints_dir: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/custom_model [default: ./checkpoints] conditionD: 0.0
    continue_train: True [default: False] dataroot: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET/ [default: None] dataroot1: None
    dataset_mode: shadowparam [default: aligned] direction: AtoB
    display_env: main
    display_freq: 100 [default: 4] display_id: 1
    display_ncols: 4
    display_port: 8002 [default: 8097] display_server: http://localhost/ [default: http://localhost:8097/] display_winsize: 256
    epoch: latest
    epoch_count: 10 [default: 1] fineSize: 256
    finetuning: False
    finetuning_dir: None
    finetuning_epoch: None
    finetuning_name: None
    gpu_ids: 0
    grid_size: 4
    illumination: 0.0
    init_gain: 0.02
    init_type: xavier
    input_nc: 3
    instance_path: None
    instance_path1: None
    isTrain: True [default: None] keep_ratio: False
    lambda_GAN: 0.1 [default: 0.0] lambda_GAN_mask: 0.0
    lambda_I1: 10.0 [default: 1] lambda_L1: 0.0
    lambda_M1: 10.0 [default: 1] lambda_P1: 1
    lambda_REF1: 1
    lambda_STN1: 1
    lambda_TV1: 1
    lambda_bd: 0.0
    lambda_smooth: 0.0
    light_path: None
    light_path1: None
    light_vis_path: None
    light_vis_path1: None
    loadSize: 256 [default: 286] lr: 0.0002
    lr_D: 0.0002
    lr_decay_iters: 50
    lr_policy: lambda
    mask_test: None
    mask_train: None
    max_dataset_size: inf
    model: SGRNet [default: cycle_gan] n_layers_D: 3
    name: SGRNet_GRESNEXT18_C32_Dpixel_lrD0.0002 [default: experiment_name] ndf: 64
    netD: pixel [default: basic] netG: RESNEXT18 [default: resnet_9blocks] new_mask_path: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET//shadownewmask [default: None] new_mask_path1: None
    ngf: 32 [default: 64] niter: 100
    niter_decay: 300 [default: 100] no_crop: True [default: False] no_dropout: False
    no_flip: True [default: False] no_html: False
    no_lsgan: True
    no_rotate: True [default: False] norm: instance [default: batch] norm_mean: [0.5, 0.5, 0.5]
    norm_std: [0.5, 0.5, 0.5]
    num_threads: 4
    output_nc: 3
    param_path: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET//SOBA_params [default: None] param_path1: None
    phase: train
    pool_size: 0
    print_freq: 100
    randomSize: False
    redark_path: None
    redark_path1: None
    residual: 0.0
    resize_or_crop: resize_and_crop
    save_epoch_freq: 1 [default: 5] save_latest_freq: 5000
    serial_batches: False
    shadowfree_path: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET//DeshadowedImage [default: None] shadowfree_path1: None
    shadowimg_path: /content/drive/MyDrive/POCs/MaskRCNN_POC/shadow_extract/Generate_Shadow_script/Object-Shadow-Generation-Dataset-DESOBA/DESOBA_DATASET//ShadowImage [default: None] shadowimg_path1: None
    shadowmask_path: None
    shadowmask_path1: None
    span_range: 0.9
    stn_model_type: unbounded_stn
    suffix:
    update_html_freq: 10000
    use_our_mask: False
    verbose: True [default: False] warp_N: 1
    wdataroot: None
    ----------------- End ------------------- ^C

    opened by sumantha-NTS 1
  • OSError: [Errno 12] Cannot allocate memory

    OSError: [Errno 12] Cannot allocate memory

    When I run sh SGRNet_train.sh，the other settings are default, it outputed the error msg OSError: [Errno 12] Cannot allocate memory.

    Does it mean I need to use the smaller loadSize or fineSize?

    image

    loadSize=256
fineSize=256

    I tried to use the smaller loadsize and finesize, it works now.

    loadSize=128
fineSize=128

    I also monitored the memory cost of the two settings.

    My free memory is 47G, when I use 128 & 128, the left free memory is 33G, which means it costs 14G. And When I use 256 & 256, the left free memory will down to 0, and output the cannot allocate memory message, which means this setting will cost at least 47G memory.

    opened by qiaojy19 1
  • Expect more training details

    Expect more training details

    Thanks for sharing the dataset and code. The test results I obtained by trained SGRNet from scratch were different from the results of the paper. The GRMSE is about 5.5 and LRMSE is about 71 on BOS test set. Are there more training details.

    opened by lingtianxia123 1
  • test problems

    test problems

    Hi thanks for you job, i test the image, but the hue of the test results is much different with src img cd Object-Shadow-Generation-Dataset-DESOBA/src/script && sh SGRNet_RealComposite_2.sh

    thanks for your replay

    opened by YIFanH 1
Owner
BCMI
Center for Brain-Like Computing and Machine Intelligence, Shanghai Jiao Tong University.
BCMI
GitHub
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 44 Dec 27, 2022
Learning from Synthetic Shadows for Shadow Detection and Removal [Inoue+, IEEE TCSVT 2020].

Learning from Synthetic Shadows for Shadow Detection and Removal (IEEE TCSVT 2020) Overview This repo is for the paper "Learning from Synthetic Shadow

Naoto Inoue 67 Dec 28, 2022
We have implemented shaDow-GNN as a general and powerful pipeline for graph representation learning. For more details, please find our paper titled Deep Graph Neural Networks with Shallow Subgraph Samplers, available on arXiv (https//arxiv.org/abs/2012.01380).

Deep GNN, Shallow Sampling Hanqing Zeng, Muhan Zhang, Yinglong Xia, Ajitesh Srivastava, Andrey Malevich, Rajgopal Kannan, Viktor Prasanna, Long Jin, R

Facebook Research 117 Dec 20, 2022
Code for the CVPR 2021 paper "Triple-cooperative Video Shadow Detection"

Triple-cooperative Video Shadow Detection Code and dataset for the CVPR 2021 paper "Triple-cooperative Video Shadow Detection"[arXiv link] [official l

Zhihao Chen 24 Oct 4, 2022
We propose a new method for effective shadow removal by regarding it as an exposure fusion problem.

Auto-exposure fusion for single-image shadow removal We propose a new method for effective shadow removal by regarding it as an exposure fusion proble

Qing Guo 146 Dec 31, 2022
HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021)

Code for HDR Video Reconstruction HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021) Guanying Chen, Cha

Guanying Chen 64 Nov 19, 2022
[AAAI-2021] Visual Boundary Knowledge Translation for Foreground Segmentation

Trans-Net Code for (Visual Boundary Knowledge Translation for Foreground Segmentation, AAAI2021). [https://ojs.aaai.org/index.php/AAAI/article/view/16

ZJU-VIPA 2 Mar 4, 2022
The first dataset of composite images with rationality score indicating whether the object placement in a composite image is reasonable.

Object-Placement-Assessment-Dataset-OPA Object-Placement-Assessment (OPA) is to verify whether a composite image is plausible in terms of the object p

BCMI 53 Nov 15, 2022
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
This is the unofficial code of Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes. which achieve state-of-the-art trade-off between accuracy and speed on cityscapes and camvid, without using inference acceleration and extra data

Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes Introduction This is the unofficial code of Deep Dual-re

chenjun 113 Dec 23, 2022
A modular, primitive-first, python-first PyTorch library for Reinforcement Learning.

TorchRL Disclaimer This library is not officially released yet and is subject to change. The features are available before an official release so that

Meta Research 860 Jan 7, 2023
Efficient 6-DoF Grasp Generation in Cluttered Scenes

Contact-GraspNet Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, Dieter

NVIDIA Research Projects 148 Dec 28, 2022
Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

Nader Akoury 27 Dec 20, 2022
Object tracking and object detection is applied to track golf puts in real time and display stats/games.

Putting_Game Object tracking and object detection is applied to track golf puts in real time and display stats/games. Works best with the Perfect Prac

Max 1 Dec 29, 2021
NAS-HPO-Bench-II is the first benchmark dataset for joint optimization of CNN and training HPs.

NAS-HPO-Bench-II API Overview NAS-HPO-Bench-II is the first benchmark dataset for joint optimization of CNN and training HPs. It helps a fair and low-

yoichi hirose 8 Nov 21, 2022
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Jan 1, 2023
Real-world Anomaly Detection in Surveillance Videos- pytorch Re-implementation

Real world Anomaly Detection in Surveillance Videos : Pytorch RE-Implementation This repository is a re-implementation of "Real-world Anomaly Detectio

seominseok 62 Dec 8, 2022
Learning Generative Models of Textured 3D Meshes from Real-World Images, ICCV 2021

Learning Generative Models of Textured 3D Meshes from Real-World Images This is the reference implementation of "Learning Generative Models of Texture

Dario Pavllo 115 Jan 7, 2023
Repo for FUZE project. I will also publish some Linux kernel LPE exploits for various real world kernel vulnerabilities here. the samples are uploaded for education purposes for red and blue teams.

Linux_kernel_exploits Some Linux kernel exploits for various real world kernel vulnerabilities here. More exploits are yet to come. This repo contains

Wei Wu 472 Dec 21, 2022