Learning from Synthetic Shadows for Shadow Detection and Removal [Inoue+, IEEE TCSVT 2020].

Overview

Learning from Synthetic Shadows for Shadow Detection and Removal (IEEE TCSVT 2020)

Overview

This repo is for the paper "Learning from Synthetic Shadows for Shadow Detection and Removal". We present SynShadow, a novel large-scale synthetic shadow/shadow-free/matte image triplets dataset and pipeline to synthesize it. We further show how to use SynShadow for robust and efficient shadow detection and removal.

In this repo, we provide

  • SynShadow dataset: ./datasets
  • SP+M implementation: ./src
  • Trained models and results: below

If you find this code or dataset useful for your research, please cite our paper:

@article{inoue_2020_tcsvt,
  author = {Inoue, Naoto and Yamasaki, Toshihiko},
  title = {Learning from Synthetic Shadows for Shadow Detection and Removal},
  journal = {IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)},
  volume={XX},
  number={XX},
  pages={XXXX-XXXX},
  year={2020},
  publisher={IEEE}
}

Trained Models and Results

We provide the models for shadow detection and removal for convenience. Downloaded models should be placed under ./checkpoints.

Shadow Detection

ALl the results are in 480x640. BER is reported for 480x640 images. Below are results evaluated on ISTD test set. DSDNet++ is a modified variant of DSDNet.

Model Train BER
DSDNet++ SynShadow 2.74 results / weights
DSDNet++ SynShadow->ISTD 1.09 results / weights
BDRAR SynShadow 2.74 results / weights
BDRAR SynShadow->ISTD 1.10 results / weights

Shadow Removal

ALl the results are in 480x640. For the pre-trained weights, we only provide SP+M weights, since this repository has full implementation of it. RMSE is reported for 480x640 images.

Model: SP+M

Train Test RMSE
SynShadow ISTD+ 4.9 results / weights / precomputed_mask
SynShadow->ISTD+ ISTD+ 4.0 results / weights / precomputed_mask
SynShadow SRD+ 5.7 results / weights / precomputed_mask
SynShadow->SRD+ SRD+ 5.2 results / weights / precomputed_mask
SynShadow USR - results / weights / precomputed_mask

Model: DHAN

Train Test RMSE
SynShadow->ISTD+ ISTD+ 4.6 results
SynShadow->SRD+ SRD+ 6.6 results
SynShadow USR - results

Note: we have accidentially removed some files and cannot provide some results.

Comments
  • about how to  Generate shadows?

    about how to Generate shadows?

    Hello, I would like to ask. I want to match the pictures that have not been taught. Can this project be realized? Do you have any recommended open source projects to realize the generation?

    opened by zhanghongyong123456 7
  • Shadow detect error

    Shadow detect error

    I have downloaded the he models for shadow detection, and put i into ./checkpoints. When I run: python test.py --dataset_mode demo --model detect --netG dsd --dataset_root ../Samples/ I get the following error: FileNotFoundError: [Errno 2] No such file or directory: '/SynShadow-main/src/models/n etworks/base/resnext/resnext_101_32x4d.pth'

    How to infer an image.

    opened by clannadcl 5
  • no DHAN training code

    no DHAN training code

    Hello, your work is very exciting, thank you for your contribution to this field. There is no training code for DHAN. I would like to know further details about DHAN under the setting of SynShadow->ISTD+. Can you provide the corresponding code? I saw the evaluation results in the paper. Thank you for your work and look forward to your reply.

    opened by nachifur 5
  • Script to generate synthesized image

    Script to generate synthesized image

    Hello, Thank for sharing your code. I want to know if there is a script to generate Synthesized image(with shadow-free image, matte, attenuation parameter)!

    opened by jij7401 3
  • Fixed Dockerfile for allowing to build the application

    Fixed Dockerfile for allowing to build the application

    A fix was needed to proceed with the current application's Docker image build, as an error was shown when building the required wheels. After the corresponding update of the required packages, the Docker image build process is accomplished without any inconvenience.

    This repository is such a great effort; thanks for sharing it.

    opened by sisco0 2
  • Simple removal test on own image

    Simple removal test on own image

    Hello,

    I would like to test your model of shadow removal on a random image but I am a bit confused where to save the image as well as which command to type in.

    For now, I saved the image under datasets/test/, but when I run

    python test.py --dataset_root ../datasets/

    I get the following error:

    AssertionError: ../datasets/test/input is not a valid directory

    If I create an input directory, then the program is asking me for a target directory and a maskdirectory. Both should contain the same number of elements as the input directory.

    But if I test on a random image, I do not have any target or mask, so I do not know how to proceed. Could you please help me?

    Thanks! Guillaume

    opened by guist 2
  • training  error

    training error

    Thank you for sharing. I want to train spmnet on ISTD, but I got the error after some epochs:

    C:\anaconda\envs\taka1\lib\site-packages\numpy\lib\polynomial.py:628: RuntimeWarning: invalid value encountered in true_divide lhs /= scale

    Intel MKL ERROR: Parameter 4 was incorrect on entry to DGELSD. Traceback (most recent call last): File "train.py", line 31, in for i, data_i in enumerate(dataloader, start=iter_counter.epoch_iter): File "C:\anaconda\envs\taka1\lib\site-packages\torch\utils\data\dataloader.py", line 521, in next data = self._next_data() File "C:\anaconda\envs\taka1\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "C:\anaconda\envs\taka1\lib\site-packages\torch\utils\data_utils\fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\anaconda\envs\taka1\lib\site-packages\torch\utils\data_utils\fetch.py", line 49, in data = [self.dataset[idx] for idx in possibly_batched_index] File "C:\SynShadow\src\data\paired_dataset.py", line 82, in getitem return self.postprocess(input_dict) File "C:\SynShadow\src\data\base_dataset.py", line 49, in postprocess d['input'], d['target'], d['mask'].squeeze()) File "C:\SynShadow\src\util\illum_affine_model.py", line 98, in fit_brightening_params 1 File "<array_function internals>", line 6, in polyfit File "C:\anaconda\envs\taka1\lib\site-packages\numpy\lib\polynomial.py", line 629, in polyfit c, resids, rank, s = lstsq(lhs, rhs, rcond) File "<array_function internals>", line 6, in lstsq File "C:\anaconda\envs\taka1\lib\site-packages\numpy\linalg\linalg.py", line 2306, in lstsq x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj) File "C:\anaconda\envs\taka1\lib\site-packages\numpy\linalg\linalg.py", line 100, in _raise_linalgerror_lstsq raise LinAlgError("SVD did not converge in Linear Least Squares") numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares

    Thanks.
    I revise code in illum_affine_model.py line 100 except ValueError: -> except : Then, I didn't get error.


    On the other hand, when I trained spmnet on ISTD,w_l2 and b_l2 loss decreased fastly, but L1 loss didn't decrease. The results from this checkpoints are bad. I trained spmnet on SynShadow(USR/shadowfree). W_l2 and b_l2 loss decreased slowly, and L1 loss also decreased. The results are good.

    In addition,do you have some validation set to evaluate benchmark during training?

    opened by fcu-d0449079 1
  • About Synthetic shadow

    About Synthetic shadow

    Thanks for your great work. Could you please provide an complete instruction for synthetic dataset generation? A little confusing about the darken function. I don't find the definition of the parameter "opt.intercepts_mode".

    Thank you~

    opened by GuoLanqing 1
  • generating matte images

    generating matte images

    Hello, Thanks for sharing your work. Could you provide base body models and ShapeNetCore.v2 in shadow_matte_generator subdirectory. Some link is failure.

    opened by duxiangcheng 1
Owner
Naoto Inoue
Ph.D student in Computer Vision and Computer Graphics at Aizawa-Yamasaki-Matsui Labratory, UTokyo
Naoto Inoue
MPRNet-Cloud-removal: Progressive cloud removal

MPRNet-Cloud-removal Progressive cloud removal Requirements 1.Pytorch >= 1.0 2.Python 3 3.NVIDIA GPU + CUDA 9.0 4.Tensorboard Installation 1.Clone the

Semi 95 Dec 18, 2022
Code for the CVPR 2021 paper "Triple-cooperative Video Shadow Detection"

Triple-cooperative Video Shadow Detection Code and dataset for the CVPR 2021 paper "Triple-cooperative Video Shadow Detection"[arXiv link] [official l

Zhihao Chen 24 Oct 4, 2022
《Towards High Fidelity Face Relighting with Realistic Shadows》(CVPR 2021)

Towards High Fidelity Face-Relighting with Realistic Shadows Andrew Hou, Ze Zhang, Michel Sarkis, Ning Bi, Yiying Tong, Xiaoming Liu. In CVPR, 2021. T

null 114 Dec 10, 2022
Deep Learning for 3D Point Clouds: A Survey (IEEE TPAMI, 2020)

??Deep Learning for 3D Point Clouds (IEEE TPAMI, 2020)

Qingyong 1.4k Jan 8, 2023
Y. Zhang, Q. Yao, W. Dai, L. Chen. AutoSF: Searching Scoring Functions for Knowledge Graph Embedding. IEEE International Conference on Data Engineering (ICDE). 2020

AutoSF The code for our paper "AutoSF: Searching Scoring Functions for Knowledge Graph Embedding" and this paper has been accepted by ICDE2020. News:

AutoML Research 64 Dec 17, 2022
🔥RandLA-Net in Tensorflow (CVPR 2020, Oral & IEEE TPAMI 2021)

RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (CVPR 2020) This is the official implementation of RandLA-Net (CVPR2020, Oral

Qingyong 1k Dec 30, 2022
The first dataset on shadow generation for the foreground object in real-world scenes.

Object-Shadow-Generation-Dataset-DESOBA Object Shadow Generation is to deal with the shadow inconsistency between the foreground object and the backgr

BCMI 105 Dec 30, 2022
A Web API for automatic background removal using Deep Learning. App is made using Flask and deployed on Heroku.

Automatic_Background_Remover A Web API for automatic background removal using Deep Learning. App is made using Flask and deployed on Heroku. ?? https:

Gaurav 16 Oct 29, 2022
This source code is implemented using keras library based on "Automatic ocular artifacts removal in EEG using deep learning"

CSP_Deep_EEG This source code is implemented using keras library based on "Automatic ocular artifacts removal in EEG using deep learning" {https://www

Seyed Mahdi Roostaiyan 2 Nov 8, 2022
[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, CVPR 2021. Ayan Kumar Bhunia, Pinaki nath Chowdhury, Yongxin Yan

Ayan Kumar Bhunia 44 Dec 12, 2022
Official pytorch code for SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal

SSAT: A Symmetric Semantic-Aware Transformer Network for Makeup Transfer and Removal This is the official pytorch code for SSAT: A Symmetric Semantic-

ForeverPupil 57 Dec 13, 2022
Deep Semisupervised Multiview Learning With Increasing Views (IEEE TCYB 2021, PyTorch Code)

Deep Semisupervised Multiview Learning With Increasing Views (ISVN, IEEE TCYB) Peng Hu, Xi Peng, Hongyuan Zhu, Liangli Zhen, Jie Lin, Huaibai Yan, Dez

null 3 Nov 19, 2022
Code for CVPR2021 paper "Robust Reflection Removal with Reflection-free Flash-only Cues"

Robust Reflection Removal with Reflection-free Flash-only Cues (RFC) Paper | To be released: Project Page | Video | Data Tensorflow implementation for

Chenyang LEI 162 Jan 5, 2023
《Single Image Reflection Removal Beyond Linearity》(CVPR 2019)

Single-Image-Reflection-Removal-Beyond-Linearity Paper Single Image Reflection Removal Beyond Linearity. Qiang Wen, Yinjie Tan, Jing Qin, Wenxi Liu, G

Qiang Wen 51 Jun 24, 2022
Self-Learned Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence

In this paper, we address the problem of rain streaks removal in video by developing a self-learned rain streak removal method, which does not require any clean groundtruth images in the training process.

Yang Wenhan 44 Dec 6, 2022
【CVPR 2021, Variational Inference Framework, PyTorch】 From Rain Generation to Rain Removal

From Rain Generation to Rain Removal (CVPR2021) Hong Wang, Zongsheng Yue, Qi Xie, Qian Zhao, Yefeng Zheng, and Deyu Meng [PDF&&Supplementary Material]

Hong Wang 48 Nov 23, 2022
Official code for "Stereo Waterdrop Removal with Row-wise Dilated Attention (IROS2021)"

Stereo-Waterdrop-Removal-with-Row-wise-Dilated-Attention This repository includes official codes for "Stereo Waterdrop Removal with Row-wise Dilated A

null 29 Oct 1, 2022
Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Jiaxi Jiang 282 Jan 2, 2023