This repository contains the code and models for the following paper.

Overview

DC-ShadowNet

Introduction

This is an implementation of the following paper DC-ShadowNet: Single-Image Hard and Soft Shadow Removal Using Unsupervised Domain-Classifier Guided Network. (ICCV'2021) Yeying Jin, Aashish Sharma and Robby T. Tan

Abstract

Shadow removal from a single image is generally still an open problem. Most existing learning-based methods use supervised learning and require a large number of paired images (shadow and corresponding non-shadow images) for training. A recent unsupervised method, Mask-ShadowGAN, addresses this limitation. However, it requires a binary mask to represent shadow regions, making it inapplicable to soft shadows. To address the problem, in this paper, we propose an unsupervised domain-classifier guided shadow removal network, DC-ShadowNet. Specifically, we propose to integrate a shadow/shadow-free domain classifier into a generator and its discriminator, enabling them to focus on shadow regions. To train our network, we introduce novel losses based on physics-based shadow-free chromaticity, shadow-robust perceptual features, and boundary smoothness. Moreover, we show that our unsupervised network can be used for test-time training that further improves the results. Our experiments show that all these novel components allow our method to handle soft shadows, and also to perform better on hard shadows both quantitatively and qualitatively than the existing state-of-the-art shadow removal methods.

Overview of the proposed method:

Datasets

  1. SRD (please download train and test from the authors). Extracted Shadow Masks in the SRD Dataset

  2. AISTD

  3. LRSS: Soft Shadow Dataset

  4. ISTD

  5. USR: Unpaired Shadow Removal Dataset

Shadow Removal Results:

  1. SDR Dataset DC-ShadowNet Results, All Results

  1. AISTD Dataset DC-ShadowNet Results, All Results

  2. LRSS Soft Shadow Dataset DC-ShadowNet Results, All Results

  3. ISTD Dataset DC-ShadowNet Results

  4. USR Dataset DC-ShadowNet Results

Evaluation

The default root mean squared error (RMSE) evaluation code used by all methods (including ours) actually computes mean absolute error (MAE).

  1. The faster version MAE evaluation code
  2. The original version MAE evaluation code

1.1 SRD Dataset, set the paths of the shadow removal result and the dataset in demo_srd_release.m and then run it.

Get the following Table 1 in the main paper on the SRD dataset (size: 256x256).

Method Training All Shadow Non-Shadow
DC-ShadowNet Unpaired 4.66 7.70 3.39
Mask-ShadowGAN Unpaired 6.40 11.46 4.29
DSC Paired 4.86 8.81 3.23
DeShadowNet Paired 5.11 3.57 8.82
Gong Prior 12.35 25.43 6.91
Input Image N/A 13.77 37.40 3.96

1.2 AISTD Dataset, set the paths of the shadow removal result and the dataset in demo_aistd_release.m and then run it.

Get the following Table 2 in the main paper on the AISTD dataset (size: 256x256).

Method Training All Shadow Non-Shadow
DC-ShadowNet Unpaired 4.6 10.3 3.5

1.3 LRSS Soft Shadow Dataset, set the paths of the shadow removal result and the dataset in demo_lrss_release.m and then run it.

Get the following Table 3 in the main paper on the LRSS dataset (size: 256x256).

Method Training All
DC-ShadowNet Unpaired 3.48
Input Image N/A 12.26

Pre-trained Model

  1. Download the pre-trained SRD model, put in results/SRD/model/

  2. Download the pre-trained AISTD model, put in results/AISTD/model/

  3. Download the pre-trained ISTD model, put in results/ISTD/model/

  4. Download the pre-trained USR model, put in results/USR/model/

Test

python main_test.py --dataset SRD --datasetpath YOURPATH --phase test

Results: results/SRD/iteration/outputB

Train

  1. Implement the papers On the removal of shadows from images (TPAMI,05) and Recovery of Chromaticity Image Free from Shadows via Illumination Invariance (ICCV,03)

Directory

  1. Download Datasets and run 1, get the Shadow-Free Chromaticity Maps after Illumination Compensation, and put them in the trainC folder, you should see the following directory structure.
${DC-ShadowNet-Hard-and-Soft-Shadow-Removal}
|-- dataset
    |-- SRD
      |-- trainA ## Shadow 
      |-- trainB ## Shadow-free 
      |-- trainC ## Shadow-Free Chromaticity Maps after Illumination Compensation
      |-- testA  ## Shadow 
      |-- testB  ## Shadow-free 
...
  1. python main.py --dataset SRD --phase train

Shadow-Robust Feature

Get the following Figure 5 in the main paper, VGG feature visualization code is in feature_release folder,

python test_VGGfeatures.py

Results: ./results_VGGfeatures/shadow_VGGfeatures/layernumber/imagenumber/visual_featurenumber_RMSE.jpg

Boundary Smoothness Loss

Get the following Figure 8 in the main paper, shadow boundary code is in boundary_smooth folder,

run getRTVdenMask.m

Results: input_softmask_boundary.jpg

Citation

Please kindly cite our paper if you are using our codes:

Comments
  • datasets preprocessing code seem not exist

    datasets preprocessing code seem not exist

    Hi! the paper was cool! thanks for you share! I want to test the model locally, but the code in project only find test model and dataset class, so the code like generate chromaticity maps function had not be release?

    Download Datasets and run 1. get the Shadow-Free Chromaticity Maps after Illumination Compensation, and put them in the trainC folder, you should see the following directory structure.

    opened by xzltc 8
  • evaluation

    evaluation

    Sorry to interrupt you, I want to ask you some questions. Is the RMSE index of the input image of the ISTD+ dataset in your paper measured with your matlab code? I tested different results, and I am confused about the evaluation indicators. I hope you can help me answer my doubts, thank you.

    opened by Halfpoint1 7
  • How to select the unpaired training data on ISTD and SRD, which contain paired shadow/shadow-free images?

    How to select the unpaired training data on ISTD and SRD, which contain paired shadow/shadow-free images?

    In order to construct unpaired data, for unpaired training on the ISTD train dataset, the shadow and shadow free images were randomly sampled? But such random sampling may yield paired data. ISTD have different shadow images for the same shadow-free image. Was I wrong about the way to sample the unpaired data

    opened by lxxcontinue 4
  • Can't reproduce the visualization results in the paper

    Can't reproduce the visualization results in the paper

    Can you tell me how to visualize the attention map in Figure 7? I visualized the attention map in column 3 with the released model, but I can not obtain the same visualization results in your paper.

    WeChat58ecc0dc23b3d916dbf5d7c5d31134e6
    opened by scarleatt 3
  • I am confused about the quantitative results in table1(RMSE on SRD).

    I am confused about the quantitative results in table1(RMSE on SRD).

    Your work is novel. Thank you for your great work! I am confused by the quantitative results in Table 1, they seem to be much better than the results in the original paper. How are these results obtained? image I am very much looking forward to your reply. Thanks again for your work.

    opened by nachifur 2
  • data processing

    data processing

    hi, thank you for your great work! I don't know how to get the Shadow-Free Chromaticity Maps, could you give me a hand? my e-mail is [email protected]

    opened by CV-deeplearning 2
  • running the code

    running the code

    hello,i found your research quit interessting and as am new to programming I coudn't figure out how to run your program properly , would it be possible to have more directions or a code implementation exemple on google colab please ?

    opened by IAGirl-dev 1
  • Can you send me the SRD test dataset?

    Can you send me the SRD test dataset?

    I found out that the SRD official link is not working. I tried to contact the author but failed. So, please, do you still have the relevant dataset? If you still have it, can you send me an email? My email : [email protected]

    opened by XiaMoooo 2
Owner
AuAgCu
Computer Vision/ Deep Learning
AuAgCu
This repository contains the code and models necessary to replicate the results of paper: How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective

Black-Box-Defense This repository contains the code and models necessary to replicate the results of our recent paper: How to Robustify Black-Box ML M

OPTML Group 2 Oct 5, 2022
This repository contains the code and models necessary to replicate the results of paper: How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective

Black-Box-Defense This repository contains the code and models necessary to replicate the results of our recent paper: How to Robustify Black-Box ML M

OPTML Group 2 Oct 5, 2022
Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

Denis Emelin 42 Nov 24, 2022
null 190 Jan 3, 2023
This repository contains the data and code for the paper "Diverse Text Generation via Variational Encoder-Decoder Models with Gaussian Process Priors" (SPNLP@ACL2022)

GP-VAE This repository provides datasets and code for preprocessing, training and testing models for the paper: Diverse Text Generation via Variationa

Wanyu Du 18 Dec 29, 2022
This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models are Pix2Pix, Pix2PixHD, CycleGAN and PointWise.

RGB2NIR_Experimental This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models

null 5 Jan 4, 2023
This repository contains the source code and data for reproducing results of Deep Continuous Clustering paper

Deep Continuous Clustering Introduction This is a Pytorch implementation of the DCC algorithms presented in the following paper (paper): Sohil Atul Sh

Sohil Shah 197 Nov 29, 2022
This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effects in Video."

Omnimatte in PyTorch This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effect

Erika Lu 728 Dec 28, 2022
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
ByteTrack with ReID module following the paradigm of FairMOT, tracking strategy is borrowed from FairMOT/JDE.

ByteTrack_ReID ByteTrack is the SOTA tracker in MOT benchmarks with strong detector YOLOX and a simple association strategy only based on motion infor

Han GuangXin 46 Dec 29, 2022
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 40 Dec 5, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

MultiModal-InfoMax This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Informa

Deep Cognition and Language Research (DeCLaRe) Lab 89 Dec 26, 2022
This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

null 1.1k Dec 30, 2022
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 697 Jan 6, 2023
This repository contains the code for "SBEVNet: End-to-End Deep Stereo Layout Estimation" paper by Divam Gupta, Wei Pu, Trenton Tabor, Jeff Schneider

SBEVNet: End-to-End Deep Stereo Layout Estimation This repository contains the code for "SBEVNet: End-to-End Deep Stereo Layout Estimation" paper by D

Divam Gupta 19 Dec 17, 2022
This GitHub repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.'

About Repository This repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.' About Code

Arun Verma 1 Nov 9, 2021
This repository contains the code for the paper "PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization"

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization News: [2020/05/04] Added EGL rendering option for training data g

Shunsuke Saito 1.5k Jan 3, 2023
This repository contains PyTorch models for SpecTr (Spectral Transformer).

SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation This repository contains PyTorch models for SpecTr (Spectral Transformer).

Boxiang Yun 45 Dec 13, 2022