This is a Tensorflow implementation of Learning to See in the Dark in CVPR 2018

Overview

Learning-to-See-in-the-Dark

This is a Tensorflow implementation of Learning to See in the Dark in CVPR 2018, by Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.

Project Website
Paper

teaser

This code includes the default model for training and testing on the See-in-the-Dark (SID) dataset.

Demo Video

https://youtu.be/qWKUFK7MWvg

Setup

Requirement

Required python (version 2.7) libraries: Tensorflow (>=1.1) + Scipy + Numpy + Rawpy.

Tested in Ubuntu + Intel i7 CPU + Nvidia Titan X (Pascal) with Cuda (>=8.0) and CuDNN (>=5.0). CPU mode should also work with minor changes but not tested.

Dataset

Update Aug, 2018: We found some misalignment with the ground-truth for image 10034, 10045, 10172. Please remove those images for quantitative results, but they still can be used for qualitative evaluations.

You can download it directly from Google drive for the Sony (25 GB) and Fuji (52 GB) sets.

There is download limit by Google drive in a fixed period of time. If you cannot download because of this, try these links: Sony (25 GB) and Fuji (52 GB).

New: we provide file parts in Baidu Drive now. After you download all the parts, you can combine them together by running: "cat SonyPart* > Sony.zip" and "cat FujiPart* > Fuji.zip".

The file lists are provided. In each row, there are a short-exposed image path, the corresponding long-exposed image path, camera ISO and F number. Note that multiple short-exposed images may correspond to the same long-exposed image.

The file name contains the image information. For example, in "10019_00_0.033s.RAF", the first digit "1" means it is from the test set ("0" for training set and "2" for validation set); "0019" is the image ID; the following "00" is the number in the sequence/burst; "0.033s" is the exposure time 1/30 seconds.

Testing

  1. Clone this repository.
  2. Download the pretrained models by running
python download_models.py
  1. Run "python test_Sony.py". This will generate results on the Sony test set.
  2. Run "python test_Fuji.py". This will generate results on the Fuji test set.

By default, the code takes the data in the "./dataset/Sony/" folder and "./dataset/Fuji/". If you save the dataset in other folders, please change the "input_dir" and "gt_dir" at the beginning of the code.

Training new models

  1. To train the Sony model, run "python train_Sony.py". The result and model will be save in "result_Sony" folder by default.
  2. To train the Fuji model, run "python train_Fuji.py". The result and model will be save in "result_Fuji" folder by default.

By default, the code takes the data in the "./dataset/Sony/" folder and "./dataset/Fuji/". If you save the dataset in other folders, please change the "input_dir" and "gt_dir" at the beginning of the code.

Loading the raw data and proccesing by Rawpy takes significant more time than the backpropagation. By default, the code will load all the groundtruth data processed by Rawpy into memory without 8-bit or 16-bit quantization. This requires at least 64 GB RAM for training the Sony model and 128 GB RAM for the Fuji model. If you need to train it on a machine with less RAM, you may need to revise the code and use the groundtruth data on the disk. We provide the 16-bit groundtruth images processed by Rawpy: Sony (12 GB) and Fuji (22 GB).

Citation

If you use our code and dataset for research, please cite our paper:

Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun, "Learning to See in the Dark", in CVPR, 2018.

License

MIT License.

FAQ

  1. Can I test my own data using the provided model?

The proposed method is designed for sensor raw data. The pretrained model probably not work for data from another camera sensor. We do not have support for other camera data. It also does not work for images after camera ISP, i.e., the JPG or PNG data.

  1. Will this be in any product?

This is a research project and a prototype to prove a concept.

  1. How can I train the model using my own raw data?

Generally, you just need to subtract the right black level and pack the data in the same way of Sony/Fuji data. If using rawpy, you need to read the black level instead of using 512 in the provided code. The data range may also differ if it is not 14 bits. You need to normalize it to [0,1] for the network input.

  1. Why the results are all black?

It is often because the pre-trained model not downloaded properly. After downloading, you should get 4 checkpoint related files for the model.

Questions

If you have additional questions after reading the FAQ, please email to [email protected].

Comments
  • ratio multiplication

    ratio multiplication

    @kkruups @prabodhw96 @cchen156 Hi, congrats for your work. In your paper and code, during training, input and corresponding ground truth are first scaled between [0-1], and then input are scaled by ratio but the ground truth is still between [0-1]. Have you tried not to scale input by ratio and keep input and gt both in range [0-1] to train the network? Will it get better result and what's your opinion? THX!

    opened by XhqGlorry11 11
  • About rgb training

    About rgb training

    paper has sRGB input comparision, I modify the last two level conv10(12 channel ) to output 3channel for the rgb input image generated by rawpy. the training loss is down to 0.025, the result is not bad, but some image is not real. is this correct way?

    opened by yutaka329 10
  • iphone 6s weights

    iphone 6s weights

    The official website contains a low light image and its result when run on SID model (http://web.engr.illinois.edu/~cchen156/SID/examples/6s.html). I could not find the weights for this in the repository. Is it possible to upload the weights for when trained on iPhone 6s?

    opened by allendoss 5
  • Could you provide training test data set within 2G.

    Could you provide training test data set within 2G.

    Hello,thank you for your help.Because my computer configuration is too low, RAM is only 4G, gpu memory 2G, although you provide sonny(12G) data set, but still a bit big for me and I can't download it .so could you provide training test data set within 2G. I just want to use a small amount of data to test the model. Thank you !

    opened by opticjian 5
  • How about training with 8-bit jpg image?

    How about training with 8-bit jpg image?

    I just tried to train Sony dataset after converting the raw data into jpg image and found the result quite terrible. So are there any tricks when training 8bit data? P.S.: The channel number of input is 12, which results from packing each channel of rgb into four.

    opened by ArsenLuca 5
  • Result of sony

    Result of sony

    I downloaded the sony data of 22 GB, the training and testing programs were executed, the folder named 'result_sony' is created, with some files in it, but those files are not executable and there were no images to be seen, can you please tell me that was result was expected once the program is executed completely.

    opened by AnukritiSinghh 4
  • discussion: could there be any correlation between this work and google night sight?

    discussion: could there be any correlation between this work and google night sight?

    google pixel 3 and 4's night sight appear to have very good results. is it possible that they got inspired by this paper to build their night sight mode? if no, how could they have done it and even improved it. on a further note, is it possible to deploy this model into a phone and do the calculations locally?

    opened by xXDarkCubeXx 3
  • The quantization

    The quantization

    Hi,

    Thanks for your novel work. I have some questions. I noticed that you restrict the value of ground truth image around 0 to 1, then force the model to generate output image which is also around 0 to 1. I am confused that why not just use the original ground truth image as the label and produce the output image directly without multiplying 255. Is there any difference between these two ways? Could you please help answer this question?

    Thanks,

    opened by yifanjiang19 3
  • Any hint for running on other camera raw (nikon,canon)

    Any hint for running on other camera raw (nikon,canon)

    Hi! I was able to run the examples for Sony raw data and the results are amazing. I would like to experiment with Canon and Nikon raws but with Canon's raws the model seems to never converge. I am able to run it but i don't get any output. I modified the test_Sony file changing the ARW extension in CR2 (Canon's raw). Any advice?

    Thank you!

    opened by DomagojKorais 3
  • Broken links in the README file

    Broken links in the README file

    Some important links in the README file are broken, for example a link to the paper shows 404. It seems particularly everything under http://cchen156.web.engr.illinois.edu/ is broken.

    opened by OlliV 2
  • Question about shutter settings

    Question about shutter settings

    Hi,

    Thanks for your great work. Since I didn't find any related information in the paper, I'd like to ask authors some questions about the Sony setting.

    1. The Sony camera A7s2 supports silent shutter, or electronic shutter. Was the setting always on when the dataset is collected? Or the setting was sometimes on? Or all images were captured with the mechanical shutter?

    2. Have you tried to estimate the black level or always used the black level embedded in the file (512)

    opened by r06922019 2
  • is it a bug?train_sony.py  line 145  in_files = glob.glob(input_dir + '%05d_00*.ARW' % train_id)

    is it a bug?train_sony.py line 145 in_files = glob.glob(input_dir + '%05d_00*.ARW' % train_id)

    why train data only use "00", such as 00001_00_0.1s.ARW, 00002_00_0.1s.ARW. other train data with "02", "03" not be used

    opened by xcnie 0
  • Loss is NaN

    Loss is NaN

    Ubuntu20.04/Python2.7/Tensorflow1.15

    Run script train_Sony.py, training starts but from the log it seems the Loss is not correct? Does anyone konw the issue?

    333 154 Loss=nan Time=0.040 333 155 Loss=nan Time=0.041 333 156 Loss=nan Time=0.043 333 157 Loss=nan Time=0.036 333 158 Loss=nan Time=0.042 333 159 Loss=nan Time=0.040 333 160 Loss=nan Time=0.036 333 161 Loss=nan Time=0.042 334 1 Loss=nan Time=0.044 334 2 Loss=nan Time=0.042 334 3 Loss=nan Time=0.036

    opened by Qiuxingzl 0
  • Dataset: how was the exposure time determined

    Dataset: how was the exposure time determined

    In SID dataset, for a particular scene there are majority of images which were taken using one exposure time. A minority of images are using different exposure times. Also the exposure time of the majority images vary among different scenes (1/30s or 0.04s or 0.1s).

    Would you mind sharing how to determine the exposure time based on the scene when you prepare the SID dataset? Thanks a lot!

    opened by hcwang95 0
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
3D ResNets for Action Recognition (CVPR 2018)

3D ResNets for Action Recognition Update (2020/4/13) We published a paper on arXiv. Hirokatsu Kataoka, Tenga Wakamiya, Kensho Hara, and Yutaka Satoh,

Kensho Hara 3.5k Jan 6, 2023
PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place

Mikaela Uy 294 Dec 12, 2022
A PyTorch Implementation of "Watch Your Step: Learning Node Embeddings via Graph Attention" (NeurIPS 2018).

Attention Walk ⠀⠀ A PyTorch Implementation of Watch Your Step: Learning Node Embeddings via Graph Attention (NIPS 2018). Abstract Graph embedding meth

Benedek Rozemberczki 303 Dec 9, 2022
Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

emmanuel 47 Nov 6, 2022
A DeepStack custom model for detecting common objects in dark/night images and videos.

DeepStack_ExDark This repository provides a custom DeepStack model that has been trained and can be used for creating a new object detection API for d

MOSES OLAFENWA 98 Dec 24, 2022
Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment This is a pytorch project for the paper Seeing Dynamic Scene i

DV Lab 21 Nov 28, 2022
Dark Finix: All in one hacking framework with almost 100 tools

Dark Finix - Hacking Framework. Dark Finix is a all in one hacking framework wit

Md. Nur habib 2 Feb 18, 2022
Python implementation of Wu et al (2018)'s registration fusion

reg-fusion Projection of a central sulcus probability map using the RF-ANTs approach (right hemisphere shown). This is a Python implementation of Wu e

Dan Gale 26 Nov 12, 2021
A PyTorch implementation of "Graph Classification Using Structural Attention" (KDD 2018).

GAM ⠀⠀ A PyTorch implementation of Graph Classification Using Structural Attention (KDD 2018). Abstract Graph classification is a problem with practic

Benedek Rozemberczki 259 Dec 5, 2022
A PyTorch Implementation of "SINE: Scalable Incomplete Network Embedding" (ICDM 2018).

Scalable Incomplete Network Embedding ⠀⠀ A PyTorch implementation of Scalable Incomplete Network Embedding (ICDM 2018). Abstract Attributed network em

Benedek Rozemberczki 69 Sep 22, 2022
A PyTorch implementation of "Signed Graph Convolutional Network" (ICDM 2018).

SGCN ⠀ A PyTorch implementation of Signed Graph Convolutional Network (ICDM 2018). Abstract Due to the fact much of today's data can be represented as

Benedek Rozemberczki 251 Nov 30, 2022
PyTorch implementation of Wide Residual Networks with 1-bit weights by McDonnell (ICLR 2018)

1-bit Wide ResNet PyTorch implementation of training 1-bit Wide ResNets from this paper: Training wide residual networks for deployment using a single

Sergey Zagoruyko 122 Dec 7, 2022
Official Keras Implementation for UNet++ in IEEE Transactions on Medical Imaging and DLMIA 2018

UNet++: A Nested U-Net Architecture for Medical Image Segmentation UNet++ is a new general purpose image segmentation architecture for more accurate i

Zongwei Zhou 1.8k Jan 7, 2023
An official TensorFlow implementation of “CLCC: Contrastive Learning for Color Constancy” accepted at CVPR 2021.

CLCC: Contrastive Learning for Color Constancy (CVPR 2021) Yi-Chen Lo*, Chia-Che Chang*, Hsuan-Chao Chiu, Yu-Hao Huang, Chia-Ping Chen, Yu-Lin Chang,

Yi-Chen (Howard) Lo 58 Dec 17, 2022
Facebook Research 605 Jan 2, 2023
Neural Magic Eye: Learning to See and Understand the Scene Behind an Autostereogram, arXiv:2012.15692.

Neural Magic Eye Preprint | Project Page | Colab Runtime Official PyTorch implementation of the preprint paper "NeuralMagicEye: Learning to See and Un

Zhengxia Zou 56 Jul 15, 2022
Learning to See by Looking at Noise

Learning to See by Looking at Noise This is the official implementation of Learning to See by Looking at Noise. In this work, we investigate a suite o

Manel Baradad Jurjo 82 Dec 24, 2022