We simulate traveling back in time with a modern camera to rephotograph famous historical subjects.

Overview

[SIGGRAPH Asia 2021] Time-Travel Rephotography

Open in Colab

[Project Website]

Many historical people were only ever captured by old, faded, black and white photos, that are distorted due to the limitations of early cameras and the passage of time. This paper simulates traveling back in time with a modern camera to rephotograph famous subjects. Unlike conventional image restoration filters which apply independent operations like denoising, colorization, and superresolution, we leverage the StyleGAN2 framework to project old photos into the space of modern high-resolution photos, achieving all of these effects in a unified framework. A unique challenge with this approach is retaining the identity and pose of the subject in the original photo, while discarding the many artifacts frequently seen in low-quality antique photos. Our comparisons to current state-of-the-art restoration filters show significant improvements and compelling results for a variety of important historical people.

Time-Travel Rephotography
Xuan Luo, Xuaner Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, and Steven M. Seitz
In SIGGRAPH Asia 2021.

Demo

We provide an easy-to-get-started demo using Google Colab! The Colab will allow you to try our method on the sample Abraham Lincoln photo or your own photos using Cloud GPUs on Google Colab.

Open in Colab

Or you can run our method on your own machine following the instructions below.

Prerequisite

  • Pull third-party packages.
    git submodule update --init --recursive
    
  • Install python packages.
    conda create --name rephotography python=3.8.5
    conda activate rephotography
    conda install pytorch==1.4.0 torchvision==0.5.0 cudatoolkit=10.1 -c pytorch
    pip install -r requirements.txt
    

Quick Start

Run our method on the example photo of Abraham Lincoln.

  • Download models:
    ./scripts/download_checkpoint.sh
    
  • Run:
    ./scripts/run.sh b "dataset/Abraham Lincoln_01.png" 0.75 
    
  • You can inspect the optimization process by
    tensorboard --logdir "log/Abraham Lincoln_01"
    
  • You can find your results as below.
    results/
      Abraham Lincoln_01/       # intermediate outputs for histogram matching and face parsing
      Abraham Lincoln_01_b.png  # the input after matching the histogram of the sibling image
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)-init.png        # the sibling image
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)-init.pt         # the sibing latent codes and initialized noise maps
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750).png             # the output result
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750).pt              # the final optimized latent codes and noise maps
      Abraham Lincoln_01-b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)-rand.png        # the result with the final latent codes but random noise maps
    
    

Run on Your Own Image

  • Crop and align the head regions of your images:

    python -m tools.data.align_images <input_raw_image_dir> <aligned_image_dir>
    
  • Run:

    ./scripts/run.sh <spectral_sensitivity> <input_image_path> <blur_radius>
    

    The spectral_sensitivity can be b (blue-sensitive), gb (orthochromatic), or g (panchromatic). You can roughly estimate the spectral_sensitivity of your photo as follows. Use the blue-sensitive model for photos before 1873, manually select between blue-sensitive and orthochromatic for images from 1873 to 1906 and among all models for photos taken afterwards.

    The blur_radius is the estimated gaussian blur radius in pixels if the input photot is resized to 1024x1024.

Historical Wiki Face Dataset

Path Size Description
Historical Wiki Face Dataset.zip 148 MB Images
spectral_sensitivity.json 6 KB Spectral sensitivity (b, gb, or g).
blur_radius.json 6 KB Blur radius in pixels

The jsons are dictionares that map input names to the corresponding spectral sensitivity or blur radius. Due to copyright constraints, Historical Wiki Face Dataset.zip contains all images in the Historical Wiki Face Dataset that were used in our user study except the photo of Mao Zedong. You can download it separately and crop it as above.

Citation

If you find our code useful, please consider citing our paper:

@article{Luo-Rephotography-2021,
  author    = {Luo, Xuan and Zhang, Xuaner and Yoo, Paul and Martin-Brualla, Ricardo and Lawrence, Jason and Seitz, Steven M.},
  title     = {Time-Travel Rephotography},
  journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2021)},
  publisher = {ACM New York, NY, USA},
  volume = {40},
  number = {6},
  articleno = {213},
  doi = {https://doi.org/10.1145/3478513.3480485},
  year = {2021},
  month = {12}
}

License

This work is licensed under MIT License. See LICENSE for details.

Codes for the StyleGAN2 model come from https://github.com/rosinality/stylegan2-pytorch.

Acknowledgments

We thank Nick Brandreth for capturing the dry plate photos. We thank Bo Zhang, Qingnan Fan, Roy Or-El, Aleksander Holynski and Keunhong Park for insightful advice.

Comments
  • Black Image Output

    Black Image Output

    The colab works perfectly fine but when I try to run the code locally, the final result is always a black image.

    Everything seems to be working as intended. The face, contextual, noise and etc all show up correctly and begin to process. You can even visualize this data on tensorboard. However after a few steps -- it's random.. sometime 10 steps sometimes 20 steps .. all the data gets changed to "nan" and as a result the output goes black.

    Not sure what is going wrong. Any help? Happens the same with any image and any model/checkpoint. Any help would be greatly appreciated. @roxanneluo

    opened by blessedcoolant 9
  • Colab execution issue

    Colab execution issue

    Thanks' very much for this code release.

    I get these errors running the Colab notebook at the end.

    `FileNotFoundError Traceback (most recent call last) in () 19 results_dir="results/" 20 )) ---> 21 main(args)

    5 frames /usr/local/lib/python3.7/dist-packages/torch/serialization.py in init(self, name, mode) 209 class _open_file(_opener): 210 def init(self, name, mode): --> 211 super(_open_file, self).init(open(name, mode)) 212 213 def exit(self, *args):

    FileNotFoundError: [Errno 2] No such file or directory: 'checkpoint/encoder/checkpoint_b.pt'`

    opened by ionultd 8
  • Permission denied

    Permission denied

    Hi, There's a problem on "Download models". with access

    Permission denied: https://drive.google.com/uc?id=1hWc2JLM58_PkwfLG23Q5IH3Ysj2Mo1nr Maybe you need to change permission over 'Anyone with the link'?

    opened by rasulhasancom 2
  • Colab Assertion Failed Issue

    Colab Assertion Failed Issue

    I've created my own copy of the colab project and have managed to successfully implement it up until the final stage. I get this error when running the program using the sample image of Lincoln:

    `/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:3635: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details. "See the documentation of nn.Upsample for details.".format(mode)

    python test.py /content/Time-Travel-Rephotography/results/Abraham Lincoln_01/input_sibling /content/Time-Travel-Rephotography/results/Abraham Lincoln_01/skin_mask/face_parsing

    0it [00:00, ?it/s]


    error Traceback (most recent call last)

    in () 19 results_dir="results/" 20 )) ---> 21 main(args)

    2 frames

    /content/Time-Travel-Rephotography/tools/match_histogram.py in main(args) 144 145 mask_A = cv2.resize(cv2.imread(args.ref_mask, 0), A.shape[:2][::-1], --> 146 interpolation=cv2.INTER_NEAREST) > 0 if args.ref_mask else None 147 mask_B = cv2.resize(cv2.imread(args.src_mask, 0), B.shape[:2][::-1], 148 interpolation=cv2.INTER_NEAREST) > 0 if args.src_mask else None

    error: OpenCV(4.1.2) /io/opencv/modules/imgproc/src/resize.cpp:3720: error: (-215:Assertion failed) !ssize.empty() in function 'resize''

    opened by malewpro 2
  • Optimization phase is unstable and returns nan loss values

    Optimization phase is unstable and returns nan loss values

    Hi

    First and foremost, thank you for making this awesome project public for us to play with, i greatly appreciate it.

    So I am running this project locally on an Nvidia ampere card using CUDA 11.3.

    Everything seems to work, but 9 out of 10 Pictures produce unstable optimization runs.

    Also some pictures need multiple runs, or they crash. So it is a big hit or miss, if a picture will process.

    By crash I mean the optimization loss values all become "nan". Example: Optimizing 64x64 face: nan; eye: nan; contextual: nan; color_transfer: nan; noise: nan: 100%| 750/750 [08:49<00:00, 1.42it/s]

    lossvalues lossvalues2

    As you can see in the above picture, the optimizing runs for a while, then the loss values turn into "nan's".

    What can I do to troubleshoot this?

    My environment: python 3.8.5 pytorch: 1.10.1 py3.8_cuda11.3_cudnn8_0 torchvision: 0.11.2 py38_cu113 cudatoolkit: 11.3.1

    opened by kogereneliteness 1
  • Docker image

    Docker image

    It would be great to provide a Docker file so people can run this code easily without having to install all of the AI frameworks that were used for training the model.

    opened by bertyhell 1
  • download_checkpoints.sh - Permission denied

    download_checkpoints.sh - Permission denied

    Greetings, when trying to download models, the Colab swears at all links that there is no access and it would be better to change the file access parameters.. How to solve this problem? Error

    opened by TarvosKorp 0
  • Colab demo no longer working

    Colab demo no longer working

    I connect successfully in Colab and run the first two actions, but on the third (download models) an error occurs which has been persistent for the past couple of weeks, making the demo of no use. Is there a way around this? 317584381_922312662088073_2520819194539939960_n

    opened by osprey1978 2
  • Colab Model Problem

    Colab Model Problem

    When i try to play Download models thing i get error.

    If the script above fails, which is largely due to exceeding download quota, try run the script below.

    !./scripts/download_checkpoints.sh

    it doesn't work too..

    opened by baboJ 1
  • What causes the

    What causes the " UnpicklingError: Invalid Load Key, '<' " error?

    I've just uploaded the missing files and integrated them to the colab project so that it is no longer necessary to download each file manually and upload them one by one, here is the project link: https://colab.research.google.com/drive/1j2_gdzgyW-VapI4GTG48DAxgeYhJUdgb

    Sadly, when I try to execute the last step I get this error:

    https://user-images.githubusercontent.com/115443146/203861085-8a90d6a5-5ca1-489d-b190-c48abfe9a4b7.mp4

    I hope there is a solution to this problem, I love this project, it would help me a lot in restoring old photos of my relatives.

    opened by lucport1298 0
  • No module named 'projector'

    No module named 'projector'

    Hi, i'm tryna to use the the colab demo but when i arrive at the "Run" part i get this error:

    ModuleNotFoundError Traceback (most recent call last)

    in 3 4 from argparse import Namespace ----> 5 from projector import ( 6 ProjectorArguments, 7 main,

    ModuleNotFoundError: No module named 'projector'

    opened by Jeil-dev 2
  • problem executing code

    problem executing code

    ### Are there solutions to make the code work? These are the errors that made the program not work:

    The autoreload extension is already loaded. To reload it, use: %reload_ext autoreload ------------ Parameters ------------- camera_lr: 0.01 ckpt: checkpoint/stylegan2-ffhq-config-f.pt coarse_min: 32 color_transfer: 10000000000.0 contextual: 0.1 cx_layers: ['relu3_4', 'relu2_2', 'relu1_2'] e4e_ckpt: checkpoint/e4e_ffhq_encode.pt e4e_size: 256 encoder_ckpt: checkpoint/encoder/checkpoint_b.pt encoder_name: b encoder_size: 256 eye: 0.1 gaussian: 0.75 generator_size: 1024 init_latent: None input: dataset/أنا1_01.png log_dir: log/ log_freq: 10 log_visual_freq: 1000 lr: 0.1 mix_layer_range: [10, 18] noise_ramp: 0.75 noise_regularize: 50000.0 noise_strength: 0.0 rand_seed: None recon_size: 256 results_dir: results/ spectral_sensitivity: b vgg: 1 vggface: 0.3 wplus_step: [250, 750]

    b-G0.75-init(10,18)-s256-vgg1-vggface0.3-eye0.1-color1.0e+10-cx0.1(relu3_4,relu2_2,relu1_2)-NR5.0e+04-lr0.1_0.01-c32-wp(250,750)

    FileNotFoundError Traceback (most recent call last) in () 19 results_dir="results/" 20 )) ---> 21 main(args)

    5 frames /content/Time-Travel-Rephotography/projector.py in main(args) 113 # initialize 114 with torch.no_grad(): --> 115 init = Initializer(args).to(device) 116 latent_init = init(imgs_orig) 117

    /content/Time-Travel-Rephotography/tools/initialize.py in init(self, args) 114 115 assert args.encoder_size is not None --> 116 self.color_encoder = create_color_encoder(args) 117 self.color_encoder.eval() 118 self.color_encoder_size = args.encoder_size

    /content/Time-Travel-Rephotography/tools/initialize.py in create_color_encoder(args) 73 def create_color_encoder(args: Namespace): 74 encoder = Encoder(1, args.encoder_size, 512) ---> 75 ckpt = torch.load(args.encoder_ckpt) 76 encoder.load_state_dict(ckpt["model"]) 77 return encoder

    /usr/local/lib/python3.7/dist-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args) 592 pickle_load_args['encoding'] = 'utf-8' 593 --> 594 with _open_file_like(f, 'rb') as opened_file: 595 if _is_zipfile(opened_file): 596 # The zipfile reader is going to advance the current file position.

    /usr/local/lib/python3.7/dist-packages/torch/serialization.py in _open_file_like(name_or_buffer, mode) 228 def _open_file_like(name_or_buffer, mode): 229 if _is_path(name_or_buffer): --> 230 return _open_file(name_or_buffer, mode) 231 else: 232 if 'w' in mode:

    /usr/local/lib/python3.7/dist-packages/torch/serialization.py in init(self, name, mode) 209 class _open_file(_opener): 210 def init(self, name, mode): --> 211 super(_open_file, self).init(open(name, mode)) 212 213 def exit(self, *args):

    FileNotFoundError: [Errno 2] No such file or directory: 'checkpoint/encoder/checkpoint_b.pt'

    opened by sekrin 2
Back to the Feature: Learning Robust Camera Localization from Pixels to Pose (CVPR 2021)

Back to the Feature with PixLoc We introduce PixLoc, a neural network for end-to-end learning of camera localization from an image and a 3D model via

Computer Vision and Geometry Lab 610 Jan 5, 2023
Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

郭飞 3.7k Jan 3, 2023
Camera-caps - Examine the camera capabilities for V4l2 cameras

camera-caps This is a graphical user interface over the v4l2-ctl command line to

Jetsonhacks 25 Dec 26, 2022
Make your master artistic punk avatar through machine learning world famous paintings.

Master-art-punk Make your master artistic punk avatar through machine learning world famous paintings. 通过机器学习世界名画制作属于你的大师级艺术朋克头像 Nowadays, NFT is beco

Philipjhc 53 Dec 27, 2022
Implementation of "GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings" in PyTorch

PyGAS: Auto-Scaling GNNs in PyG PyGAS is the practical realization of our G NN A uto S cale (GAS) framework, which scales arbitrary message-passing GN

Matthias Fey 139 Dec 25, 2022
This is a simple backtesting framework to help you test your crypto currency trading. It includes a way to download and store historical crypto data and to execute a trading strategy.

You can use this simple crypto backtesting script to ensure your trading strategy is successful Minimal setup required and works well with static TP a

Andrei 154 Sep 12, 2022
A two-stage U-Net for high-fidelity denoising of historical recordings

A two-stage U-Net for high-fidelity denoising of historical recordings Official repository of the paper (not submitted yet): E. Moliner and V. Välimäk

Eloi Moliner Juanpere 57 Jan 5, 2023
Accommodating supervised learning algorithms for the historical prices of the world's favorite cryptocurrency and boosting it through LightGBM.

Accommodating supervised learning algorithms for the historical prices of the world's favorite cryptocurrency and boosting it through LightGBM.

null 1 Nov 27, 2021
Learning to Simulate Dynamic Environments with GameGAN (CVPR 2020)

Learning to Simulate Dynamic Environments with GameGAN PyTorch code for GameGAN Learning to Simulate Dynamic Environments with GameGAN Seung Wook Kim,

null 199 Dec 26, 2022
Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectrum sensing.

Deep-Learning-based-Spectrum-Sensing Use MATLAB to simulate the signal and extract features. Use PyTorch to build and train deep network to do spectru

null 10 Dec 14, 2022
Manipulation OpenAI Gym environments to simulate robots at the STARS lab

Manipulator Learning This repository contains a set of manipulation environments that are compatible with OpenAI Gym and simulated in pybullet. In par

STARS Laboratory 5 Dec 8, 2022
Simulate genealogical trees and genomic sequence data using population genetic models

msprime msprime is a population genetics simulator based on tskit. Msprime can simulate random ancestral histories for a sample of individuals (consis

Tskit developers 150 Dec 14, 2022
duralava is a neural network which can simulate a lava lamp in an infinite loop.

duralava duralava is a neural network which can simulate a lava lamp in an infinite loop. Example This is not a real lava lamp but a "fake" one genera

Maximilian Bachl 87 Dec 20, 2022
Jetson Nano-based smart camera system that measures crowd face mask usage in real-time.

MaskCam MaskCam is a prototype reference design for a Jetson Nano-based smart camera system that measures crowd face mask usage in real-time, with all

BDTI 212 Dec 29, 2022
(CVPR 2021) Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds

BRNet Introduction This is a release of the code of our paper Back-tracing Representative Points for Voting-based 3D Object Detection in Point Clouds,

null 86 Oct 5, 2022
meProp: Sparsified Back Propagation for Accelerated Deep Learning

meProp The codes were used for the paper meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (ICML 2017) [pdf]

LancoPKU 107 Nov 18, 2022
meProp: Sparsified Back Propagation for Accelerated Deep Learning (ICML 2017)

meProp The codes were used for the paper meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting (ICML 2017) [pdf]

LancoPKU 107 Nov 18, 2022
Tweesent-back - Tweesent backend uses fastAPI as the web framework

TweeSent Backend Tweesent backend. This repo uses fastAPI as the web framework.

null 0 Mar 26, 2022
[CVPR 2022] Back To Reality: Weak-supervised 3D Object Detection with Shape-guided Label Enhancement

Back To Reality: Weak-supervised 3D Object Detection with Shape-guided Label Enhancement Announcement ?? We have not tested the code yet. We will fini

Xiuwei Xu 7 Oct 30, 2022