PyTorch implementation for SDEdit: Image Synthesis and Editing with Stochastic Differential Equations

Related tags

Deep Learning pytorch image-editing image-manipulation image-generation score-matching controllable-generation
Overview

SDEdit: Image Synthesis and Editing with Stochastic Differential Equations


Project | Paper | Colab

PyTorch implementation of SDEdit: Image Synthesis and Editing with Stochastic Differential Equations.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon

Stanford and CMU

Overview

The key intuition of SDEdit is to "hijack" the reverse stochastic process of SDE-based generative models, as illustrated in the figure below. Given an input image for editing, such as a stroke painting or an image with color strokes, we can add a suitable amount of noise to make its artifacts undetectable, while still preserving the overall structure of the image. We then initialize the reverse SDE with this noisy input, and simulate the reverse process to obtain a denoised image of high quality. The final output is realistic while resembling the overall image structure of the input.

Getting Started

The code will automatically download pretrained SDE (VP) PyTorch models on CelebA-HQ, LSUN bedroom, and LSUN church outdoor.

Data format

We save the image and the corresponding mask in an array format [image, mask], where "image" is the image with range [0,1] in the PyTorch tensor format, "mask" is the corresponding binary mask (also the PyTorch tensor format) specifying the editing region. We provide a few examples, and functions/process_data.py will automatically download the examples to the colab_demo folder.

Stroke-based image generation

Given an input stroke painting, our goal is to generate a realistic image that shares the same structure as the input painting. SDEdit can synthesize multiple diverse outputs for each input on LSUN bedroom, LSUN church and CelebA-HQ datasets.

To generate results on LSUN datasets, please run

python main.py --exp ./runs/ --config bedroom.yml --sample -i images --npy_name lsun_bedroom1 --sample_step 3 --t 500  --ni

python main.py --exp ./runs/ --config church.yml --sample -i images --npy_name lsun_church --sample_step 3 --t 500  --ni

Stroke-based image editing

Given an input image with user strokes, we want to manipulate a natural input image based on the user's edit. SDEdit can generate image edits that are both realistic and faithful (to the user edit), while avoid introducing undesired changes.

To perform stroke-based image editing, run 
python main.py --exp ./runs/  --config church.yml --sample -i images --npy_name lsun_edit --sample_step 3 --t 500  --ni

Additional results

References

If you find this repository useful for your research, please cite the following work.

@article{meng2021sdedit,
      title={SDEdit: Image Synthesis and Editing with Stochastic Differential Equations},
      author={Chenlin Meng and Yang Song and Jiaming Song and Jiajun Wu and Jun-Yan Zhu and Stefano Ermon},
      year={2021},
      journal={arXiv preprint arXiv:2108.01073},
}

This implementation is based on / inspired by:

Comments
  • TypeError in Colab demo

    TypeError in Colab demo

    All 3 blocks for image generation produce this error: TypeError: image_editing_denoising_step_flexible_mask() got an unexpected keyword argument 'return_pred_xstart'

    opened by wandrzej 5
  • Wrong repository for training the models?

    Wrong repository for training the models?

    HI @chenlin9

    First of all, congratulations on such a fantastic project. I had a doubt regarding the training models. As I understood, you employed a trained SDE model to perturb the target image into a noise image, and after that, you reverse the stochastic process to get the final image. In that case, we should use the repository of @yang-song https://github.com/yang-song/score_sde (this one for PyTorch implementation) to train SDE models instead of https://github.com/ermongroup/ddim right?

    Thank you so much.

    opened by alvarogonjim 2
  • Generate hd images

    Generate hd images

    Hey @chenlin9 and @junyanz, Awesome work! I have a question about training and generating high-resolution images (1024x1024).

    How can I train on FFHQ or CelebA-HQ. I have looked for the config file but couldn't find any config for training 1024x1024 resolution. All the config files are for images with sizes 256x256. I would like to train the model on my custom dataset to generate 1024x1024 images. Would it be possible to provide the config file for CelebA-HQ or can you elaborate on how can I train for such a dataset to generate 1024x1024?

    Thanks!

    opened by Rm1n90 1
  • How can I test Image compositing

    How can I test Image compositing

    Thanks for your share of wonderful experiments.

    I'm currently testing on re-generate of experiments on paper,

    everything goes well, but I can't find image compositing on our distribution, Is there any ways to do that?

    opened by bell-one 0
  • some questions

    some questions

    Hi, thanks for your greate works I have re-trained the model from ddim on lsun dataset, and the ckpt has saved on 95000.pth. However, when i use this ckpt in image_editing.py, i do not get some satisfied samples? Can you help me? @chenlin9 @yang-song @willieneis @junyanz @jiamings @KellyYutongHe image image

    opened by pokameng 0
  • How to change the step length or the total denoising steps (N), so that the image generation process could be faster?

    How to change the step length or the total denoising steps (N), so that the image generation process could be faster?

    Hi there, thank you for releasing the code!

    I tried several ways as below to change the step length (delta t) and the total denosing steps but none of them works:

    1. Changing the num_timesteps hyperparameter in the config file and changing thetotal_noise_levels accordingly.
    2. Changing the step length of the enumeration of i in the SDEditing demonstration function

    It seems the synthesized images remain noised after the hyperparameters are tuned. Is there a way to modify N safely? Or is it possible to accelerate the generation process by other means?

    Thank you!

    opened by yutxie 0
  • fix some minor bugs with the masks

    fix some minor bugs with the masks

    Just change

    x = x0 * a[i].sqrt() + e * (1.0 - a[i]).sqrt()

    to

    if i > 0:
    x = x0 * a[i - 1].sqrt() + e * (1.0 - a[i - 1]).sqrt()
else:
    x = x0

    Otherwise, the generated images have some small dots.

    opened by lmxyy 0
Owner
null
GitHub https://sde-image-editing.github.io/
Official code for Score-Based Generative Modeling through Stochastic Differential Equations

Score-Based Generative Modeling through Stochastic Differential Equations This repo contains the official implementation for the paper Score-Based Gen

Yang Song 818 Jan 6, 2023
Code for "Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations"

Infinitely Deep Bayesian Neural Networks with SDEs This library contains JAX and Pytorch implementations of neural ODEs and Bayesian layers for stocha

Winnie Xu 95 Nov 26, 2021
Python framework for Stochastic Differential Equations modeling

SDElearn: a Python package for SDE modeling This package implements functionalities for working with Stochastic Differential Equations models (SDEs fo

null 4 May 10, 2022
Leibniz is a python package which provide facilities to express learnable partial differential equations with PyTorch

Leibniz is a python package which provide facilities to express learnable partial differential equations with PyTorch

Beijing ColorfulClouds Technology Co.,Ltd. 16 Aug 7, 2022
Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary Differential Equations

ODE GAN (Prototype) in PyTorch Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary

Somshubra Majumdar 15 Feb 10, 2022
Deep learning library for solving differential equations and more

DeepXDE Voting on whether we should have a Slack channel for discussion. DeepXDE is a library for scientific machine learning. Use DeepXDE if you need

Lu Lu 1.4k Dec 29, 2022
Supplementary code for the paper "Meta-Solver for Neural Ordinary Differential Equations" https://arxiv.org/abs/2103.08561

Meta-Solver for Neural Ordinary Differential Equations Towards robust neural ODEs using parametrized solvers. Main idea Each Runge-Kutta (RK) solver w

Julia Gusak 25 Aug 12, 2021
Implementation of Stochastic Image-to-Video Synthesis using cINNs.

Stochastic Image-to-Video Synthesis using cINNs Official PyTorch implementation of Stochastic Image-to-Video Synthesis using cINNs accepted to CVPR202

CompVis Heidelberg 135 Dec 28, 2022
iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis

iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis Andreas Bl

CompVis Heidelberg 36 Dec 25, 2022
[CVPR 2021] Anycost GANs for Interactive Image Synthesis and Editing

Anycost GAN video | paper | website Anycost GANs for Interactive Image Synthesis and Editing Ji Lin, Richard Zhang, Frieder Ganz, Song Han, Jun-Yan Zh

MIT HAN Lab 726 Dec 28, 2022
A fast poisson image editing implementation that can utilize multi-core CPU or GPU to handle a high-resolution image input.

Poisson Image Editing - A Parallel Implementation Jiayi Weng (jiayiwen), Zixu Chen (zixuc) Poisson Image Editing is a technique that can fuse two imag

Jiayi Weng 110 Dec 27, 2022
PyTorch implementation for Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuous Sign Language Recognition.

Stochastic CSLR This is the PyTorch implementation for the ECCV 2020 paper: Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuou

Zhe Niu 28 Dec 19, 2022
PyTorch implementation of SCAFFOLD (Stochastic Controlled Averaging for Federated Learning, ICML 2020).

Scaffold-Federated-Learning PyTorch implementation of SCAFFOLD (Stochastic Controlled Averaging for Federated Learning, ICML 2020). Environment numpy=

KI 30 Dec 29, 2022
Official implementation for "Style Transformer for Image Inversion and Editing" (CVPR 2022)

Style Transformer for Image Inversion and Editing (CVPR2022) https://arxiv.org/abs/2203.07932 Existing GAN inversion methods fail to provide latent co

Xueqi Hu 153 Dec 2, 2022
Implementation for HFGI: High-Fidelity GAN Inversion for Image Attribute Editing

HFGI: High-Fidelity GAN Inversion for Image Attribute Editing High-Fidelity GAN Inversion for Image Attribute Editing Update: We released the inferenc

Tengfei Wang 371 Dec 30, 2022
Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks

Bayesian-Torch is a library of neural network layers and utilities extending the core of PyTorch to enable the user to perform stochastic variational inference in Bayesian deep neural networks. Bayesian-Torch is designed to be flexible and seamless in extending a deterministic deep neural network architecture to corresponding Bayesian form by simply replacing the deterministic layers with Bayesian layers.

Intel Labs 210 Jan 4, 2023
Using NumPy to solve the equations of fluid mechanics together with Finite Differences, explicit time stepping and Chorin's Projection methods

Computational Fluid Dynamics in Python Using NumPy to solve the equations of fluid mechanics ?? ?? ?? together with Finite Differences, explicit time

Felix Köhler 4 Nov 12, 2022
The official implementation of You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient.

You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient (paper) @misc{zhang2021compress,

null 46 Dec 7, 2022
Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

Storchastic is a PyTorch library for stochastic gradient estimation in Deep Learning

Emile van Krieken 140 Dec 30, 2022