This repository contains a collection of resources and papers on Diffusion Models and Score-based Models.

If there are any missing valuable resources or papers or any materials related to diffusion model, please do not hesitate to create or pull request to issues. I am happy to reflect them.

Resources

Introductory Post

A Unified Approach to Variational Autoencoders and Stochastic Normalizing Flows via Markov Chains
Johannes Hertrich, Paul Hagemann, Gabriele Steidl
arXiv 2021. [Paper]
24 Nov 2021

Introduction to deep generative modeling: Diffusion-based Deep Generative Models
Jakub Tomczak
[Website]
30 Aug 2021

What are Diffusion Models?
Lilian Weng
2021. [Website]
11 Jul 2021

Diffusion Models as a kind of VAE
Angus Turner
[Website]
29 June 2021

Generative Modeling by Estimating Gradients of the Data Distribution
Yang Song
[Website]
5 May 2021

Papers

Image

Image Generation

Conditional Image Generation with Score-Based Diffusion Models
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, Christian Etmann
arXiv 2021. [Paper]
26 Nov 2021

Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
Sam Bond-Taylor¹, Peter Hessey¹, Hiroshi Sasaki, Toby P. Breckon, Chris G. Willcocks
arXiv 2021. [Paper] [Github]
24 Nov 2021

Diffusion Normalizing Flow
Qinsheng Zhang, Yongxin Chen
NeurIPS 2021. [Paper] [Github]
14 Oct 2021

Denoising Diffusion Gamma Models
Eliya Nachmani¹, Robin San Roman¹, Lior Wolf
arXiv 2021. [Paper]
10 Oct 2021

Score-based Generative Neural Networks for Large-Scale Optimal Transport
Max Daniels, Tyler Maunu, Paul Hand
arXiv 2021. [Paper]
7 Oct 2021

Score-Based Generative Classifiers
Roland S. Zimmermann, Lukas Schott, Yang Song, Benjamin A. Dunn, David A. Klindt
arXiv 2021. [Paper]
1 Oct 2021

Bilateral Denoising Diffusion Models
Max W. Y. Lam, Jun Wang, Rongjie Huang, Dan Su, Dong Yu
arXiv 2021. [Paper] [Project]
26 Aug 2021

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
Patrick Esser¹, Robin Rombach¹, Andreas Blattmann¹, Björn Ommer
NeurIPS 2021. [Paper] [Project]
19 Aug 2021

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, Sungroh Yoon
ICCV 2021 (Oral). [Paper] [Github]
6 Aug 2021

SDEdit: Image Synthesis and Editing with Stochastic Differential Equations
Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, Stefano Ermon
arXiv 2021. [Paper] [Project] [Github]
2 Aug 2021

Score-Based Point Cloud Denoising
Shitong Luo, Wei Hu
arXiv 2021. [Paper] [Github]
23 Jul 2021

Structured Denoising Diffusion Models in Discrete State-Spaces
Jacob Austin¹, Daniel D. Johnson¹, Jonathan Ho, Daniel Tarlow, Rianne van den Berg
arXiv 2021. [Paper]
7 Jul 2021

Variational Diffusion Models
Diederik P. Kingma¹, Tim Salimans¹, Ben Poole, Jonathan Ho
arXiv 2021. [Paper] [Github]
1 Jul 2021

Deep Generative Learning via Schrödinger Bridge
Gefei Wang, Yuling Jiao, Qian Xu, Yang Wang, Can Yang
ICML 2021. [Paper]
19 Jun 2021

Non Gaussian Denoising Diffusion Models
Eliya Nachmani¹, Robin San Roman¹, Lior Wolf
arXiv 2021. [Paper] [Project]
14 Jun 2021

D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
Abhishek Sinha¹, Jiaming Song¹, Chenlin Meng, Stefano Ermon
arXiv 2021. [Paper] [Project] [Github]
12 Jun 2021

Score-based Generative Modeling in Latent Space
Arash Vahdat¹, Karsten Kreis¹, Jan Kautz
arXiv 2021. [Paper]
10 Jun 2021

Learning to Efficiently Sample from Diffusion Probabilistic Models
Daniel Watson, Jonathan Ho, Mohammad Norouzi, William Chan
arXiv 2021. [Paper]
7 Jun 2021

A Variational Perspective on Diffusion-Based Generative Models and Score Matching
Chin-Wei Huang, Jae Hyun Lim, Aaron Courville
ICML Workshop 2021. [Paper] [Github]
5 Jun 2021

Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
Valentin De Bortoli, James Thornton, Jeremy Heng, Arnaud Doucet
arXiv 2021. [Paper] [Project] [Github]
1 Jun 2021

On Fast Sampling of Diffusion Probabilistic Models
Zhifeng Kong, Wei Ping
ICML Workshop 2021. [Paper] [Github]
31 May 2021

Cascaded Diffusion Models for High Fidelity Image Generation
Jonathan Ho¹, Chitwan Saharia¹, William Chan, David J. Fleet, Mohammad Norouzi, Tim Salimans
arXiv 2021. [Paper] [Project]
30 May 2021

Gotta Go Fast When Generating Data with Score-Based Models
Alexia Jolicoeur-Martineau, Ke Li, Rémi Piché-Taillefer, Tal Kachman, Ioannis Mitliagkas
arXiv 2021. [Paper] [Github]
28 May 2021

Diffusion Models Beat GANs on Image Synthesis
Prafulla Dhariwal¹, Alex Nichol¹
arXiv 2021. [Paper] [Github]
11 May 2021

Image Super-Resolution via Iterative Refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, Mohammad Norouzi
arXiv 2021. [Paper] [Project] [Github]
15 Apr 2021

Noise Estimation for Generative Diffusion Models
Robin San-Roman¹, Eliya Nachmani¹, Lior Wolf
arXiv 2021. [Paper]
6 Apr 2021

Diffusion Probabilistic Models for 3D Point Cloud Generation
Shitong Luo, Wei Hu
CVPR 2021. [Paper] [Github]
2 Mar 2021

Improved Denoising Diffusion Probabilistic Models
Alex Nichol¹, Prafulla Dhariwal¹
ICLR 2021. [Paper] [Github]
18 Feb 2021

Maximum Likelihood Training of Score-Based Diffusion Models
Yang Song¹, Conor Durkan¹, Iain Murray, Stefano Ermon
arXiv 2021. [Paper]
22 Jan 2021

Learning Energy-Based Models by Diffusion Recovery Likelihood
Ruiqi Gao, Yang Song, Ben Poole, Ying Nian Wu, Diederik P. Kingma
ICLR 2021. [Paper] [Github]
15 Dec 2020

Score-Based Generative Modeling through Stochastic Differential Equations
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, Ben Poole
ICLR 2021 (Oral). [Paper] [Github]
26 Nov 2020

Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
Fan Bao, Kun Xu, Chongxuan Li, Lanqing Hong, Jun Zhu, Bo Zhang
ICML 2021. [Paper]
16 Oct 2020

Denoising Diffusion Implicit Models
Jiaming Song, Chenlin Meng, Stefano Ermon
ICLR 2021. [Paper] [Github]
6 Oct 2020

Adversarial score matching and improved sampling for image generation
Alexia Jolicoeur-Martineau¹, Rémi Piché-Taillefer¹, Rémi Tachet des Combes, Ioannis Mitliagkas
ICLR 2021. [Paper] [Github]
11 Sep 2020

Denoising Diffusion Probabilistic Models
Jonathan Ho, Ajay Jain, Pieter Abbeel
NeurIPS 2020. [Paper] [Github] [Github2]
19 Jun 2020

Improved Techniques for Training Score-Based Generative Models
Yang Song, Stefano Ermon
NeurIPS 2020. [Paper] [Github]
16 Jun 2020

Generative Modeling by Estimating Gradients of the Data Distribution
Yang Song, Stefano Ermon
NeurIPS 2019. [Paper] [Project] [Github]
12 Jul 2019

Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
Belinda Tzen, Maxim Raginsky
arXiv 2019. [Paper]
23 May 2019

Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli
ICML 2015. [Paper] [Github]
2 Mar 2015

A Connection Between Score Matching and Denoising Autoencoders
Pascal Vincent
Neural Computation 2011. [Paper]
7 Jul 2011

Bayesian Learning via Stochastic Gradient Langevin Dynamics
Max Welling, Yee Whye Teh
ICML 2011. [Paper] [Github]
28 June 2011

Image-to-Image Translation

Conditional Image Generation with Score-Based Diffusion Models
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, Christian Etmann
arXiv 2021. [Paper]
26 Nov 2021

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, Sungroh Yoon
ICCV 2021 (Oral). [Paper] [Github]
6 Aug 2021

UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
Hiroshi Sasaki, Chris G. Willcocks, Toby P. Breckon
arXiv 2021. [Paper]
12 Apr 2021

Image Editing

Conditional Image Generation with Score-Based Diffusion Models
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, Christian Etmann
arXiv 2021. [Paper] 26 Nov 2021

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, Sungroh Yoon
ICCV 2021 (Oral). [Paper] [Github]
6 Aug 2021

Super Resolution

Conditional Image Generation with Score-Based Diffusion Models
Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, Christian Etmann
arXiv 2021. [Paper]
26 Nov 2021

S3RP: Self-Supervised Super-Resolution and Prediction for Advection-Diffusion Process
Chulin Wang, Kyongmin Yeo, Xiao Jin, Andres Codas, Levente J. Klein, Bruce Elmegreen
arXiv 2021. [Paper]
8 Nov 2021

Autoregressive Diffusion Models
Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, Tim Salimans
arXiv 2021. [Paper]
5 Oct 2021

ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, Sungroh Yoon
ICCV 2021 (Oral). [Paper] [Github]
6 Aug 2021

SRDiff: Single Image Super-Resolution with Diffusion Probabilistic Models
Haoying Li, Yifan Yang, Meng Chang, Huajun Feng, Zhihai Xu, Qi Li, Yueting Chen
arXiv 2021. [Paper]
30 Apr 2021

Image Super-Resolution via Iterative Refinement
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, Mohammad Norouzi
arXiv 2021. [Paper] [Project] [Github]
15 Apr 2021

Text-to-Image

Vector Quantized Diffusion Model for Text-to-Image Synthesis
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, Baining Guo
arXiv 2021. [Paper] [Github]
29 Nov 2021

Blended Diffusion for Text-driven Editing of Natural Images
Omri Avrahami, Dani Lischinski, Ohad Fried
arXiv 2021. [Paper] [Github]
29 Nov 2021

DiffusionCLIP: Text-guided Image Manipulation Using Diffusion Models
Gwanghyun Kim, Jong Chul Ye
arXiv 2021. [Paper]
6 Oct 2021

Adversarial Attack and Defense

Adversarial purification with Score-based generative models
Jongmin Yoon, Sung Ju Hwang, Juho Lee
ICML 2021. [Paper] [Github]
11 Jun 2021

Medical Imaging

Score-based diffusion models for accelerated MRI
Hyungjin Chung, Jong chul Ye
arXiv 2021. [Paper]
8 Oct 2021

Graph Generation

Permutation Invariant Graph Generation via Score-Based Generative Modeling
Chenhao Niu, Yang Song, Jiaming Song, Shengjia Zhao, Aditya Grover, Stefano Ermon
AISTATS 2021. [Paper] [Github]
2 Mar 2020

Audio

Audio Generation

Denoising Diffusion Gamma Models
Eliya Nachmani¹, Robin San Roman¹, Lior Wolf
arXiv 2021. [Paper]
10 Oct 2021

Variational Diffusion Models
Diederik P. Kingma, Tim Salimans, Ben Poole, Jonathan Ho
arXiv 2021. [Paper] [Github]
1 Jul 2021

CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis
Simon Rouard¹, Gaëtan Hadjeres¹
arXiv 2021. [Paper] [Project]
14 Jun 2021

PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior
Sang-gil Lee, Heeseung Kim, Chaehun Shin, Xu Tan, Chang Liu, Qi Meng, Tao Qin, Wei Chen, Sungroh Yoon, Tie-Yan Liu
arXiv 2021. [Paper] [Project]
11 Jun 2021

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Jinglin Liu¹, Chengxi Li¹, Yi Ren¹, Feiyang Chen, Peng Liu, Zhou Zhao
arXiv 2021. [Paper] [Project] [Github]
6 May 2021

Symbolic Music Generation with Diffusion Models
Gautam Mittal, Jesse Engel, Curtis Hawthorne, Ian Simon
arXiv 2021. [Paper] [Code]
30 Mar 2021

DiffWave with Continuous-time Variational Diffusion Models
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, Bryan Catanzaro
ICLR 2021 [Paper] [Project] [Github]
21 Sep 2020

DiffWave: A Versatile Diffusion Model for Audio Synthesis
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli
ICML 2021 (Oral) [Paper] [Github] [Github2]
21 Sep 2020

WaveGrad: Estimating Gradients for Waveform Generation
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, William Chan
ICLR 2021. [Paper] [Project] [Github]
2 Sep 2020

Audio Conversion

DiffSVC: A Diffusion Probabilistic Model for Singing Voice Conversion
Songxiang Liu¹, Yuewen Cao¹, Dan Su, Helen Meng
arXiv 2021. [Paper] [Github]
28 May 2021

Audio Enhancement

A Study on Speech Enhancement Based on Diffusion Probabilistic Model
Yen-Ju Lu¹, Yu Tsao¹, Shinji Watanabe
arXiv 2021. [Paper]
25 Jul 2021

Restoring degraded speech via a modified diffusion model
Jianwei Zhang, Suren Jayasuriya, Visar Berisha
Interspeech 2021. [Paper]
22 Apr 2021

NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling
Junhyeok Lee, Seungu Han
Interspeech 2021. [Paper] [Project] [Github]
6 Apr 2021

Text-to-Speech

Guided-TTS:Text-to-Speech with Untranscribed Speech
Heeseung Kim, Sungwon Kim, Sungroh Yoon
arXiv 2021. [Paper]
32 Nov 2021

EdiTTS: Score-based Editing for Controllable Text-to-Speech
Jaesung Tae¹, Hyeongju Kim¹, Taesu Kim
arXiv 2021. [Paper]
6 Oct 2021

WaveGrad 2: Iterative Refinement for Text-to-Speech Synthesis
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, Najim Dehak, William Chan
arXiv 2021. [Paper] [Project] [Github] [Github2]
17 Jun 2021

Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
Vadim Popov¹, Ivan Vovk¹, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov
ICML 2021. [Paper] [Project] [Github]
13 May 2021

DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism
Jinglin Liu¹, Chengxi Li¹, Yi Ren¹, Feiyang Chen, Peng Liu, Zhou Zhao
arXiv 2021. [Paper] [Project] [Github]
6 May 2021

Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, Nam Soo Kim
Interspeech 2021. [Paper]
3 Apr 2021

Miscellaneous

Data Imputation

CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation
Yusuke Tashiro, Jiaming Song, Yang Song, Stefano Ermon
arXiv 2021. [Paper]
7 Jul 2021

Handwriting Synthesis

Diffusion models for Handwriting Generation
Troy Luhman¹, Eric Luhman¹
arXiv 2020. [Paper] [Github]
13 Nov 2020

Natural Language Processing

Zero-Shot Translation using Diffusion Models
Eliya Nachmani¹, Shaked Dovrat¹
arXiv 2021. [Paper]
2 Nov 2021

Time-Series Forecasting

Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
Kashif Rasul, Calvin Seward, Ingmar Schuster, Roland Vollgraf
ICLR 2021. [Paper]
2 Feb 2021

Applications

Deep Diffusion Models for Robust Channel Estimation
Marius Arvinte, Jonathan I Tamir
arXiv 2021. [Paper] [Github]
16 Nov 2021

Deep diffusion-based forecasting of COVID-19 by incorporating network-level mobility information
Padmaksha Roy, Shailik Sarkar, Subhodip Biswas, Fanglan Chen, Zhiqian Chen, Naren Ramakrishnan, Chang-Tien Lu
arXiv 2021. [Paper]
9 Nov 2021

Crystal Diffusion Variational Autoencoder for Periodic Material Generation
Tian Xie¹, Xiang Fu¹, Octavian-Eugen Ganea¹, Regina Barzilay, Tommi Jaakkola
arXiv 2021. [Paper]
12 Oct 2021

typo & change the category of one paper

(1) In the table of content: 3D-view => 3D (2) To be more precise, the paper "LION: Latent Point Diffusion Models for 3D Shape Generation" should be listed under "Point Cloud"

opened by chenguolin 2
Another new paper

Hi, thanks for making this great repo to track the recent awesome works.

We have recently posted our paper to arXiv (https://arxiv.org/abs/2205.12952) with the webpage (https://tengfei-wang.github.io/PITI/index.html), which uses diffusion model for image-to-image translation. Could you please include this paper in the repo?

Thanks!

opened by zhangmozhe 2
Some papers to add

The Schrodinger bridge problem is a question from statistical physics to find the shortest path that evolves a dynamic system into another system. (I'm not familiar with the concepts, though) https://arxiv.org/abs/1608.05862

And here are some papers that relate this problem with the diffusion probabilistic model. https://arxiv.org/abs/2106.01357 http://proceedings.mlr.press/v139/wang21l/wang21l.pdf

opened by jmyoon1 2
Suggestion to add CVPR 2022 Diffusion Tutorial video link

CVPR 2022 Tutorial - Denoising Diffusion-based Generative Modeling: Foundations and Applications

Video link: https://www.youtube.com/watch?v=cS6JQpEY9cs

opened by Siddharth-Shrivastava7 1
New papers for diffusion models!

The first paper of diffusion models in instance segmentation: DiffusionInst: Diffusion Model for Instance Segmentation arxiv: https://arxiv.org/abs/2212.02773 code: https://github.com/chenhaoxing/DiffusionInst

Nice job!

opened by zhangxgu 1
New paper about scene graph-based diffusion model

Hello, thanks for your great repo, it is really awesome. Recently we published our new paper "Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training" on arXiv, which proposes a diffusion model that directly conditions on scene graphs to generate images. Could you please add this paper to the repo? Thanks a lot!

arxiv: https://arxiv.org/abs/2211.11138 GitHub: https://github.com/YangLing0818/SGDiff

opened by YangLing0818 1
New paper

Hi, thank you for your great repo. We have recently published our survey paper, "Diffusion Models for Medical Image Analysis: A Comprehensive Survey," to arXiv. Could you please add this paper to the repo? Thank you very much.

Arxiv: https://arxiv.org/abs/2211.07804 GitHub: https://github.com/amirhossein-kz/Awesome-Diffusion-Models-in-Medical-Imaging

opened by amirhossein-kz 1
New paper in language generaion

You may consider adding this paper: Composable Text Controls in Latent Space with ODEs https://arxiv.org/abs/2208.00638 And the code is released at https://github.com/guangyliu/LatentOps

opened by guangyliu 1
A paper not related to diffusion model

Deep diffusion-based forecasting of COVID-19 by incorporating network-level mobility information This paper seems not related to diffusion model, where "diffusion" means the spreading process of pandemic in this paper

opened by zhou-zl18 1
adding a lecture

lecture from Jascha Sohl-Dickstein, the author of the foundational paper Deep Unsupervised Learning using Nonequilibrium Thermodynamics. I found it quite helpful for my understanding.

I added a lectures section, if it makes sense to distinguish these in-depth videos.

opened by gabriben 1
Update README.md
DiffFace: Diffusion-based Face Swapping with Facial Guidance added,

project link of MIDMs: Matching Interleaved Diffusion Models for Exemplar-based Image Translation added.
opened by j0seo 0
add detextify

Adds Detextify a Python library to remove unwanted pseudo-text from images generated by your favorite generative AI models (Stable Diffusion, Midjourney, DALL·E)

opened by mihail911 0