Implements the training, testing and editing tools for "Pluralistic Image Completion"

Related tags

Deep Learning generative-adversarial-networks inpainting image-completion multimodal
Overview

Pluralistic Image Completion

ArXiv | Project Page | Online Demo | Video(demo)

This repository implements the training, testing and editing tools for "Pluralistic Image Completion" by Chuanxia Zheng, Tat-Jen Cham and Jianfei Cai at NTU. Given one masked image, the proposed Pluralistic model is able to generate multiple and diverse plausible results with various structure, color and texture.

Editing example

Example results

Example completion results of our method on images of face (CelebA), building (Paris), and natural scenes (Places2) with center masks (masks shown in gray). For each group, the masked input image is shown left, followed by sampled results from our model without any post-processing. The results are diverse and plusible.

More results on project page

Getting started

Installation

This code was tested with Pytoch 0.4.0, CUDA 9.0, Python 3.6 and Ubuntu 16.04

pip install visdom dominate
  • Clone this repo:
git clone https://github.com/lyndonzheng/Pluralistic
cd Pluralistic

Datasets

  • face dataset: 24183 training images and 2824 test images from CelebA and use the algorithm of Growing GANs to get the high-resolution CelebA-HQ dataset
  • building dataset: 14900 training images and 100 test images from Paris
  • natural scenery: original training and val images from Places2
  • object original training images from ImageNet.

Training

  • Train a model (default: random irregular and irregular holes):
python train.py --name celeba_random --img_file your_image_path
  • Set --mask_type in options/base_options.py for different training masks. --mask_file path is needed for external irregular mask, such as the irregular mask dataset provided by Liu et al. and Karim lskakov .
  • To view training results and loss plots, run python -m visdom.server and copy the URL http://localhost:8097.
  • Training models will be saved under the checkpoints folder.
  • The more training options can be found in options folder.

Testing

  • Test the model
python test.py  --name celeba_random --img_file your_image_path
  • Set --mask_type in options/base_options.py to test various masks. --mask_file path is needed for 3. external irregular mask,
  • The default results will be saved under the results folder. Set --results_dir for a new path to save the result.

Pretrained Models

Download the pre-trained models using the following links and put them undercheckpoints/ directory.

Our main novelty of this project is the multiple and diverse plausible results for one given masked image. The center_mask models are trained with images of resolution 256*256 with center holes 128x128, which have large diversity for the large missing information. The random_mask models are trained with random regular and irregular holes, which have different diversity for different mask sizes and image backgrounds.

GUI

Download the pre-trained models from Google drive and put them undercheckpoints/ directory.

  • Install the PyQt5 for GUI operation
pip install PyQt5

Basic usage is:

python -m visdom.server
python ui_main.py

The buttons in GUI:

  • Options: Select the model and corresponding dataset for editing.
  • Bush Width: Modify the width of bush for free_form mask.
  • draw/clear: Draw a free_form or rectangle mask for random_model. Clear all mask region for a new input.
  • load: Choose the image from the directory.
  • random: Random load the editing image from the datasets.
  • fill: Fill the holes ranges and show it on the right.
  • save: Save the inputs and outputs to the directory.
  • Original/Output: Switch to show the original or output image.

The steps are as follows:

1. Select a model from 'options'
2. Click the 'random' or 'load' button to get an input image.
3. If you choose a random model, click the 'draw/clear' button to input free_form mask.
4. If you choose a center model, the center mask has been given.
5. click 'fill' button to get multiple results.
6. click 'save' button to save the results.

Editing Example Results

  • Results (original, input, output) for object removing
  • Results (input, output) for face playing. When mask half or right face, the diversity will be small for the short+long term attention layer will copy information from other side. When mask top or down face, the diversity will be large.

Next

  • Free form mask for various Datasets
  • Higher resolution image completion

License


This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

This software is for educational and academic research purpose only. If you wish to obtain a commercial royalty bearing license to this software, please contact us at [email protected].

Citation

If you use this code for your research, please cite our paper.

@inproceedings{zheng2019pluralistic,
  title={Pluralistic Image Completion},
  author={Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={1438--1447},
  year={2019}
}

@article{zheng2021pluralistic,
  title={Pluralistic Free-From Image Completion},
  author={Zheng, Chuanxia and Cham, Tat-Jen and Cai, Jianfei},
  journal={International Journal of Computer Vision},
  pages={1--20},
  year={2021},
  publisher={Springer}
}
Comments
  • About using self-attention

    About using self-attention

    Hello, i also use self-attention layer in my model, and some results look like this, where there are strange textures in the face.Have u ever met this situation and any advice on dealing with this? THX!~ ps:trained on celeba_hq, rectangle masks.

    opened by cmyyy 6
  • GUI on CPU

    GUI on CPU

    Wonderful work. I've downloaded your pre-trained model. I'd like to run the GUI on my CPU since the server seemly could not open your GUI program on Command Line: qt.qpa.screen: QXcbConnection: Could not connect to display Could not connect to any X display.

    I tried to run 'python3 ui_main.py --gpu_ids -1', but there are lots of bugs: ConnectionRefusedError: [Errno 61] Connection refused requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8095): Max retries exceeded with url: /env/main (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x126ff4438>: Failed to establish a new connection: [Errno 61] Connection refused',)) ERROR:visdom:[Errno 61] Connection refused AttributeError: 'ui_model' object has no attribute 'model' [1] 87369 abort python3 ui_main.py --gpu_ids -1

    Please help me to solve it. Thank you!

    opened by yuanjunchai 5
  • fine-tune failure

    fine-tune failure

    Thank you for your work! I am trying to fine-tune on places365 on the basis of your trained model 'place2_random'. However, the inpainted result is not as good as yours. I set the relative training arguments as default, and I have trained the network for 177420 iterations with batch size of 128. What else should be taken into account? Thank you. The 1st pic is of your result, 2nd is of mine. lawn01

    lawn01

    opened by ghost 4
  • object removing in video as delogo

    object removing in video as delogo

    thank you share your project, The inpaint results are different every time as the same image. How can I get the same result? because i want to remove object in video. thank you very much

    opened by chen849157649 4
  • bad outputs when testing

    bad outputs when testing

    dear author,when i tried to test your model by images in celeba, i got this error: latest_net_G.pth G:\Pluralistic-Inpainting\model\checkpoints\celeba_random\latest_net_G.pth Pretrained network G has fewer layers; The following are not initialized: ['attn1', 'decoder0', 'decoder1', 'decoder2', 'decoder3', 'decoder4', 'generator', 'out1', 'out2', 'out3', 'out4'] why can't i load the pretrained model? i will appreciate it very much if you can help me.

    opened by shihaobai 4
  • How can we get the loss plot of testing and training

    How can we get the loss plot of testing and training

    Hello, @lyndonzheng! Thank you for your fantastic code. In my computer vision class, I will present your paper as part of my final project.

    When I ran the code on Colab, I only can see the loss_log.txt in the checkpoints local folder. How can we get the loss plot? thanks

    opened by twyunting 3
  • I just can't figure out how to put image datasets and mask datasets

    I just can't figure out how to put image datasets and mask datasets

    sorry, I just can not figure out where to store image and where to store irregular mask. And I see three pictures in every kind of datasets.So i directly start train without any movement.After half day and a night, I stop training beacuse i can not understand the generated data.

    At last, I donwloaded your pretrained model and put it to checkpoint, after gui launch successly, I draw mask and fill, result is strange ,like this: Screenshot from 2020-08-11 17-31-06

    Is it normal?

    opened by biexiangduo 3
  • Image amount of testing split on CelebA-HQ dataset

    Image amount of testing split on CelebA-HQ dataset

    Hello, sorry to bother you.

    I created the CelebA-HQ dataset by following the steps in this GitHub repository (https://github.com/nperraud/download-celebA-HQ), and I split the dataset into training, validation, and testing split as you described in #4 (label 0 as training, label 1 as validation, and label 2 as testing). Eventually, I got 24183/2993/2824 images for training/validation/testing split.

    The README.md file in this repository shows that your testing set contains 2796 images; however, I got different amount of testing images.

    Below is the list of my testing split. Could you please provide your testing images list or point out any possible mistake I might have made while generating CelebA-HQ dataset? Thank you very much!

    My list of testing split: test_split.txt

    opened by andrewkgs 3
  • color discrepancy

    color discrepancy

    I have tried your model. It's really amazing. Thank you for your code, demo and paper, thank you for your work!

    However, I found that the model usually suffers from color discrepancy (especially in Places2/ImageNet) even in simple cases.

    Do you have any idea why such problems happened? And how can we solve or improve it?

    Thanks!

    opened by zengyh1900 3
  • about hinge loss

    about hinge loss

    hi! I have a problem. If I use hinge loss, I see in Generator, it will produce negative adv loss, while others are positive. And the total loss is the sum of positive value and negative value . Can you explain it? Thanks a lot!

    opened by 2018hello 3
  • Meaning of 50 outputs

    Meaning of 50 outputs

    In options/test_options.py there is a parameter nsampling which refers to number of outputs generated for an image-mask pair. What is the logic behind these multiple outputs?

    opened by prerna-grg 3
  • About the training set of Places2 and its training parameters

    About the training set of Places2 and its training parameters

    Hello, thank you for the project, how was the pre-trained model of Places2_random you provided in Readme trained, which data sets in Places2 you selected, and how many iterations were performed during training？

    opened by haolin512900 0
  • About face dataset and training procedures

    About face dataset and training procedures

    @lyndonzheng Thank you for the amazing work! I would like to experiment with training your model and there are several questions I would like to ask:

    1. About the face dataset you mentioned with 24183 training images and 2824 test images, I cannot find that version of the dataset from the link you provided (http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html). The aligned version of "In-The-Wild Images" folder has 202,599 images and CelebAMask-HQ Dataset (https://github.com/switchablenorms/CelebAMask-HQ) has 30000 images. Would you please direct me to the image folder you were using?
    2. I read your paper which stated that the training time using CelebA dataset and centered mask is roughly 3 days using 1 GPU with batch size 20. May I ask how many training iterations does it take since I don't have a GPU with the same specification which means training days may change on my GPU.
    3. Is the rest of the training procedure the same as the default parameter values in the options folder? More specfically, does the pretrained weight come from sizing images to 266 by 266 and crop them to 256 by 256 with random flip/rotation when training?

    Thank you for your time and consideration!

    opened by SherryXTChen 2
  • about

    about

    Thanking you for sharing work~ https://github.com/lyndonzheng/Pluralistic-Inpainting/blob/1ca1855615fed8b686ca218c6494f455860f9996/model/network.py#L39 -L137 In probabilistic framework, don't consider Ic, just look at Im, In forward propagation, Im passes through the modules (block0 -> encoder i -> infer_prior i -> prior) in turn. What I am puzzled is that the latent code of Im is the output of 'encoder i' or 'infer_prior i' ? in other words， What is the role of infer_prior i? In addition, is prior module used to get the distribution?

    Looking forward to your reply~

    opened by liuliuliu11 1
Owner
Chuanxia Zheng
Chuanxia Zheng
GitHub
QueryFuzz implements a metamorphic testing approach to test Datalog engines.

Datalog is a popular query language with applications in several domains. Like any complex piece of software, Datalog engines may contain bugs. The mo

null 34 Sep 10, 2022
This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations at CVPR'21. According to some product reasons, we are not planning to release the training/testing codes and models. However, we will release the dataset and the scripts to prepare the dataset.

TransFill-Reference-Inpainting This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transf

Yuqian Zhou 80 Dec 8, 2022
A set of tools for creating and testing machine learning features, with a scikit-learn compatible API

Feature Forge This library provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, e

Machinalis 380 Nov 5, 2022
Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR

Image Restoration Toolbox (PyTorch). Training and testing codes for DPIR, USRNet, DnCNN, FFDNet, SRMD, DPSR, BSRGAN, SwinIR

Kai Zhang 2k Dec 31, 2022
A simple program for training and testing vit

Vit This is a simple program for training and testing vit. Key requirements: torch, torchvision and timm. Dataset I put 5 categories of the cub classi

xiezhenyu 2 Oct 11, 2022
CIFAR-10_train-test - training and testing codes for dataset CIFAR-10

CIFAR-10_train-test - training and testing codes for dataset CIFAR-10

Frederick Wang 3 Apr 26, 2022
Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

[ICLR 2022] Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity by Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu

VITA 18 Dec 31, 2022
Implements Stacked-RNN in numpy and torch with manual forward and backward functions

Recurrent Neural Networks Implements simple recurrent network and a stacked recurrent network in numpy and torch respectively. Both flavours implement

Vishal R 1 Nov 16, 2021
A python-image-classification web application project, written in Python and served through the Flask Microframework. This Project implements the VGG16 covolutional neural network, through Keras and Tensorflow wrappers, to make predictions on uploaded images.

Image Classification in Python Implementing image classification in Flask using Keras. The VGG16 is a convolution neural network model architecture th

Gerald Maduabuchi 19 Dec 12, 2022
Implements Gradient Centralization and allows it to use as a Python package in TensorFlow

Gradient Centralization TensorFlow This Python package implements Gradient Centralization in TensorFlow, a simple and effective optimization technique

Rishit Dagli 101 Nov 1, 2022
This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of Coordinate Independent Convolutional Networks.

Orientation independent Möbius CNNs This repository implements and evaluates convolutional networks on the Möbius strip as toy model instantiations of

Maurice Weiler 59 Dec 9, 2022
deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

deep-table implements various state-of-the-art deep learning and self-supervised learning algorithms for tabular data using PyTorch.

null 63 Oct 17, 2022
Some code of the implements of Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network

3D-GMPDCNN Geological Modeling Using 3D Pixel-Adaptive and Deformable Convolutional Neural Network PyTorch implementation of "Geological Modeling Usin

null 5 Nov 21, 2022
Model search is a framework that implements AutoML algorithms for model architecture search at scale

Model search (MS) is a framework that implements AutoML algorithms for model architecture search at scale. It aims to help researchers speed up their exploration process for finding the right model architecture for their classification problems (i.e., DNNs with different types of layers).

Google 3.2k Dec 31, 2022
Implements MLP-Mixer: An all-MLP Architecture for Vision.

MLP-Mixer-CIFAR10 This repository implements MLP-Mixer as proposed in MLP-Mixer: An all-MLP Architecture for Vision. The paper introduces an all MLP (

Sayak Paul 51 Jan 4, 2023
This framework implements the data poisoning method found in the paper Adversarial Examples Make Strong Poisons

Adversarial poison generation and evaluation. This framework implements the data poisoning method found in the paper Adversarial Examples Make Strong

null 31 Nov 1, 2022
Implements an infinite sum of poisson-weighted convolutions

An infinite sum of Poisson-weighted convolutions Kyle Cranmer, Aug 2018 If viewing on GitHub, this looks better with nbviewer: click here Consider a v

Kyle Cranmer 26 Dec 7, 2022
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
FastReID is a research platform that implements state-of-the-art re-identification algorithms.

FastReID is a research platform that implements state-of-the-art re-identification algorithms.

JDAI-CV 2.8k Jan 7, 2023