CR-Fill: Generative Image Inpainting with Auxiliary Contextual Reconstruction. ICCV 2021

Related tags

Deep Learning pytorch generative-adversarial-network gan generative-adversarial-networks image-inpainting inpainting image-completion iccv2021
Overview

crfill

Usage | Web App | | Paper | Supplementary Material | More results |

code for paper ``CR-Fill: Generative Image Inpainting with Auxiliary Contextual Reconstruction". This repo (including code and models) are for research purposes only.

Usage

Dependencies

  1. Download code
git clone --single-branch https://github.com/zengxianyu/crfill
git submodule init
git submodule update
  1. Download data and model
chmod +x download/*
./download/download_model.sh
./download/download_datal.sh
  1. Install dependencies:
conda env create -f environment.yml

or install these packages manually in a Python 3.6 enviroment:

pytorch=1.3.1, opencv=3.4.2, tqdm, torchvision, dill, matplotlib, opencv

Inference

./test.sh

These script will run the inpainting model on the samples I provided. Modify the options --image_dir, --mask_dir, --output_dir in test.sh to test on custom data.

Train

  1. Prepare training datasets and put them in ./datasets/ following the example ./datasets/places

  2. run the training script:

./train.sh

open the html files in ./output to visualize training

After the training is finished, the model files can be found in ./checkpoints/debugarr0

you may modify the training script to use different settings, e.g., batch size, hyperparameters

Finetune

For finetune on custom dataset based on my pretrained models, use the following command:

  1. download checkpoints
./download/download_pretrain.sh
  1. run the training script
./finetune.sh

you may change the options in finetune.sh to use different hyperparameters or your own dataset

Web APP

To use the web app, these additional packages are required:

flask, requests, pillow

./demo.sh

then open http://localhost:2334 in the browser to use the web app

Citing

@inproceedings{zeng2021generative,
  title={CR-Fill: Generative Image Inpainting with Auxiliary Contextual Reconstruction},
  author={Zeng, Yu and Lin, Zhe and Lu, Huchuan and Patel, Vishal M.},
  booktitle={Proceedings of the IEEE International Conference on Computer Vision},
  year={2021}
}

Acknowledgement

Comments
  • Changing masks generated in training data

    Changing masks generated in training data

    Hello, thank you for your interesting work. I was wondering where you generate masks for the training data images. Because I wanted to know if I could change the method of generation of these masks to make them not be generated randomly in any part of the training images.

    opened by AhmedHashish123 2
  • opt.load_baseg

    opt.load_baseg

    In the second train.py "update_part=fine" of train.sh, I got the error about opt.load_baseg in the util.py.

    image

    I thought there isn't the part for adding load_baseg arguments in the train_options, So, I add it as test_options.

    is it right for training?

    opened by CVStack 2
  • Can I test an image without a mask?

    Can I test an image without a mask?

    I mean, in real life, we just have a damaged picture. How can I inpaint it when I don't have a mask? Is there any algorithm that detects the mask automatically?

    opened by yc-cui 1
  • warnings of multigpu

    warnings of multigpu

    hello~ when I ran your model with 2 GPUs, each epoch was with such warnings: "UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. warnings.warn('Was asked to gather along dimension 0, but all"

    These warnings usually happened due to parallel training.

    I believe your codes had already been set well for multi GPU, cause your argument “--gpu 0,1 ” naturally enabled multiple running. Then, do I need to deal with such warnings? Or just ignore it?

    Would be grateful if you can help me！

    opened by mac744mail 1
  • Will you release the training script?

    Will you release the training script?

    hi, nice work for image inpainting!

    I only find the test.py, will you plan to release the training script? If yes, when will you release?

    Thanks in advance!

    opened by freesouls 1
  • Fix download error fish: No matches for wildcard

    Fix download error fish: No matches for wildcard 

    fish: No matches for wildcard https://maildluteducn-my.sharepoint.com/:u:/g/personal/zengyu_mail_dlut_edu_cn/EbyP4NnHSulOtsVGpft82a8BKkZEklxgUO-z396H6UkzWQ?download=1

    It will compatibility for bash as the same well

    opened by FavorMylikes 0
  • Fix download error `fish: No matches for wildcard`

    Fix download error `fish: No matches for wildcard` 

    fish: No matches for wildcard “https://maildluteducn-my.sharepoint.com/:u:/g/personal/zengyu_mail_dlut_edu_cn/EbyP4NnHSulOtsVGpft82a8BKkZEklxgUO-z396H6UkzWQ?download=1”.

    It will compatibility for bash as the same well

    opened by FavorMylikes 0
  • AttributeError: 'Namespace' object has no attribute 'load_baseg'

    AttributeError: 'Namespace' object has no attribute 'load_baseg'

    Traceback (most recent call last): File "/root/crfill/train.py", line 27, in trainer = create_trainer(opt) File "/root/crfill/trainers/init.py", line 25, in create_trainer instance = model(opt) File "/root/crfill/trainers/pix2pix_trainer.py", line 20, in init self.pix2pix_model = models.create_model(opt) File "/root/crfill/models/init.py", line 41, in create_model instance = model(opt) File "/root/crfill/models/inpaint_model.py", line 32, in init self.netG, self.netD = self.initialize_networks(opt) File "/root/crfill/models/inpaint_model.py", line 110, in initialize_networks netG = util.load_network(netG, 'G', opt.which_epoch, opt) File "/root/crfill/util/util.py", line 222, in load_network if opt.load_baseg: AttributeError: 'Namespace' object has no attribute 'load_baseg'

    在python train.py 的第二和第三阶段我都遇到了这个问题，请问该如何解决呢，期待您的回复

    opened by zborger 0
  • raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format

    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format

    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format( RuntimeError: Error(s) in loading state_dict for BaseConvGenerator: size mismatch for conv14.weight: copying a param with shape torch.Size([96, 96, 3, 3]) from checkpoint, the shape in current model is torch.Size([96, 48, 3, 3]). size mismatch for conv16.weight: copying a param with shape torch.Size([48, 48, 3, 3]) from checkpoint, the shape in current model is torch.Size([24, 24, 3, 3]). size mismatch for conv16.bias: copying a param with shape torch.Size([48]) from checkpoint, the shape in current model is torch.Size([24]). size mismatch for conv17.weight: copying a param with shape torch.Size([3, 24, 3, 3]) from checkpoint, the shape in current model is torch.Size([3, 12, 3, 3]).

    opened by manhnguyen92 3
  • RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory (checkDL at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/DynamicLibrary.cpp:21)

    RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory (checkDL at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/DynamicLibrary.cpp:21)

    When I run sh train.sh, I get the error: RuntimeError: Error in dlopen or dlsym: libcaffe2_nvrtc.so: cannot open shared object file: No such file or directory (checkDL at /opt/conda/conda-bld/pytorch_1573049304260/work/aten/src/ATen/DynamicLibrary.cpp:21). But I can not solve this error.

    opened by zhangyonle 2
  • what is

    what is "raw.png" stand for? it seems not the raw input data(without mask)

    when i'm trying to analysis the training process by check the plots in "./output". I found there is a pic called "raw.png". But it seems not the real raw pic for training. What this "raw.png" is?

    opened by wonder-hy 0
Owner
null
GitHub
[ICCV'2021] Image Inpainting via Conditional Texture and Structure Dual Generation

[ICCV'2021] Image Inpainting via Conditional Texture and Structure Dual Generation

Xiefan Guo 122 Dec 11, 2022
official Pytorch implementation of ICCV 2021 paper FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting.

FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting By Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu

null 77 Dec 27, 2022
Implementation of Common Image Evaluation Metrics by Sayed Nadim (sayednadim.github.io). The repo is built based on full reference image quality metrics such as L1, L2, PSNR, SSIM, LPIPS. and feature-level quality metrics such as FID, IS. It can be used for evaluating image denoising, colorization, inpainting, deraining, dehazing etc. where we have access to ground truth.

Image Quality Evaluation Metrics Implementation of some common full reference image quality metrics. The repo is built based on full reference image q

Sayed Nadim 10 Jan 1, 2023
Code for the paper Relation Prediction as an Auxiliary Training Objective for Improving Multi-Relational Graph Representations (AKBC 2021).

Relation Prediction as an Auxiliary Training Objective for Knowledge Base Completion This repo provides the code for the paper Relation Prediction as

Facebook Research 85 Jan 2, 2023
CVPR 2021: "Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE"

Diverse Structure Inpainting ArXiv | Papar | Supplementary Material | BibTex This repository is for the CVPR 2021 paper, "Generating Diverse Structure

null 152 Nov 4, 2022
[ACM MM 2021] Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

Diverse Image Inpainting with Bidirectional and Autoregressive Transformers Installation pip install -r requirements.txt Dataset Preparation Given the

Yingchen Yu 25 Nov 9, 2022
Given a 2D triangle mesh, we could randomly generate cloud points that fill in the triangle mesh

generate_cloud_points Given a 2D triangle mesh, we could randomly generate cloud points that fill in the triangle mesh. Run python disp_mesh.py Or you

Peng Yu 2 Dec 24, 2021
Bayesian Image Reconstruction using Deep Generative Models

Bayesian Image Reconstruction using Deep Generative Models R. Marinescu, D. Moyer, P. Golland For technical inquiries, please create a Github issue. F

Razvan Valentin Marinescu 51 Nov 23, 2022
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

null 117 Dec 28, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 274 Jan 5, 2023
HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021)

Code for HDR Video Reconstruction HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021) Guanying Chen, Cha

Guanying Chen 64 Nov 19, 2022
[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

Linyi Jin 89 Jan 5, 2023
Toward Realistic Single-View 3D Object Reconstruction with Unsupervised Learning from Multiple Images (ICCV 2021)

Table of Content Introduction Getting Started Datasets Installation Experiments Training & Testing Pretrained models Texture fine-tuning Demo Toward R

VinAI Research 42 Dec 5, 2022
Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models

Patch-Rotation(PatchRot) Patch Rotation: A Self-Supervised Auxiliary Task for Robustness and Accuracy of Supervised Models Submitted to Neurips2021 To

null 4 Jul 12, 2021
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Junjie Hu 13 Dec 10, 2022
MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks

MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks Introduction This repo contains the pytorch impl

Meta Research 38 Oct 10, 2022
Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

null 105 Dec 23, 2022
ICCV2021 - Mining Contextual Information Beyond Image for Semantic Segmentation

Introduction The official repository for "Mining Contextual Information Beyond Image for Semantic Segmentation". Our full code has been merged into ss

null 55 Nov 9, 2022
Official implementation of "SinIR: Efficient General Image Manipulation with Single Image Reconstruction" (ICML 2021)

SinIR (Official Implementation) Requirements To install requirements: pip install -r requirements.txt We used Python 3.7.4 and f-strings which are in

null 47 Oct 11, 2022