Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

Overview

Stylized Neural Painting

Open in RunwayML Badge

Preprint | Project Page | Colab Runtime 1 | Colab Runtime 2

Official PyTorch implementation of the preprint paper "Stylized Neural Painting", accepted to CVPR 2021.

We propose an image-to-painting translation method that generates vivid and realistic painting artworks with controllable styles. Different from previous image-to-image translation methods that formulate the translation as pixel-wise prediction, we deal with such an artistic creation process in a vectorized environment and produce a sequence of physically meaningful stroke parameters that can be further used for rendering. Since a typical vector render is not differentiable, we design a novel neural renderer which imitates the behavior of the vector renderer and then frame the stroke prediction as a parameter searching process that maximizes the similarity between the input and the rendering output. Experiments show that the paintings generated by our method have a high degree of fidelity in both global appearance and local textures. Our method can be also jointly optimized with neural style transfer that further transfers visual style from other images.

In this repository, we implement the complete training/inference pipeline of our paper based on Pytorch and provide several demos that can be used for reproducing the results reported in our paper. With the code, you can also try on your own data by following the instructions below.

The implementation of the sinkhorn loss in our code is partially adapted from the project SinkhornAutoDiff.

License

Creative Commons License Stylized Neural Painting by Zhengxia Zou is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

One-min video result

IMAGE ALT TEXT HERE

**Updates on CPU mode (Nov 29, 2020)

PyTorch-CPU mode is now supported! You can try out on your local machine without any GPU cards.

**Updates on lightweight renderers (Nov 26, 2020)

We have provided some lightweight renderers where users now can easily generate high resolution paintings with much more stroke details. With the lightweight renders, the rendering speed also improves a lot (x3 faster). This update also solves the out-of-memory problem when running our demo on a GPU card with limited memory (e.g. 4GB).

Please check out the following for more details.

Requirements

See Requirements.txt.

Setup

  1. Clone this repo:
git clone https://github.com/jiupinjia/stylized-neural-painting.git 
cd stylized-neural-painting
  1. Download one of the pretrained neural renderers from Google Drive (1. oil-paint brush, 2. watercolor ink, 3. marker pen, 4. color tapes), and unzip them to the repo directory.
unzip checkpoints_G_oilpaintbrush.zip
unzip checkpoints_G_rectangle.zip
unzip checkpoints_G_markerpen.zip
unzip checkpoints_G_watercolor.zip
  1. We have also provided some lightweight renderers where users can generate high-resolution paintings on their local machine with limited GPU memory. Please feel free to download and unzip them to your repo directory. (1. oil-paint brush (lightweight), 2. watercolor ink (lightweight), 3. marker pen (lightweight), 4. color tapes (lightweight)).
unzip checkpoints_G_oilpaintbrush_light.zip
unzip checkpoints_G_rectangle_light.zip
unzip checkpoints_G_markerpen_light.zip
unzip checkpoints_G_watercolor_light.zip

To produce our results

Photo to oil painting

  • Progressive rendering
python demo_prog.py --img_path ./test_images/apple.jpg --canvas_color 'white' --max_m_strokes 500 --max_divide 5 --renderer oilpaintbrush --renderer_checkpoint_dir checkpoints_G_oilpaintbrush --net_G zou-fusion-net
  • Progressive rendering with lightweight renderer (with lower GPU memory consumption and faster speed)
python demo_prog.py --img_path ./test_images/apple.jpg --canvas_color 'white' --max_m_strokes 500 --max_divide 5 --renderer oilpaintbrush --renderer_checkpoint_dir checkpoints_G_oilpaintbrush_light --net_G zou-fusion-net-light
  • Rendering directly from mxm image grids
python demo.py --img_path ./test_images/apple.jpg --canvas_color 'white' --max_m_strokes 500 --m_grid 5 --renderer oilpaintbrush --renderer_checkpoint_dir checkpoints_G_oilpaintbrush --net_G zou-fusion-net

Photo to marker-pen painting

  • Progressive rendering
python demo_prog.py --img_path ./test_images/diamond.jpg --canvas_color 'black' --max_m_strokes 500 --max_divide 5 --renderer markerpen --renderer_checkpoint_dir checkpoints_G_markerpen --net_G zou-fusion-net
  • Progressive rendering with lightweight renderer (with lower GPU memory consumption and faster speed)
python demo_prog.py --img_path ./test_images/diamond.jpg --canvas_color 'black' --max_m_strokes 500 --max_divide 5 --renderer markerpen --renderer_checkpoint_dir checkpoints_G_markerpen_light --net_G zou-fusion-net-light
  • Rendering directly from mxm image grids
python demo.py --img_path ./test_images/diamond.jpg --canvas_color 'black' --max_m_strokes 500 --m_grid 5 --renderer markerpen --renderer_checkpoint_dir checkpoints_G_markerpen --net_G zou-fusion-net

Style transfer

  • First, you need to generate painting and save stroke parameters to output dir
python demo.py --img_path ./test_images/sunflowers.jpg --canvas_color 'white' --max_m_strokes 500 --m_grid 5 --renderer oilpaintbrush --renderer_checkpoint_dir checkpoints_G_oilpaintbrush --net_G zou-fusion-net --output_dir ./output
  • Then, choose a style image and run style transfer on the generated stroke parameters
python demo_nst.py --renderer oilpaintbrush --vector_file ./output/sunflowers_strokes.npz --style_img_path ./style_images/fire.jpg --content_img_path ./test_images/sunflowers.jpg --canvas_color 'white' --net_G zou-fusion-net --renderer_checkpoint_dir checkpoints_G_oilpaintbrush --transfer_mode 1

You may also specify the --transfer_mode (0: transfer color only, 1: transfer both color and texture)

Also, please note that in the current version, the style transfer are not supported by the progressive rendering mode. We will be working on this feature in the near future.

Generate 8-bit graphic artworks

python demo_8bitart.py --img_path ./test_images/monalisa.jpg --canvas_color 'black' --max_m_strokes 300 --max_divide 4

Running through SSH

If you would like to run remotely through ssh and do not have something like X-display installed, you will need --disable_preview to turn off cv2.imshow on the run.

python demo_prog.py --disable_preview

Google Colab

Here we also provide a minimal working example of the inference runtime of our method. Check out the following runtimes and see your result on Colab.

Colab Runtime 1 : Image to painting translation (progressive rendering)

Colab Runtime 2 : Image to painting translation with image style transfer

To retrain your neural renderer

You can also choose a brush type and train the stroke renderer from scratch. The only thing to do is to run the following common. During the training, the ground truth strokes are generated on-the-fly, so you don't need to download any external dataset.

python train_imitator.py --renderer oilpaintbrush --net_G zou-fusion-net --checkpoint_dir ./checkpoints_G --vis_dir val_out --max_num_epochs 400 --lr 2e-4 --batch_size 64

Citation

If you use our code for your research, please cite the following paper:

@inproceedings{zou2020stylized,
    title={Stylized Neural Painting},
      author={Zhengxia Zou and Tianyang Shi and Shuang Qiu and Yi Yuan and Zhenwei Shi},
      year={2020},
      eprint={2011.08114},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
Comments
  • i can not use GPU

    i can not use GPU

    hello I tested this project on both win10 powershell and WSL, but it always uses my poor cpu, but I don't know how to check the cause of this problem. I tried to execute the following statement in python: "import torch" "print(torch.cuda.is_available())" It returns true in Powershell and false in WSL. Of course, I managed to execute the example, but it took quite a while.

    So I need some help, thank you. please forgive my terrible english.

    opened by z-spider 8
  • No GPU

    No GPU

    Hi,

    Thanks indeed for your work. I wanted to run a quick test on one of our virtual machines and got stuck as there is no GPU on the VM. Is there a way I can run it without a GPU?

    Many tanks in deed! DF

    opened by dfaghani 8
  • cannot to connect to x window error on google colab

    cannot to connect to x window error on google colab

    Great project!!!

    I'm trying to train my own strokes on google colab. However, when I try to run the demo:

    !python demo.py --img_path ./test_images/sunflowers.jpg --canvas_color 'white' --max_m_strokes 500 --m_grid 5 --renderer oilpaintbrush --renderer_checkpoint_dir checkpoints_G_oilpaintbrush --output_dir ./output

    I get this error: initialize network with normal loading renderer from pre-trained checkpoint... begin to draw... iteration step 0, G_loss: 0.00000, step_psnr: 4.52816, strokes: 25 / 500 : cannot connect to X server

    Guessing that it is some kind of image show function(s) somewhere...

    opened by terekita 7
  • Unable run demo on remote machine without desktop environment

    Unable run demo on remote machine without desktop environment

    Hi @jiupinjia, good job! the result of your repo seems fantastic! I try to run your demo on my remote GPU server which doesn't have a desktop environment(no x11 installed). and get the following error

    initialize network with normal
    loading renderer from pre-trained checkpoint...
    begin to draw...
    iteration step 0, G_loss: 0.00000, step_psnr: 3.38946, strokes: 25 / 500
    qt.qpa.xcb: could not connect to display 
    qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/shir/miniconda3/envs/tf1.15/lib/python3.6/site-packages/cv2/qt/plugins" even though it was found.
    This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.
    
    Available platform plugins are: xcb.
    
    Aborted (core dumped)
    

    Is there any way to solve this problem?

    opened by zerollzeng 7
  • Using Cpu instead of GPU?

    Using Cpu instead of GPU?

    Hi, when i run the program it uses 80 percent of my cpu and almost no gpu. Renders take forever too. Is there a way that i can check to see if it's using my 2080ti? Can i force it to use my gpu? Thanks.

    opened by giodude12 4
  • `AttributeError: 'ProgressivePainter' object has no attribute 'x'`

    `AttributeError: 'ProgressivePainter' object has no attribute 'x'`

    Hello Zhengxia,

    Thank you for creating this project, very interesting!

    I spent some time playing around with the colab runtime 1, I tried to increase the default hyper-parameters by multiplying 2, so far so good.

    However,

    # settings
    args.canvas_size = 1024 # size of the canvas for stroke rendering'
    args.max_m_strokes = 1000 # max number of strokes
    args.max_divide = 15 # divide an image up-to max_divide x max_divide patches
    ...
    

    I have updated the args.canvas_size, args.max_m_strokes by multiplying 2, while args.max_divide from 10 to 15, then in the "drawing" phase:

    pt = ProgressivePainter(args=args)
    final_rendered_image = optimize_x(pt)
    

    It throws error like this:

    ---------------------------------------------------------------------------
    AttributeError                            Traceback (most recent call last)
    <ipython-input-30-8f14c24057d5> in <module>()
          1 pt = ProgressivePainter(args=args)
    ----> 2 final_rendered_image = optimize_x(pt)
    
    <ipython-input-23-25a3da86e7fa> in optimize_x(pt)
         51                 pt.step_id += 1
         52 
    ---> 53         v = pt._normalize_strokes(pt.x)
         54         v = pt._shuffle_strokes_and_reshape(v)
         55         PARAMS = np.concatenate([PARAMS, v], axis=1)
    
    AttributeError: 'ProgressivePainter' object has no attribute 'x'
    

    Do you know why this error is thrown? Personally, I think maybe this is because that there should be some "math formula" behind these three (or more) parameters ?

    What do you think?

    Thanks!

    opened by Zhenye-Na 4
  • Error with Notebook

    Error with Notebook

    It throws the following error,

    initialize network with normal
    pre-trained renderer does not exist...
    ---------------------------------------------------------------------------
    NameError                                 Traceback (most recent call last)
    <ipython-input-11-a13d7e15e5e1> in <module>()
          1 pt = Painter(args=args)
    ----> 2 optimize_x(pt)
    
    1 frames
    /content/stylized-neural-painting/painter.py in _load_checkpoint(self)
         74         else:
         75             print('pre-trained renderer does not exist...')
    ---> 76             exit()
         77 
         78 
    
    NameError: name 'exit' is not defined
    
    opened by Zumbalamambo 4
  • Error:CUDA out of memory

    Error:CUDA out of memory

    This project is so cool and I want to run it on my PC, I downloaded the repo,configured and the error(CUDA out of memory) occurred.I change max_m_strokes ,max_divide two parameters, it works correctly, but the outpu is not meet my demand. I know nothing about the torch framework, can you help me to solve this problem, please ?

    opened by WShihan 4
  • Help runnig in CPU

    Help runnig in CPU

    (SNP2) C:\Users\Administrator\Nova pasta\SNP>python demo_prog.py --img_path ./test_images/1.jpg --canvas_color 'white' --max_m_strokes 500 --max_divide 5 --renderer oilpaintbrush --renderer_checkpoint_dir checkpoints_G_oilpaintbrush --net_G zou-fusion-net --disable_preview Traceback (most recent call last): File "demo_prog.py", line 4, in torch.cuda.current_device() File "C:\ProgramData\Anaconda3\envs\SNP2\lib\site-packages\torch\cuda_init_.py", line 366, in current_device lazy_init() File "C:\ProgramData\Anaconda3\envs\SNP2\lib\site-packages\torch\cuda_init.py", line 166, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled

    opened by sirciro 3
  • Error while rendering

    Error while rendering

    Hi,

    Your project really looks stunning and I want to try it out on google colab. This is the code i run:

    !python /content/stylized-neural-painting/demo_prog.py
    --img_path /content/stylized-neural-painting/test_images/apple.jpg
    --canvas_color 'white'
    --canvas_size 512
    --max_m_strokes 500
    --max_divide 5
    --with_ot_loss
    --renderer "oilpaintbrush"
    --renderer_checkpoint_dir "checkpoints_G_oilpaintbrush_light"
    --net_G "zou-fusion-net-light"
    --disable_preview

    This is the error i get back. I do not know enough programming to fix it myself.

    rendering canvas... Traceback (most recent call last): File "/content/stylized-neural-painting/demo_prog.py", line 113, in optimize_x(pt) File "/content/stylized-neural-painting/demo_prog.py", line 102, in optimize_x CANVAS_tmp = pt._render(PARAMS, save_jpgs=False, save_video=False) File "/content/stylized-neural-painting/painter.py", line 137, in _render self.rderr.draw_stroke() File "/content/stylized-neural-painting/renderer.py", line 148, in draw_stroke return self._draw_oilpaintbrush() File "/content/stylized-neural-painting/renderer.py", line 326, in _draw_oilpaintbrush x0, y0, w, h, theta, R0, G0, B0, R2, G2, B2) File "/content/stylized-neural-painting/utils.py", line 286, in create_transformed_brush brush_alpha = (brush_alpha > 0).astype(np.float32) TypeError: '>' not supported between instances of 'NoneType' and 'int'

    Thanks for your help!

    opened by wutti90 3
  • Style Transfer Colab Notebook error

    Style Transfer Colab Notebook error

    The first part of the image-to-painting-nst.ipynb works fine since this is creating a painting through brush strokes and then a video.

    The second part is the style transfer code, and I am getting an error here: Screen Shot 2021-12-12 at 4 57 40 PM

    opened by metaphorz 2
  • Ask some questions about this project

    Ask some questions about this project

    Ask if the G_loss variable in this project refers to the loss of the generator?

    In addition, I would like to ask where D_loss is written? I don't seem to see it.

    opened by dadasx 0
  • 2 Questions about the method of this paper

    2 Questions about the method of this paper

    Nice Job! I would like to ask two questions about the paper? Please just tell me in principle, Thanks!

    1. How to initialize the strokes? The strokes are initialize one by one or all togethor?
    2. How to generate the results that transfer color only?
    opened by TZYSJTU 0
  • Cannot find zipfile directory

    Cannot find zipfile directory

    Hi,

    Thanks for the great work!

    Was trying to run the model using your demo colab notebook. I get stuck on the part where I need to download and unzip the checkpoint file. The unzip command gives:

    Archive:  checkpoints_G_oilpaintbrush.zip
      End-of-central-directory signature not found.  Either this file is not
      a zipfile, or it constitutes one disk of a multi-part archive.  In the
      latter case the central directory and zipfile comment will be found on
      the last disk(s) of this archive.
    unzip:  cannot find zipfile directory in one of checkpoints_G_oilpaintbrush.zip or
            checkpoints_G_oilpaintbrush.zip.zip, and cannot find checkpoints_G_oilpaintbrush.zip.ZIP, period.
    

    Any idea how I can fix this? I tried wget for the same file and no luck there either.

    opened by vshesh 2
  • The difference of  _render and _drawing_step_states

    The difference of _render and _drawing_step_states

    I tried to use the stylized-neural-painting code and saw that _render and _drawing_step_states in painter.py is somewhat different. Does anyone know this reason?

    opened by MADONOKOUKI 0
  • Graphics card not working

    Graphics card not working

    I have followed all the step as you mentioned in your video but my system didn't using graphics card when rendering the photo and its taking too much time to render. I have used the same joker image for trail and used brush strokes to 500 and divide to 5 and it took almost half and hour to render. I have NVIDIA RTX 3080 12 gb.

    opened by PrithviPrajapati 0
  • About training new painting styles

    About training new painting styles

    Hello, your work is very creative, but I found that you didn't have train dataset, so how did you train the strokes of different painting styles? If I want to train a Chinese style painting stroke, what should I do? Hope your reply. Thanks so much!

    opened by hensiesp32 1
Owner
Zhengxia Zou
Postdoc at the University of Michigan. Research interest: computer vision and applications in remote sensing, self-driving, and video games.
Zhengxia Zou
Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN", accepted to ACM MM 2021 BNI Track.

RecycleD Official PyTorch implementation of the paper "Recycling Discriminator: Towards Opinion-Unaware Image Quality Assessment Using Wasserstein GAN

Yunan Zhu 23 Nov 5, 2022
This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

Underwater Light Field Retention : Neural Rendering for Underwater Imaging (UWNR) (Accepted by CVPR Workshop2022 NTIRE) Authors: Tian Ye†, Sixiang Che

jmucsx 17 Dec 14, 2022
[Preprint] ConvMLP: Hierarchical Convolutional MLPs for Vision, 2021

Convolutional MLP ConvMLP: Hierarchical Convolutional MLPs for Vision Preprint link: ConvMLP: Hierarchical Convolutional MLPs for Vision By Jiachen Li

SHI Lab 143 Jan 3, 2023
This project is the official implementation of our accepted ICLR 2021 paper BiPointNet: Binary Neural Network for Point Clouds.

BiPointNet: Binary Neural Network for Point Clouds Created by Haotong Qin, Zhongang Cai, Mingyuan Zhang, Yifu Ding, Haiyu Zhao, Shuai Yi, Xianglong Li

Haotong Qin 59 Dec 17, 2022
This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis, accepted at EMNLP 2021.

MultiModal-InfoMax This repository contains the official implementation code of the paper Improving Multimodal Fusion with Hierarchical Mutual Informa

Deep Cognition and Language Research (DeCLaRe) Lab 89 Dec 26, 2022
Python and C++ implementation of "MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation". Accepted at LXCV @ CVPR 2021.

MarkerPose: Robust real-time planar target tracking for accurate stereo pose estimation This is a PyTorch and LibTorch implementation of MarkerPose: a

Jhacson Meza 47 Nov 18, 2022
This repository provides the official implementation of 'Learning to ignore: rethinking attention in CNNs' accepted in BMVC 2021.

inverse_attention This repository provides the official implementation of 'Learning to ignore: rethinking attention in CNNs' accepted in BMVC 2021. Le

Firas Laakom 5 Jul 8, 2022
the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.

EmbedSeg Introduction This repository hosts the version of the code used for the preprint Embedding-based Instance Segmentation of Microscopy Images.

JugLab 88 Dec 25, 2022
[Preprint] "Chasing Sparsity in Vision Transformers: An End-to-End Exploration" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang

Chasing Sparsity in Vision Transformers: An End-to-End Exploration Codes for [Preprint] Chasing Sparsity in Vision Transformers: An End-to-End Explora

VITA 64 Dec 8, 2022
[Preprint] "Bag of Tricks for Training Deeper Graph Neural Networks A Comprehensive Benchmark Study" by Tianlong Chen*, Kaixiong Zhou*, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, Zhangyang Wang

Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Study Codes for [Preprint] Bag of Tricks for Training Deeper Graph

VITA 101 Dec 29, 2022
This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021.

PyTorch implementation of DAQ This is an official implementation of the paper "Distance-aware Quantization", accepted to ICCV2021. For more informatio

CV Lab @ Yonsei University 36 Nov 4, 2022
The official implementation of paper Siamese Transformer Pyramid Networks for Real-Time UAV Tracking, accepted by WACV22

SiamTPN Introduction This is the official implementation of the SiamTPN (WACV2022). The tracker intergrates pyramid feature network and transformer in

Robotics and Intelligent Systems Control @ NYUAD 28 Nov 25, 2022
Implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hashing by Maximizing Bit Entropy

Deep Unsupervised Image Hashing by Maximizing Bit Entropy This is the PyTorch implementation of accepted AAAI 2021 paper: Deep Unsupervised Image Hash

null 62 Dec 30, 2022
Implementation of Geometric Vector Perceptron, a simple circuit for 3d rotation equivariance for learning over large biomolecules, in Pytorch. Idea proposed and accepted at ICLR 2021

Geometric Vector Perceptron Implementation of Geometric Vector Perceptron, a simple circuit with 3d rotation equivariance for learning over large biom

Phil Wang 59 Nov 24, 2022
This repo contains the pytorch implementation for Dynamic Concept Learner (accepted by ICLR 2021).

DCL-PyTorch Pytorch implementation for the Dynamic Concept Learner (DCL). More details can be found at the project page. Framework Grounding Physical

Zhenfang Chen 31 Jan 6, 2023
This is the pytorch implementation for the paper: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation, which is accepted to ICCV2021.

GMPQ: Generalizable Mixed-Precision Quantization via Attribution Rank Preservation This is the pytorch implementation for the paper: Generalizable Mix

null 18 Sep 2, 2022
Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

Jie Liu 111 Dec 31, 2022
An official source code for paper Deep Graph Clustering via Dual Correlation Reduction, accepted by AAAI 2022

Dual Correlation Reduction Network An official source code for paper Deep Graph Clustering via Dual Correlation Reduction, accepted by AAAI 2022. Any

yueliu1999 109 Dec 23, 2022
The official PyTorch code for 'DER: Dynamically Expandable Representation for Class Incremental Learning' accepted by CVPR2021

DER.ClassIL.Pytorch This repo is the official implementation of DER: Dynamically Expandable Representation for Class Incremental Learning (CVPR 2021)

rhyssiyan 108 Jan 1, 2023