pytorch implementation of fast-neural-style

Overview

fast-neural-style ๐ŸŒ‡ ๐Ÿš€

NOTICE: This codebase is no longer maintained, please use the codebase from pytorch examples repository available at pytorch/examples/fast_neural_style.

This repository contains a pytorch implementation of an algorithm for artistic style transfer. The algorithm can be used to mix the content of an image with the style of another image. For example, here is a photograph of a door arch rendered in the style of a stained glass painting.

The model uses the method described in Perceptual Losses for Real-Time Style Transfer and Super-Resolution along with Instance Normalization. The saved-models for examples shown in the README can be downloaded from here.

DISCLAIMER: This implementation is also a part of the pytorch examples repository. Implementation in this repository uses pretrained Caffe2 VGG whereas the pytorch examples repository implementation uses pretrained Pytorch VGG. The two VGGs have different preprocessings which results in different --content-weight and --style-weight parameters. The styled output images also look slightly different.

Requirements

The program is written in Python, and uses pytorch, scipy. A GPU is not necessary, but can provide a significant speed up especially for training a new model. Regular sized images can be styled on a laptop, desktop using saved models.

Setup the environnment

Run with virtualenv

Create a virtualenv with python3.5 or python3.6. Older versions are not supported due to a lack of compatibilty with pytorch.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Run with Docker

Build the image:

docker build . -t fast-neural-style

Run the container:

docker run --rm --volume "$(pwd)/:/data" style eval --content-image /data/image.jpg --model /app/saved-models/mosaic.pth --output-image /data/output.jpg --cuda 0

Usage

Stylize image

python neural_style/neural_style.py eval --content-image </path/to/content/image> --model </path/to/saved/model> --output-image </path/to/output/image> --cuda 0
  • --content-image: path to content image you want to stylize.
  • --model: saved model to be used for stylizing the image (eg: mosaic.pth)
  • --output-image: path for saving the output image.
  • --content-scale: factor for scaling down the content image if memory is an issue (eg: value of 2 will halve the height and width of content-image)
  • --cuda: set it to 1 for running on GPU, 0 for CPU.

Train model

python neural_style/neural_style.py train --dataset </path/to/train-dataset> --style-image </path/to/style/image> --vgg-model-dir </path/to/vgg/folder> --save-model-dir </path/to/save-model/folder> --epochs 2 --cuda 1

There are several command line arguments, the important ones are listed below

  • --dataset: path to training dataset, the path should point to a folder containing another folder with all the training images. I used COCO 2014 Training images dataset [80K/13GB] (download).
  • --style-image: path to style-image.
  • --vgg-model-dir: path to folder where the vgg model will be downloaded.
  • --save-model-dir: path to folder where trained model will be saved.
  • --cuda: set it to 1 for running on GPU, 0 for CPU.

Refer to neural_style/neural_style.py for other command line arguments.

Models

Models for the examples shown below can be downloaded from here or by running the script download_styling_models.sh.


Comments
  • How to set output_image_size when Eval

    How to set output_image_size when Eval

    Hi 1ใ€Can you explain what's different of the image_size of "content and style" when Train and Eval? 2ใ€How to set output_image_size when Eval? (if your default setting is 1080,can you show me where you set in your code?) 3ใ€Really thanks for your work

    opened by dovanchan 18
  • Query Regarding Transforms used for input image and style image

    Query Regarding Transforms used for input image and style image

    What is the purpose of this line in your transforms block for the input :

    transforms.Lambda(lambda x: x.mul(255))

    Aren't the input and style images already in the range 0-255? Wouldn't this cause an overflow?

    opened by ssundar6087 5
  • train doesn't run: ValueError: not enough values to unpack (expected 3, got 2)

    train doesn't run: ValueError: not enough values to unpack (expected 3, got 2)

    Hello. train doesn't run:

    danusya@werewolf:~/fast-neural-style$ python3 neural_style/neural_style.py train --dataset ~/dataset/ --style-image ~/Caleido/Color_Kaleidoscope_2017-6-24_15141.png  --vgg-model-dir caleido_vgg --save-model-dir caleido_model --epochs 2 --cuda 0
    /usr/local/lib/python3.5/dist-packages/torchvision/transforms/transforms.py:156: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
      "please use transforms.Resize instead.")
    Traceback (most recent call last):
      File "neural_style/neural_style.py", line 210, in <module>
        main()
      File "neural_style/neural_style.py", line 204, in main
        train(args)
      File "neural_style/neural_style.py", line 50, in train
        style = utils.preprocess_batch(style)
      File "/home/danusya/fast-neural-style/neural_style/utils.py", line 59, in preprocess_batch
        (r, g, b) = torch.chunk(batch, 3)
    ValueError: not enough values to unpack (expected 3, got 2)
    
    opened by xaionaro 5
  • About adding Tanh() at the end of the model

    About adding Tanh() at the end of the model

    Hi, when adding Tanh() at the end of the transformer model, I got error during the training. RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation Is there a way to fix this problem? Thanks!

    opened by zhanghang1989 5
  • Time for training a new style

    Time for training a new style

    Hi, I am using p2.xlarge in AWS (NVIDIA K80 GPU, 61GB RAM , 128GB Space). To check it out I was using COCO's val2017(5K images). It downloaded the vgg-model file and it has been blank for a very long while. 0.How much time does it take to get trained for 5000 images? 1.Also when I am trying to kill the python process with "kill -9 PID", it is not getting killed, is it because of the multi threading ? How do I end the training process in between ?

    --> I am able to run 'eval' option using cuda 1 and it takes about 1-2 seconds to generate output images. So I am assuming all the package installations are fine on my side.

    Please help me resolve the issues.

    opened by nitish116 3
  • Runtime error while trying out the toy example

    Runtime error while trying out the toy example

    I am getting a runtime error w.r.t tensor size while trying out the toy example.

    RuntimeError: The expanded size of the tensor (1080) must match the existing size (32) at non-singleton dimension 2. at /opt/conda/conda-bld/pytorch_1502001039157/work/torch/lib/TH/generic/THTensor.c:308

    I am using Python 2.7 and used conda install pytorch torchvision -c soumith to install pytorch

    Stack trace

    File "~/envs/py27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__ result = self.forward(*input, **kwargs) File "~/fast-neural-style/neural_style/transformer_net.py", line 131, in forward mean = torch.mean(t, 2).unsqueeze(2).expand_as(x) File "~/envs/py27/lib/python2.7/site-packages/torch/autograd/variable.py", line 725, in expand_as return Expand.apply(self, (tensor.size(),)) File "~/envs/py27/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward result = i.expand(*new_size)

    opened by whiletruelearn 2
  • Python error: <stdin> is a directory, cannot continue

    Python error: is a directory, cannot continue

    DOVAN$ python neural_style/neural_style.py eval --content-image </Users/DOVAN/Desktop/WechatIMG128.jpeg> --model </Users/DOVAN/deeprely/fast-neural-style/saved-models/starry-night.pth> --output-image </Users/DOVAN/Desktop/test> --cuda 0 Python error: is a directory, cannot continue

    opened by dovanchan 1
  • Could you upload vgg16.weight file?

    Could you upload vgg16.weight file?

    Hi,

    I'm trying to train new styles images on Windows (anaconda, pytorch 0.3). Using vgg.t7 file gives me the following error messages:

    ... ... File "c:\Anaconda3\envs\py36torch\lib\site-packages\torch\utils\serialization\read_lua_file.py", line 572, in read_table v = self.read() File "c:\Anaconda3\envs\py36torch\lib\site-packages\torch\utils\serialization\read_lua_file.py", line 598, in read "corrupted.".format(typeidx)) torch.utils.serialization.read_lua_file.T7ReaderException: unknown type id 1056626884. The file may be corrupted.

    I assume this is due to linux/windows compatibility and would like to try with vgg16.weight file.

    Thank you for your code.

    opened by jmhong-simulation 1
  • content loss calculation is wrong?

    content loss calculation is wrong?

    Hi, I found the vgg16 loss network returns [relu1_2, relu2_2, relu3_3, relu4_3], and in your neural_style.py,

    y = transformer(x)
    features_y = vgg(y)
    f_xc_c = Variable(features_xc[1].data, requires_grad=False)
    content_loss = args.content_weight * mse_loss(features_y[1], f_xc_c)
    

    the features_y is a list of loss corresponding to [relu1_2, relu2_2, relu3_3, relu4_3], the index for relu1_2, relu2_2, relu3_3 and relu4_3 is 0, 1, 2, and 3, right? In the paper, it used relu3_3 as content loss, so, here should it be the following? f_xc_c = Variable(features_xc[2].data, requires_grad=False) content_loss = args.content_weight * mse_loss(features_y[2], f_xc_c) Thanks.

    opened by RyanCV 1
  • some question in utils.py

    some question in utils.py

    def tensor_save_rgbimage(tensor, filename, cuda=False):
        if cuda:
            img = tensor.clone().cpu().clamp(0, 255).numpy()
        else:
            img = tensor.clone().clamp(0, 255).numpy()
        img = img.transpose(1, 2, 0).astype('uint8')
        img = Image.fromarray(img)
        img.save(filename)
    
    
    def tensor_save_bgrimage(tensor, filename, cuda=False):
        (b, g, r) = torch.chunk(tensor, 3)
        tensor = torch.cat((r, g, b))
        tensor_save_rgbimage(tensor, filename, cuda)
    

    it seems like if you use function tensor_save_bgrimage to save a tensor, it will first transform from BGR to RGB and then transform from RGB to GBR in the function tensor_save_rgbimage?

    opened by poptree 1
  • download_styling_models.sh error

    download_styling_models.sh error

    When I run download_styling_models.sh, it gets error: failed: Connection refused. I couldn't download the pretrained model from the website. Any ideas? Thanks!

    opened by bliunlpr 1
  • Bump numpy from 1.14.3 to 1.22.0

    Bump numpy from 1.14.3 to 1.22.0

    Bumps numpy from 1.14.3 to 1.22.0.

    Release notes

    Sourced from numpy's releases.

    v1.22.0

    NumPy 1.22.0 Release Notes

    NumPy 1.22.0 is a big release featuring the work of 153 contributors spread over 609 pull requests. There have been many improvements, highlights are:

    • Annotations of the main namespace are essentially complete. Upstream is a moving target, so there will likely be further improvements, but the major work is done. This is probably the most user visible enhancement in this release.
    • A preliminary version of the proposed Array-API is provided. This is a step in creating a standard collection of functions that can be used across application such as CuPy and JAX.
    • NumPy now has a DLPack backend. DLPack provides a common interchange format for array (tensor) data.
    • New methods for quantile, percentile, and related functions. The new methods provide a complete set of the methods commonly found in the literature.
    • A new configurable allocator for use by downstream projects.

    These are in addition to the ongoing work to provide SIMD support for commonly used functions, improvements to F2PY, and better documentation.

    The Python versions supported in this release are 3.8-3.10, Python 3.7 has been dropped. Note that 32 bit wheels are only provided for Python 3.8 and 3.9 on Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit integer OpenBLAS, which should fix the occasional problems encountered by folks using truly huge arrays.

    Expired deprecations

    Deprecated numeric style dtype strings have been removed

    Using the strings "Bytes0", "Datetime64", "Str0", "Uint32", and "Uint64" as a dtype will now raise a TypeError.

    (gh-19539)

    Expired deprecations for loads, ndfromtxt, and mafromtxt in npyio

    numpy.loads was deprecated in v1.15, with the recommendation that users use pickle.loads instead. ndfromtxt and mafromtxt were both deprecated in v1.17 - users should use numpy.genfromtxt instead with the appropriate value for the usemask parameter.

    (gh-19615)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • RuntimeError: The expanded size of the tensor (700) must match the existing size (32) at non-singleton dimension 2

    RuntimeError: The expanded size of the tensor (700) must match the existing size (32) at non-singleton dimension 2

    When I run your code in Pytorch 0.4.0,it will be this error: RuntimeError: The expanded size of the tensor (700) must match the existing size (32) at non-singleton dimension 2

    I think it just because of the version of Pytorch ,Can you fix it,thank you?

    opened by dovanchan 1
  • download_styling_models.sh error

    download_styling_models.sh error

    When I run download_styling_models.sh, it gets error๏ผš --2018-03-16 09:26:26-- (try: 2) https://www.dropbox.com/s/gtwnyp9n49lqs7t/saved-models.zip?dl=1 Connecting to www.dropbox.com (www.dropbox.com)|162.125.82.1|:443... failed: Connection timed out. Connecting to www.dropbox.com (www.dropbox.com)|2620:100:6032:1::a27d:5201|:443... failed: Cannot assign requested address. Retrying.

    Is there any other way to download it? Thanks๏ผ

    opened by mqchen1993 0
  • trained model not working

    trained model not working

    epoch_2_Sat_Oct_21_17:31:07_2017_1.0_5.0.model.tar.gz i have trained a model, program has finished successfully, yet evaluating the model seems to produce a blank screen. i have attached the model. https://www.dropbox.com/s/r9fxsy129he7rd1/epoch_2_Sat_Oct_21_17%3A31%3A07_2017_1.0_5.0.model.tar.gz?dl=0 please help ):

    opened by ksesalebangim 1
Owner
Abhishek Kadian
Engineer @facebookresearch
Abhishek Kadian
pytorch implementation of fast-neural-style

fast-neural-style ?? ?? NOTICE: This codebase is no longer maintained, please use the codebase from pytorch examples repository available at pytorch/e

Abhishek Kadian 405 Dec 15, 2022
Only a Matter of Style: Age Transformation Using a Style-Based Regression Model

Only a Matter of Style: Age Transformation Using a Style-Based Regression Model The task of age transformation illustrates the change of an individual

null 444 Dec 30, 2022
PyTorch implementation of neural style transfer algorithm

neural-style-pt This is a PyTorch implementation of the paper A Neural Algorithm of Artistic Style by Leon A. Gatys, Alexander S. Ecker, and Matthias

null 770 Jan 2, 2023
Official PyTorch implementation of "ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows"

ArtFlow Official PyTorch implementation of the paper: ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows Jie An*, Siyu Huang*, Yibing

null 123 Dec 27, 2022
PyTorch implementation of neural style randomization for data augmentation

README Augment training images for deep neural networks by randomizing their visual style, as described in our paper: https://arxiv.org/abs/1809.05375

null 84 Nov 23, 2022
PyTorch implementation of paper: AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer, ICCV 2021.

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer [Paper] [PyTorch Implementation] [Paddle Implementation] Overview This reposit

null 148 Dec 30, 2022
Implementation of Neural Style Transfer in Pytorch

PytorchNeuralStyleTransfer Code to run Neural Style Transfer from our paper Image Style Transfer Using Convolutional Neural Networks. Also includes co

Leon Gatys 396 Dec 1, 2022
An implementation of "Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport"

Optex An implementation of Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport for TU Delft CS4240. You c

Hans Brouwer 33 Jan 5, 2023
Super-Fast-Adversarial-Training - A PyTorch Implementation code for developing super fast adversarial training

Super-Fast-Adversarial-Training This is a PyTorch Implementation code for develo

LBK 26 Dec 2, 2022
Neural style transfer in PyTorch.

style-transfer-pytorch An implementation of neural style transfer (A Neural Algorithm of Artistic Style) in PyTorch, supporting CPUs and Nvidia GPUs.

Katherine Crowson 395 Jan 6, 2023
Neural style transfer as a class in PyTorch

pt-styletransfer Neural style transfer as a class in PyTorch Based on: https://github.com/alexis-jacq/Pytorch-Tutorials Adds: StyleTransferNet as a cl

Tyler Kvochick 31 Jun 27, 2022
Pytorch tutorials for Neural Style transfert

PyTorch Tutorials This tutorial is no longer maintained. Please use the official version: https://pytorch.org/tutorials/advanced/neural_style_tutorial

Alexis David Jacq 135 Jun 26, 2022
FAST-RIR: FAST NEURAL DIFFUSE ROOM IMPULSE RESPONSE GENERATOR

This is the official implementation of our neural-network-based fast diffuse room impulse response generator (FAST-RIR) for generating room impulse responses (RIRs) for a given acoustic environment.

Anton Jeran Ratnarajah 89 Dec 22, 2022
Fast Style Transfer in TensorFlow

Fast Style Transfer in TensorFlow Add styles from famous paintings to any photo in a fraction of a second! You can even style videos! It takes 100ms o

Jefferson 5 Oct 24, 2021
TensorFlow CNN for fast style transfer

Fast Style Transfer in TensorFlow Add styles from famous paintings to any photo in a fraction of a second! It takes 100ms on a 2015 Titan X to style t

null 1 Dec 14, 2021
An implementation of the paper "A Neural Algorithm of Artistic Style"

A Neural Algorithm of Artistic Style implementation - Neural Style Transfer This is an implementation of the research paper "A Neural Algorithm of Art

Srijarko Roy 27 Sep 20, 2022
This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

DeepMind 892 Dec 28, 2022
A PyTorch implementation of "SimGNN: A Neural Network Approach to Fast Graph Similarity Computation" (WSDM 2019).

SimGNN โ €โ €โ € A PyTorch implementation of SimGNN: A Neural Network Approach to Fast Graph Similarity Computation (WSDM 2019). Abstract Graph similarity s

Benedek Rozemberczki 534 Dec 25, 2022
Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

null 364 Dec 14, 2022