A web porting for NVlabs' StyleGAN2, to facilitate exploring all kinds characteristic of StyleGAN networks

Overview

Explorer Demo

This project is a web porting for NVlabs' StyleGAN2, to facilitate exploring all kinds characteristic of StyleGAN networks.

Thanks for NVlabs' excellent work.

Features

Explorer

See how the result image response to changes of latent code & psi.

Projector Demo 1

Projector

Test the projection from image to latent code. Left is target image, right is result from generator model. Total step count and yielding interval can be configured in this page. And another hyperparameter regularize noise weight can be configured statically, see Environment Configurations.

You can save the projection result in a zip package, and this page can accept projector zip file dropping, so this feature enable you to share your projector result to others.

Projector Demo 1 Projector Demo 2

W latent space extension

We added an env switch of UNIFORM_LATENTS to denote using uniform or separated W latent code when projecting image. This is the results comparison (center is the target):

W latents result   β‡¨   target image   β‡¦   W+ latents result

Projection animation exporting

You can also export image projection result sequence as a gif animation:

  β‡¦  

Face image pose alignment

Dataset of FFHQ's generation has a crop process to align face area.see paper, appendix C. So the output distribution of StyleGAN model learned on FFHQ has a strong prior tendency on features position. We observed that many face images projection suffers semantic mistakes, e.g. erasing original eyes and transforming eyebrow into eyes during projection fitting (however you can get a similar face at last, but it may yield freak results when you manipulate the latent code). Finally we figured out that mainly caused by unalignment with training dataset prior distribution.

Then we import the face-api to measure and align human face images as below:

- πŸ™‚ β†’ - βœ„ β†’

Gratefulness for the authorization by @芈砾 to use his nice opus.

Click the button [ πŸ™‚ ] after target image loaded, if the face detection succeed, you will get the face landmark and proposed crop box. The detection result may be not very accurate, now you can adjust 3 anchor marks manually to align left eye (red), right eye (green) and mouth (blue). Then click button [βœ„] to apply the crop.

Merger

Once you get some latent codes by projector or turning, you can test to mix features by interpolating latent values on every W layer. This is a demo.

Merger Demo

The pair of top-left images are the source to merge, press Ctrl+V in the hash box below either image to paste input latent code via clipboard, and Ctrl+C on the right blank area to copy result latent code.

Mapping Network Research

mapping plot

I attempt to explore the StyleGAN mapping network high-dimensional terrain aspect, read this article for details.

Usage

Run the web server:

python ./http_server.py

If this works, open http://localhost:8186 in your browser.

To ensure it working, please read the following requirements before do this.

Requirements

Python

Install requirement libraries with pip, reference to requirements.txt.

Network Files

Before run the web server, StyleGAN2 pre-trained network files must be placed in local disk (recommended the folder models). You can download network files following to StyleGAN2's code.

For memory reason, only one generator model can be loaded when running the web server. Network file paths can be configured by env variables. Create a file named .env.local under project root to configure chosen model and network file paths. Network file name/paths are configured in key-value style, e.g.:

MODEL_NAME=ffhq		# ffhq is the default value, so this line can be ignored 

MODEL_PATH_ffhq=./models/stylegan2-ffhq-config-f.pkl
MODEL_PATH_cat=./models/stylegan2-cat-config-f.pkl
# And so on...

Alternately, you can also choose generator model name by start command argument, e.g.:

python ./http_server.py cat

Or, for nodejs developer:

yarn start cat

Besides generators, the network LPIPS is required when run image projector, the default local path is ./models/vgg16_zhang_perceptual.pkl, download link. You can also change local path by env variable MODEL_PATH_LPIPS.

For Windows

According to StyleGAN2 README.md, here are our additional help instructions:

  • MSVC

    NOTE: Visual Studio 2019 Community Edition seems not compatible with CUDA 10.0, Visual Studio 2017 works.

    Append the actual msvc binary directory (find in your own disk) into dnnlib/tflib/custom_ops.py, the array of compiler_bindir_search_path. For example:

     -	'C:/Program Files (x86)/Microsoft Visual Studio 14.0/vc/bin',
     +	'C:/Program Files (x86)/Microsoft Visual Studio/2017/BuildTools/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64',
  • NVCC

    To test if nvcc is configured properly, dowload test_nvcc.cu in StyleGAN2 project. And the test command should specify binary path:

     nvcc test_nvcc.cu -o test_nvcc -run -ccbin "C:\Program Files (x86)\Microsoft VisualStudio\2017\BuildTools\VC\Tools\MSVC\14.16.27023\bin\Hostx64\x64"

    Actual path to different msvc edition may have difference in detail. If this succeed, it will build a file named test_nvcc.exe.

  • Tips for tensorflow 1.15

    Tensorflow 1.15 can work under Windows, but NVCC compiling may encounter C++ including path problem. Here is an easy workaround: make a symbolic link in python installation directory Python36\Lib\site-packages\tensorflow_core:

     mklink /J tensorflow tensorflow_core
  • Tips for tensorflow 2.x

    Tensorflow 2.0+ can work now! I have solved the compatibility issues with TF2 already, including some modification of code bundled in pickle. Except one problem on Windows, if you encountered this:

    C:/Users/xxx/AppData/Local/Programs/Python/Python36/lib/site-packages/tensorflow/include\unsupported/Eigen/CXX11/Tensor(74): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory

    Just open this file and comment out this line simply:

    #include

    It seems a bug of tensorflow, and I have committed an issue for them.

  • cudafe++ issue

    If you encountered python console error like:

     nvcc error : 'cudafe++' died with status 0xC0000005 (ACCESS_VIOLATION)
    

    That may caused by a bug from CUDA 10.0, you can fix this issue by replacing your cudafe++.exe file in CUDA program bin directory by the same name file from CUDA 10.1 or higher version. And you are welcome to download my backup to avoid install a whole new version CUDA.

Environment Configurations

To manage environment variables conveniently, create a configuration file named .env.local. All avaiable env list:

Key Description Default Value
HTTP_HOST Web server host. 127.0.0.1
HTTP_PORT Web server port. 8186
MODEL_NAME Name for the generator model to load, this can be overwrite by the first argument of start script. ffhq
MODEL_PATH_LPIPS File path for LPIPS model. ./models/vgg16_zhang_perceptual.pkl
MODEL_PATH_* Generator network file path dictionary. See examples.
REGULARIZE_NOISE_WEIGHT Projector training hyperparameter. Float. 1e5
INITIAL_NOISE_FACTOR Projector training hyperparameter. Float. 0.05
EUCLIDEAN_DIST_WEIGHT Projector training hyperparameter. Float. 1
REGULARIZE_MAGNITUDE_WEIGHT Projector training hyperparameter. Float. 0
UNIFORM_LATENTS Use uniform latents for all feature layers (consistent with origin StyleGAN2 paper). Boolean, 0 or 1 0

 

 

 

A Bonus :)

Comments
  • Frontend Build Error - Because of fast-api.js

    Frontend Build Error - Because of fast-api.js

    Hello,

    First of all, I'd like to thank you for this amazing repository! My friend and I are examining it in this fork: https://github.com/aiksir/stylegan-web

    When I run this command to build the frontend: npm run-script build, I get the following error:

    (forked-stylegan-web) baris@bariss-MacBook-Pro forked-stylegan-web % npm run-script build
    
    > [email protected] build /Users/baris/Documents/PycharmProjects/forked-stylegan-web
    > rollup -c rollup.config.js
    
    
    ./app/index.js β†’ dist/index.bundle.js...
    created dist/index.bundle.js in 880ms
    
    ./app/projector.js β†’ dist/projector.bundle.js...
    [!] Error: 'import' and 'export' may only appear at the top level
    node_modules/face-api.js/build/es6/env/isNodejs.js (1:0)
    1: export function isNodejs() {
       ^
    2:     return typeof global === 'object'
    3:         && typeof require === 'function'
    Error: 'import' and 'export' may only appear at the top level
        at error (/Users/baris/Documents/PycharmProjects/forked-stylegan-web/node_modules/rollup/dist/shared/node-entry.js:5400:30)
        at Module.error (/Users/baris/Documents/PycharmProjects/forked-stylegan-web/node_modules/rollup/dist/shared/node-entry.js:9824:16)
        at tryParse (/Users/baris/Documents/PycharmProjects/forked-stylegan-web/node_modules/rollup/dist/shared/node-entry.js:9717:23)
        at Module.setSource (/Users/baris/Documents/PycharmProjects/forked-stylegan-web/node_modules/rollup/dist/shared/node-entry.js:10080:33)
        at /Users/baris/Documents/PycharmProjects/forked-stylegan-web/node_modules/rollup/dist/shared/node-entry.js:12366:20
        at async Promise.all (index 4)
        at async Promise.all (index 6)
        at async Promise.all (index 2)
        at async Promise.all (index 0)
        at async Promise.all (index 1)
        at async Promise.all (index 0)
        at async Promise.all (index 0)
        at async Promise.all (index 0)
    
    npm ERR! code ELIFECYCLE
    npm ERR! errno 1
    npm ERR! [email protected] build: `rollup -c rollup.config.js`
    npm ERR! Exit status 1
    npm ERR! 
    npm ERR! Failed at the [email protected] build script.
    npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
    
    npm ERR! A complete log of this run can be found in:
    npm ERR!     /Users/baris/.npm/_logs/2020-10-15T10_02_32_356Z-debug.log
    

    I have made a quick search on the internet but I couldn't figure it out because I'm not much familiar with Vue. Are you aware of this issue?

    opened by BarisSari 2
  • Couldn't set up environment  using vcvars64.bat

    Couldn't set up environment using vcvars64.bat

    nvcc fatal : Could not set up the environment for Microsoft Visual Studio using 'C:/Program Files (x86)/Microsoft Visual Studio/2017/Community/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/../../../../../../../VC/Auxiliary/Build/vcvars64.bat'

    opened by punyaslokdutta 0
  • Introduce a configurable gist / with downloadable npy / latent files that we could show to every user

    Introduce a configurable gist / with downloadable npy / latent files that we could show to every user

    Some one creates a keanu reaves latent representation - we hook into this list

    { "latentRepresentations": [ { "title": "Keanu", "file": "Keanu.npy", "thumb": "https://github.com/name/Keanu.png", "url": "https://github.com/name/Keanu.npy" },

    }

    opened by johndpope 0
  • Enhancement - prototype photoshop like layers for web ui / image pipeline

    Enhancement - prototype photoshop like layers for web ui / image pipeline

    https://github.com/battleaxedotco/brutalism

    We maybe able to drop this into the main view - and achieve some organization.

    eg. might make sense to collapse the maths stuff into one of the tabs at bottom.

    Another one for the 512 vectors. Another one for the latent directions Another one for ....

    opened by johndpope 4
  • Enhancement - Feature  load / save / find latent_representations / augmentation

    Enhancement - Feature load / save / find latent_representations / augmentation

    https://github.com/jasonlbx13/FaceHack/blob/b4beda81354806a13efe3c992d819fc6a9f83d65/face_gan/main.py

    Given an existing star / celebrity load / save

    this code could be put into a pipeline / for subparsers maybe it may have to look like photoshop layers (will raise another ticket)

    Uploading Screen Shot 2021-01-17 at 9.05.41 am.png…

    
    def augmentation(args):
    
        os.makedirs(args.dst_dir+'/augmentation', exist_ok=True)
    
        print('Loading networks from "{}"...'.format(args.network_pkl))
        tflib.init_tf()
        with open(args.network_pkl, "rb") as f:
            generator_network, discriminator_network, Gs_network = pickle.load(f)
    
        Gs_syn_kwargs = dnnlib.EasyDict()
        Gs_syn_kwargs.output_transform = dict(func=tflib.convert_images_to_uint8, nchw_to_nhwc=True)
        Gs_syn_kwargs.randomize_noise = False
        Gs_syn_kwargs.minibatch_size = 1
        pic = np.load(args.src_dir)
        for i in tqdm(range(int(args.num))):
    
            scale1 = random.uniform(-float(args.smile_range), float(args.smile_range))
            scale2 = random.uniform(-float(args.age_range), float(args.age_range))
            scale3 = random.uniform(-float(args.angleh_range), float(args.angleh_range))
            scale4 = random.uniform(-float(args.anglep_range), float(args.anglep_range))
            scale5 = random.uniform(-float(args.angry_range), float(args.angry_range))
            scale6 = random.uniform(-float(args.sad_range), float(args.sad_range))
            scale7 = random.uniform(-float(args.eye_range), float(args.eye_range))
    
            latent_vector = pic[np.newaxis, :]
            new_latent_vector = latent_vector.copy()
            new_latent_vector[0][:8] = (latent_vector[0] + scale1 * smile_drt + scale2 * age_drt
                                        + scale3 * angleh_drt + scale4 * anglep_drt
                                        + scale5 * angry_drt + scale6 * sad_drt
                                        + scale7 * eye_drt)[:8]
            img = Gs_network.components.synthesis.run(new_latent_vector, **Gs_syn_kwargs)
            img = PIL.Image.fromarray(img[0])
            img.save('{}/augmentation/{}.png'.format(args.dst_dir, str(i).zfill(4)))
    
    
    opened by johndpope 1
  • Feature Suggestion - include latent_directions

    Feature Suggestion - include latent_directions

    So I've been trying out different stylegan repos - and I'm imaging a fusion of two. Basically the guts of this repo https://github.com/GreenLimeSia/GenEdi/issues/3

    Screenshot_2020-12-27_08-12-00

    While it is in pytorch - I think it would be great to surface these in side bar as in the 512 vectors currently in another tab.

    Also checkout artbreeder https://www.artbreeder.com/ it has some simple controls to switch up / infuse latent vectors.

    Side note - the nvidia labs are switching over to pytorch - https://github.com/NVlabs/stylegan2-ada/issues/32#issuecomment-748225014

    opened by johndpope 10
  • Feature suggestion - merger animation exporting

    Feature suggestion - merger animation exporting

    It would be very useful to have the ability to export the intermediate "Merge" images into a gif.

    Also as a major feature suggestion:

    • having the ability of learning and exploring meaningful latent directions
    opened by leoffx 3
Owner
K.L.
K.L.
Non-Official Pytorch implementation of "Face Identity Disentanglement via Latent Space Mapping" https://arxiv.org/abs/2005.07728 Using StyleGAN2 instead of StyleGAN

Face Identity Disentanglement via Latent Space Mapping - Implement in pytorch with StyleGAN 2 Description Pytorch implementation of the paper Face Ide

Daniel Roich 58 Dec 24, 2022
(CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic

ClassSR (CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic Paper Authors: Xiangtao Kong, Hengyuan

Xiangtao Kong 308 Jan 5, 2023
Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Implementation of Analyzing and Improving the Image Quality of StyleGAN (StyleGAN 2) in PyTorch

Kim Seonghyeon 2.2k Jan 1, 2023
[SIGGRAPH'22] StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets

[Project] [PDF] This repository contains code for our SIGGRAPH'22 paper "StyleGAN-XL: Scaling StyleGAN to Large Diverse Datasets" by Axel Sauer, Katja

null 742 Jan 4, 2023
The repository forked from NVlabs uses our data. (Differentiable rasterization applied to 3D model simplification tasks)

nvdiffmodeling [origin_code] Differentiable rasterization applied to 3D model simplification tasks, as described in the paper: Appearance-Driven Autom

Qiujie (Jay) Dong 2 Oct 31, 2022
StyleGAN2-ADA - Official PyTorch implementation

Abstract: Training generative adversarial networks (GAN) using too little data typically leads to discriminator overfitting, causing training to diverge. We propose an adaptive discriminator augmentation mechanism that significantly stabilizes training in limited data regimes.

NVIDIA Research Projects 3.2k Dec 30, 2022
StyleGAN2-ada for practice

This version of the newest PyTorch-based StyleGAN2-ada is intended mostly for fellow artists, who rarely look at scientific metrics, but rather need a working creative tool. Tested on Python 3.7 + PyTorch 1.7.1, requires FFMPEG for sequence-to-video conversions. For more explicit details refer to the original implementations.

vadim epstein 170 Nov 16, 2022
Navigating StyleGAN2 w latent space using CLIP

Navigating StyleGAN2 w latent space using CLIP an attempt to build sth with the official SG2-ADA Pytorch impl kinda inspired by Generating Images from

Mike K. 55 Dec 6, 2022
StyleGAN2 - Official TensorFlow Implementation

StyleGAN2 - Official TensorFlow Implementation

NVIDIA Research Projects 10.1k Dec 28, 2022
StyleGAN2-ADA - Official PyTorch implementation

Need Help? If you’re new to StyleGAN2-ADA and looking to get started, please check out this video series from a course Lia Coleman and I taught in Oct

Derrick Schultz 217 Jan 4, 2023
StyleGAN2 Webtoon / Anime Style Toonify

StyleGAN2 Webtoon / Anime Style Toonify Korea Webtoon or Japanese Anime Character Stylegan2 base high Quality 1024x1024 / 512x512 Generate and Transfe

null 121 Dec 21, 2022
Pretrained models for Jax/Flax: StyleGAN2, GPT2, VGG, ResNet.

Pretrained models for Jax/Flax: StyleGAN2, GPT2, VGG, ResNet.

Matthias Wright 169 Dec 26, 2022
Fine-tuning StyleGAN2 for Cartoon Face Generation

Cartoon-StyleGAN ?? : Fine-tuning StyleGAN2 for Cartoon Face Generation Abstract Recent studies have shown remarkable success in the unsupervised imag

Jihye Back 520 Jan 4, 2023
A colab notebook for training Stylegan2-ada on colab, transfer learning onto your own dataset.

Stylegan2-Ada-Google-Colab-Starter-Notebook A no thrills colab notebook for training Stylegan2-ada on colab. transfer learning onto your own dataset h

Harnick Khera 66 Dec 16, 2022
StyleGAN2 with adaptive discriminator augmentation (ADA) - Official TensorFlow implementation

StyleGAN2 with adaptive discriminator augmentation (ADA) β€” Official TensorFlow implementation Training Generative Adversarial Networks with Limited Da

NVIDIA Research Projects 1.7k Dec 29, 2022
A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

Justin 1.1k Dec 24, 2022
Pytorch implementation of Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization https://arxiv.org/abs/2008.11646

[TCSVT] Each Part Matters: Local Patterns Facilitate Cross-view Geo-localization LPN [Paper] NEWs Prerequisites Python 3.6 GPU Memory >= 8G Numpy > 1.

null 46 Dec 14, 2022
StocksMA is a package to facilitate access to financial and economic data of Moroccan stocks.

Creating easier access to the Moroccan stock market data What is StocksMA ? StocksMA is a package to facilitate access to financial and economic data

Salah Eddine LABIAD 28 Jan 4, 2023
Implementation of: "Exploring Randomly Wired Neural Networks for Image Recognition"

RandWireNN Unofficial PyTorch Implementation of: Exploring Randomly Wired Neural Networks for Image Recognition. Results Validation result on Imagenet

Seung-won Park 684 Nov 2, 2022