Fewshot-face-translation-GAN - Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.

Last update: Dec 24, 2022

Overview

Few-shot face translation

A GAN based approach for one model to swap them all.

The table below shows our priliminary face-swapping results requiring one source face and <=5 target face photos. Notice that almost all of the identities, except Stephen Curry, are not in our training data (which is a subset of VGGFace2). More translation results can be found here.

Also, our model is capable of producing faces that has its gaze direction, glasses, and hiar occlusions being consistent with given source face. However, our model has suboptimal performance in terms of translating to asian faces. This is possibly due to limited representability of the feature extractor.

Src.\Tar.	Andrej Karpathy	Andrew Y. Ng	Du Fu	Elon Musk	Emilia Clarke	Geoffrey Hinton	Stephen Curry	Yann Lecun	Yoshua Benjio
Andrej Karpathy	N/A
Andrew Y. Ng		N/A
Du Fu			N/A
Elon Musk				N/A
Emilia Clarke					N/A
Geoffrey Hinton						N/A
Stephen Curry							N/A
Yann Lecun								N/A
Yoshua Benjio									N/A

I really like the Du Fu translation: such an interesting demostration how the GAN imagine the appearance of the prominent Chinese poet from just a painting.

Try in Google Colab

master branch (Jun. 2019)
dev branch (Oct. 2019)

We only provide pre-trained weights and inference script for now. Training script will be released after code cleanup.

Architecture

The above image illustrates our generator, which is a encoder-decoder based network, at test phase. Our swap-them-all approach is basically a GAN conditioned on the latent embeddings extracted from a pre-trained face recognition model. SPADE and AdaIN modules are incorporated in order to inject semantic priors to the networks.

During training phase, the input face A is heavily blurred and we train the model with resonctruction loss. Other objectives that aimed to improve translation performance while keeping semantic consistency, such as perceptual loss on rgb output and cosine similarity loss on latent embeddings, are also introduced.

Things that didn't work

We tried to distort (spline warp, downsample) the input image as in faceswap-GAN instead of masking it. However, the model did not learn proper identity translation but output face that is similar to its input.

Requirements

Python 3.6
Keras 2.2.4
TensorFlow 1.12.0 or 1.13.1

References

Comments

ipynb colab error

inputs are 256x256 already aligned faces

Building FAN for landmarks detection...
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.

WARNING:tensorflow:No training configuration found in save file: the model was *not* compiled. Compile it manually.

Done.

---------------------------------------------------------------------------

InvalidArgumentError                      Traceback (most recent call last)

<ipython-input-17-a57db805edd7> in <module>()
      1 src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet)
----> 2 tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv)

6 frames

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in __call__(self, *args, **kwargs)
   1470         ret = tf_session.TF_SessionRunCallable(self._session._session,
   1471                                                self._handle, args,
-> 1472                                                run_metadata_ptr)
   1473         if run_metadata:
   1474           proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: input image must be of non-zero size
	 [[{{node model_5/lambda_39/ResizeBilinear}}]]
  (1) Invalid argument: input image must be of non-zero size
	 [[{{node model_5/lambda_39/ResizeBilinear}}]]
	 [[model_6/lambda_41/l2_normalize/_12935]]
0 successful operations.
0 derived errors ignored.

opened by iperov 2

InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'bn10.20053864458101422_5/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

manual load data: code: fn_src = 'source.jpg' fns_tar = ['dest.jpg']

temp = []

for name in os.listdir('data_cnn/'):

name = 'data_cnn/' + name

temp.append(name)

fns_tar = temp

print(fn_src) print(fns_tar) out: source.jpg ['dest.jpg'] then while using : src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet) tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv) occur error: Building FAN for landmarks detection...

InvalidArgumentError Traceback (most recent call last) ~/anaconda2/envs/kjj/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _create_c_op(graph, node_def, inputs, control_inputs) 1627 try: -> 1628 c_op = c_api.TF_FinishOperation(op_desc) 1629 except errors.InvalidArgumentError as e:

InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'bn10.20053864458101422_5/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last) in ----> 1 src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet) 2 tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv)

~/kjj/deepfakes/fewshot-face-translation-GAN-master/utils/utils.py in get_src_inputs(fn, fd, fp, idet) 177 im = cv2.imread(fn)[..., ::-1] 178 im = auto_resize(im) --> 179 (x0, y0, x1, y1), landmarks = detect_face(im, fd) 180 aligned_im = align_image(im, x0, y0, x1, y1, landmarks) 181

~/kjj/deepfakes/fewshot-face-translation-GAN-master/utils/utils.py in detect_face(im, fd, with_landmarks) 94 landmarks = None 95 if with_landmarks: ---> 96 bboxes, landmarks = fd.detect_face(im, with_landmarks=with_landmarks) 97 else: 98 bboxes = fd.detect_face(im, with_landmarks=with_landmarks)

~/kjj/deepfakes/fewshot-face-translation-GAN-master/face_toolbox_keras/models/detector/face_detector.py in detect_face(self, image, with_landmarks) 64 if self.lmd == None: 65 print("Building FAN for landmarks detection...") ---> 66 self.build_FAN() 67 print("Done.") 68 landmarks_list = []

~/kjj/deepfakes/fewshot-face-translation-GAN-master/face_toolbox_keras/models/detector/face_detector.py in build_FAN(self) 45 46 def build_FAN(self): ---> 47 self.lmd = FANLandmarksDetector(self.lmd_weights_path) 48 49 def detect_face(self, image, with_landmarks=True):

~/kjj/deepfakes/fewshot-face-translation-GAN-master/face_toolbox_keras/models/detector/landmarks_detector.py in init(self, path_to_weights_file) 20 def init(self, path_to_weights_file=FILE_PATH+"/FAN/2DFAN-4_keras.h5"): 21 if not tf.version >= '1.13': ---> 22 self.net = load_model(path_to_weights_file) 23 else: 24 self.net = tf.keras.models.load_model(path_to_weights_file)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/saving.py in load_model(filepath, custom_objects, compile)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/saving.py in _deserialize_model(f, custom_objects, compile)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/saving.py in model_from_config(config, custom_objects)

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/layers/init.py in deserialize(config, custom_objects) 53 module_objects=globs, 54 custom_objects=custom_objects, ---> 55 printable_module_name='layer')

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/utils/generic_utils.py in deserialize_keras_object(identifier, module_objects, custom_objects, printable_module_name) 143 custom_objects=dict(list(_GLOBAL_CUSTOM_OBJECTS.items()) + 144 list(custom_objects.items()))) --> 145 with CustomObjectScope(custom_objects): 146 return cls.from_config(config['config']) 147 else:

~/anaconda2/envs/kjj/lib/python3.6/site-packages/keras/engine/network.py in from_config(cls, config, custom_objects)

opened by GingerCohle 0
Bug in the function get_square_bbox

In the function utils/utils.py->def get_square_bbox(x0, x1, y0, y1, input_img) The parameters should be in the order of x0, y0, x1, y1, such that it can match the call in x0, y0, x1, y1 = map(np.int32, get_square_bbox(x0, y0, x1, y1, im)). Otherwise the cropped face will often lose part of the jaw.

opened by Vandmoon 0
eyes jitter

eyes jitter was removed by simply using colored_parsing_map instead of parsing_map_with_iris returned from src_inputs function. was there any reason iris was being drawn on the segmentation map as i couldnt see any improvements?

opened by saksham2001aggarwal 0
added tf version to colab master notebook

Hi. Added tf version 1.13.1 (from requirements), that needed to be installed, now notebook works well. Founded some issues, that will be resolved: #30, #26, #35

opened by amrzv 4

ImportError: cannot import name 'keras_export'

I am trying to run this notebook, but I am getting below error at this line from models import FaceTranslationGANInferenceModel

Using TensorFlow backend.
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-13-e0f766e65a9c> in <module>()
----> 1 from models import FaceTranslationGANInferenceModel

13 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/initializers/initializers_v1.py in <module>()
     21 from tensorflow.python.framework import dtypes
     22 from tensorflow.python.ops import init_ops
---> 23 from tensorflow.python.util.tf_export import keras_export
     24 
     25 keras_export(v1=['keras.initializers.Zeros', 'keras.initializers.zeros'])(

ImportError: cannot import name 'keras_export'

Libraries: tensorflow-gpu==1.13.1, keras==2.2.4

opened by atinesh-s 7

$InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512] [[{{node input_14}}]]$

InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512] [[{{node input_14}}]]

InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512] [[{{node input_14}}]] In colab notebook I get the following error after this cell out = model.inference(src, mask, tar, emb_tar)

Is the error related to image size or some thing else

I am using intel-tensorflow==1.13.2 because of the tensorflow.contrib error followed by the this error

opened by shfaizan 2

Owner

GitHub

NR-GAN: Noise Robust Generative Adversarial Networks

NR-GAN: Noise Robust Generative Adversarial Networks (CVPR 2020) This repository provides PyTorch implementation for noise robust GAN (NR-GAN). NR-GAN

59 Dec 11, 2022

π-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

π-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis Project Page | Paper | Data Eric Ryan Chan*, Marco Monteiro*, Pe

375 Dec 31, 2022

Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary Differential Equations

ODE GAN (Prototype) in PyTorch Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary

15 Feb 10, 2022

Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)

Flickr-Faces-HQ Dataset (FFHQ) Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative

2.9k Dec 28, 2022

Generate high quality pictures. GAN. Generative Adversarial Networks

ESRGAN generate high quality pictures. GAN. Generative Adversarial Networks """ Super-resolution of CelebA using Generative Adversarial Networks. The

1 Dec 14, 2021

Official code for "End-to-End Optimization of Scene Layout" -- including VAE, Diff Render, SPADE for colorization (CVPR 2020 Oral)

End-to-End Optimization of Scene Layout Code release for: End-to-End Optimization of Scene Layout CVPR 2020 (Oral) Project site, Bibtex For help conta

41 Dec 9, 2022

Semantic Image Synthesis with SPADE

Semantic Image Synthesis with SPADE New implementation available at imaginaire repository We have a reimplementation of the SPADE method that is more

6.8k Nov 26, 2021

A simple PyTorch Implementation of Generative Adversarial Networks, focusing on anime face drawing.

AnimeGAN A simple PyTorch Implementation of Generative Adversarial Networks, focusing on anime face drawing. Randomly Generated Images The images are

1.2k Jan 3, 2023

House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent for Professional Architects

House-GAN++ Code and instructions for our paper: House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent

122 Dec 28, 2022

A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

3 Dec 29, 2022

Fewshot-face-translation-GAN - Generative adversarial networks integrating modules from FUNIT and SPADE for face-swapping.

Related tags

Overview

Few-shot face translation

I really like the Du Fu translation: such an interesting demostration how the GAN imagine the appearance of the prominent Chinese poet from just a painting.

Try in Google Colab

We only provide pre-trained weights and inference script for now. Training script will be released after code cleanup.

Architecture

Things that didn't work

Requirements

References

Comments

ipynb colab error

InvalidArgumentError: Shape must be rank 1 but is rank 0 for 'bn10.20053864458101422_5/cond/Reshape_4' (op: 'Reshape') with input shapes: [1,64,1,1], [].

temp = []

for name in os.listdir('data_cnn/'):

name = 'data_cnn/' + name

temp.append(name)

fns_tar = temp

print(fn_src) print(fns_tar) out: source.jpg ['dest.jpg'] then while using : src, mask, aligned_im, (x0, y0, x1, y1), landmarks = utils.get_src_inputs(fn_src, fd, fp, idet) tar, emb_tar = utils.get_tar_inputs(fns_tar, fd, fv) occur error: Building FAN for landmarks detection...

Bug in the function get_square_bbox

eyes jitter

added tf version to colab master notebook

ImportError: cannot import name 'keras_export'

InvalidArgumentError: You must feed a value for placeholder tensor 'input_14' with dtype float and shape [?,512] [[{{node input_14}}]]

Owner

NR-GAN: Noise Robust Generative Adversarial Networks

π-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Partial implementation of ODE-GAN technique from the paper Training Generative Adversarial Networks by Solving Ordinary Differential Equations

Flickr-Faces-HQ (FFHQ) is a high-quality image dataset of human faces, originally created as a benchmark for generative adversarial networks (GAN)

Generate high quality pictures. GAN. Generative Adversarial Networks

Official code for "End-to-End Optimization of Scene Layout" -- including VAE, Diff Render, SPADE for colorization (CVPR 2020 Oral)

Semantic Image Synthesis with SPADE

A simple PyTorch Implementation of Generative Adversarial Networks, focusing on anime face drawing.

House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent for Professional Architects

A method that utilized Generative Adversarial Network (GAN) to interpret the black-box deep image classifier models by PyTorch.

HiFi-GAN: High Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks

Pytorch implementation of paper "Learning Co-segmentation by Segment Swapping for Retrieval and Discovery"

Code of 3D Shape Variational Autoencoder Latent Disentanglement via Mini-Batch Feature Swapping for Bodies and Faces

Code for the paper "Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks"

Official Implementation of Swapping Autoencoder for Deep Image Manipulation (NeurIPS 2020)

Official PyTorch Implementation for InfoSwap: Information Bottleneck Disentanglement for Identity Swapping

Pytorch Implementation of Continual Learning With Filter Atom Swapping (ICLR'22 Spolight) Paper

Implementation for Paper "Inverting Generative Adversarial Renderer for Face Reconstruction"