Official PyTorch implementation of the paper Image-Based CLIP-Guided Essence Transfer.

Last update: Dec 13, 2022

Overview

TargetCLIP- official pytorch implementation of the paper Image-Based CLIP-Guided Essence Transfer

This repository finds a global direction in StyleGAN's space to edit images according to a target image. We transfer the essence of a target image to any source image.

Pretrained directions notebook:

The notebook allows to use the directions on the sources presented in the examples. In addition, there's an option to edit your own inverted images with the pretrained directions, by uploading your latent vector to the dirs folder.

Examples:

NOTE: all the examples presented are available in our colab notebook. The recommended coefficient to use is between 0.5-1

Targets that were not inverted- The Joker and Keanu Reeves

The targets are plain images, that were not inverted, the direction optimization is initialized at random.

NOTE: for the joker, we use relatively large coefficients- 0.9-1.3

Out of domain targets- Elsa and Pocahontas

The targets are plain images that are out of the domain StyleGAN was trained on, the direction optimization is initialized at random.

Targets that were inverted- Trump

The targets are inverted images, and the latents are used as initialization for the optimization.

Credits

The code in this repo draws from the StyleCLIP code base.

Comments

Problem about finding direction

I ran this command to reproduce your results: python3 optimization/find_dirs.py --target_path dirs/targets/avatar.jpg --dir_name results_folder_avatar --weight_decay 3e-3 --lambda_consistency 0.6 --step 1000 --lr 0.2 --num_directions 8 --num_images 8 But I got strange result as bellow: here is some results. Anything wrong with me?

opened by rainsoulsrx 4
Role of coefficients
Hello again,

in these marked lines you initialize a set of coefficients to optimize over. As far as I can see, these are not mentioned in the paper. The coefficients are multiplied by the direction per source image, so I get that you want to optimize for a different scale of the direction vector per source vector. I have some questions on this:

Did you try it without these coefficients?

To what values do the coefficients converge to? Do they stay close to 1?

You re-initialize the Adam optimizer for the coefficients for every step within the optimization, hence drastically changing the behavior of the optimizer. Is this intended or a misplacement? If it is intended, what is it used for?

Thanks again for your work! I hope I am not too picky on this - I'm just curious about the topic of semantics in these latent spaces :-)

https://github.com/hila-chefer/TargetCLIP/blob/b5dd2a492bf436fa26cfa4c02021a957b6a2a5ec/optimization/find_dirs.py#L123-L140
opened by NotNANtoN 3
amazing work

amazing work

let me congratulate you on this incredible job, unfortunately I don't have the computing power to train targets, I'd like you to add some celebrity targets like megan fox and emilia clacke, and other celebrities. and add new ones, for people who don't have a gpu, every week if they have free time and no work is busy.

opened by molo32 2
Efficiency: no recalculation of original latents needed

https://github.com/hila-chefer/TargetCLIP/blob/b5dd2a492bf436fa26cfa4c02021a957b6a2a5ec/optimization/find_dirs.py#L142-L144

In these lines the latents are recalculated in the nested loop in every inner loop. The latents themselves stay constant though, so it can be done once outside of these loops. Only the augmented latents (latents + direction) need to be recalculated here.

opened by NotNANtoN 2

ModuleNotFoundError bug colab

steps to reproduce

!git clone  https://github.com/hila-chefer/TargetCLIP.git

%cd /content/TargetCLIP

!python /content/TargetCLIP/optimization/find_dirs.py

Traceback (most recent call last):
  File "/content/TargetCLIP/optimization/find_dirs.py", line 9, in <module>
    from criteria.clip_loss import CLIPLoss
ModuleNotFoundError: No module named 'criteria'

but criteria is in the root directory

opened by loboere 2

Owner

Hila Chefer

MSc Student @ Tel Aviv University & Intern @ Microsoft's Innovation Labs

GitHub

CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

CLIP-GEN [简体中文][English] 本项目在萤火二号集群上用 PyTorch 实现了论文《CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP》。 CLIP-GEN 是一个 Language-F

75 Dec 29, 2022

Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space"

MotionCLIP Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space". Please visit our webpage for mor

173 Dec 26, 2022

Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

CLIP-GLaSS Repository for the paper Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search An in-browser demo is

172 Dec 22, 2022

Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab.

CLIP-Guided-Diffusion Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab. Original colab notebooks by Ka

336 Dec 9, 2022

The pytorch implementation of the paper "text-guided neural image inpainting" at MM'2020

TDANet: Text-Guided Neural Image Inpainting, MM'2020 (Oral) MM | ArXiv This repository implements the paper "Text-Guided Neural Image Inpainting" by L

75 Dec 22, 2022

Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

SCL Introduction Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)' We evaluated our approach using two baseline

34 Oct 8, 2022

Transfer style api - An API to use with Tranfer Style App, where you can use two image and transfer the style

Transfer Style API It's an API to use with Tranfer Style App, where you can use

1 Feb 13, 2022

Official implementation of the paper WAV2CLIP: LEARNING ROBUST AUDIO REPRESENTATIONS FROM CLIP

Wav2CLIP ?? WIP ?? Official implementation of the paper WAV2CLIP: LEARNING ROBUST AUDIO REPRESENTATIONS FROM CLIP ?? ?? Ho-Hsiang Wu, Prem Seetharaman

240 Dec 13, 2022

Official PyTorch implementation of "ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows"

ArtFlow Official PyTorch implementation of the paper: ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows Jie An*, Siyu Huang*, Yibing

123 Dec 27, 2022

code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

SHOT++ Code for our TPAMI submission "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer" that is ext

75 Dec 16, 2022

Sound-guided Semantic Image Manipulation - Official Pytorch Code (CVPR 2022)

?? Sound-guided Semantic Image Manipulation (CVPR2022) Official Pytorch Implementation Sound-guided Semantic Image Manipulation IEEE/CVF Conference on

58 Dec 28, 2022

Pytorch implementation of CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation"

MUST-GAN Code | paper The Pytorch implementation of our CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generat

46 Dec 26, 2022

Official PyTorch implementation of "RMGN: A Regional Mask Guided Network for Parser-free Virtual Try-on" (IJCAI-ECAI 2022)

RMGN-VITON RMGN: A Regional Mask Guided Network for Parser-free Virtual Try-on In IJCAI-ECAI 2022(short oral). [Paper] [Supplementary Material] Abstra

27 Dec 1, 2022

Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

458 Jan 2, 2023

Python package to generate image embeddings with CLIP without PyTorch/TensorFlow

imgbeddings A Python package to generate embedding vectors from images, using OpenAI's robust CLIP model via Hugging Face transformers. These image em

81 Jan 4, 2023

Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

364 Dec 14, 2022

This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations at CVPR'21. According to some product reasons, we are not planning to release the training/testing codes and models. However, we will release the dataset and the scripts to prepare the dataset.

TransFill-Reference-Inpainting This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transf

80 Dec 8, 2022

Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

2.9k Jan 4, 2023

Transfer-Learn is an open-source and well-documented library for Transfer Learning.

Transfer-Learn is an open-source and well-documented library for Transfer Learning. It is based on pure PyTorch with high performance and friendly API. Our code is pythonic, and the design is consistent with torchvision. You can easily develop new algorithms, or readily apply existing algorithms.

2.2k Jan 3, 2023