Official PyTorch implementation of the paper Image-Based CLIP-Guided Essence Transfer.

Related tags

Deep Learning image-generation clip stylegan2
Overview

TargetCLIP- official pytorch implementation of the paper Image-Based CLIP-Guided Essence Transfer

This repository finds a global direction in StyleGAN's space to edit images according to a target image. We transfer the essence of a target image to any source image.

Pretrained directions notebook:

Open In Colab

The notebook allows to use the directions on the sources presented in the examples. In addition, there's an option to edit your own inverted images with the pretrained directions, by uploading your latent vector to the dirs folder.

Examples:

NOTE: all the examples presented are available in our colab notebook. The recommended coefficient to use is between 0.5-1

Targets that were not inverted- The Joker and Keanu Reeves

The targets are plain images, that were not inverted, the direction optimization is initialized at random.

NOTE: for the joker, we use relatively large coefficients- 0.9-1.3

Out of domain targets- Elsa and Pocahontas

The targets are plain images that are out of the domain StyleGAN was trained on, the direction optimization is initialized at random.

Targets that were inverted- Trump

The targets are inverted images, and the latents are used as initialization for the optimization.

Credits

The code in this repo draws from the StyleCLIP code base.

Comments
  • Problem about finding direction

    Problem about finding direction

    I ran this command to reproduce your results: python3 optimization/find_dirs.py --target_path dirs/targets/avatar.jpg --dir_name results_folder_avatar --weight_decay 3e-3 --lambda_consistency 0.6 --step 1000 --lr 0.2 --num_directions 8 --num_images 8 But I got strange result as bellow: here is some results. Anything wrong with me? image

    opened by rainsoulsrx 4
  • Role of coefficients

    Role of coefficients

    Hello again,

    in these marked lines you initialize a set of coefficients to optimize over. As far as I can see, these are not mentioned in the paper. The coefficients are multiplied by the direction per source image, so I get that you want to optimize for a different scale of the direction vector per source vector. I have some questions on this:

    1. Did you try it without these coefficients?
    2. To what values do the coefficients converge to? Do they stay close to 1?
    3. You re-initialize the Adam optimizer for the coefficients for every step within the optimization, hence drastically changing the behavior of the optimizer. Is this intended or a misplacement? If it is intended, what is it used for?

    Thanks again for your work! I hope I am not too picky on this - I'm just curious about the topic of semantics in these latent spaces :-)

    https://github.com/hila-chefer/TargetCLIP/blob/b5dd2a492bf436fa26cfa4c02021a957b6a2a5ec/optimization/find_dirs.py#L123-L140

    opened by NotNANtoN 3
  • amazing work

    amazing work

    amazing work

    let me congratulate you on this incredible job, unfortunately I don't have the computing power to train targets, I'd like you to add some celebrity targets like megan fox and emilia clacke, and other celebrities. and add new ones, for people who don't have a gpu, every week if they have free time and no work is busy.

    opened by molo32 2
  • Efficiency: no recalculation of original latents needed

    Efficiency: no recalculation of original latents needed

    https://github.com/hila-chefer/TargetCLIP/blob/b5dd2a492bf436fa26cfa4c02021a957b6a2a5ec/optimization/find_dirs.py#L142-L144

    In these lines the latents are recalculated in the nested loop in every inner loop. The latents themselves stay constant though, so it can be done once outside of these loops. Only the augmented latents (latents + direction) need to be recalculated here.

    opened by NotNANtoN 2
  • ModuleNotFoundError bug colab

    ModuleNotFoundError bug colab

    steps to reproduce

    !git clone  https://github.com/hila-chefer/TargetCLIP.git

%cd /content/TargetCLIP

!python /content/TargetCLIP/optimization/find_dirs.py

Traceback (most recent call last):
  File "/content/TargetCLIP/optimization/find_dirs.py", line 9, in <module>
    from criteria.clip_loss import CLIPLoss
ModuleNotFoundError: No module named 'criteria'

    but criteria is in the root directory

    opened by loboere 2
Owner
Hila Chefer
MSc Student @ Tel Aviv University & Intern @ Microsoft's Innovation Labs
Hila Chefer
GitHub
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

CLIP-GEN [简体中文][English] 本项目在萤火二号集群上用 PyTorch 实现了论文 《CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP》。 CLIP-GEN 是一个 Language-F

null 75 Dec 29, 2022
Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space"

MotionCLIP Official Pytorch implementation of the paper "MotionCLIP: Exposing Human Motion Generation to CLIP Space". Please visit our webpage for mor

Guy Tevet 173 Dec 26, 2022
Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search

CLIP-GLaSS Repository for the paper Generating images from caption and vice versa via CLIP-Guided Generative Latent Space Search An in-browser demo is

Federico Galatolo 172 Dec 22, 2022
Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab.

CLIP-Guided-Diffusion Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab. Original colab notebooks by Ka

Nerdy Rodent 336 Dec 9, 2022
The pytorch implementation of the paper "text-guided neural image inpainting" at MM'2020

TDANet: Text-Guided Neural Image Inpainting, MM'2020 (Oral) MM | ArXiv This repository implements the paper "Text-Guided Neural Image Inpainting" by L

LisaiZhang 75 Dec 22, 2022
Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)'

SCL Introduction Code for 'Self-Guided and Cross-Guided Learning for Few-shot segmentation. (CVPR' 2021)' We evaluated our approach using two baseline

null 34 Oct 8, 2022
Transfer style api - An API to use with Tranfer Style App, where you can use two image and transfer the style

Transfer Style API It's an API to use with Tranfer Style App, where you can use

Brian Alejandro 1 Feb 13, 2022
Official implementation of the paper WAV2CLIP: LEARNING ROBUST AUDIO REPRESENTATIONS FROM CLIP

Wav2CLIP ?? WIP ?? Official implementation of the paper WAV2CLIP: LEARNING ROBUST AUDIO REPRESENTATIONS FROM CLIP ?? ?? Ho-Hsiang Wu, Prem Seetharaman

Descript 240 Dec 13, 2022
Official PyTorch implementation of "ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows"

ArtFlow Official PyTorch implementation of the paper: ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows Jie An*, Siyu Huang*, Yibing

null 123 Dec 27, 2022
code for our paper "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer"

SHOT++ Code for our TPAMI submission "Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer" that is ext

null 75 Dec 16, 2022
Sound-guided Semantic Image Manipulation - Official Pytorch Code (CVPR 2022)

?? Sound-guided Semantic Image Manipulation (CVPR2022) Official Pytorch Implementation Sound-guided Semantic Image Manipulation IEEE/CVF Conference on

CVLAB 58 Dec 28, 2022
Pytorch implementation of CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation"

MUST-GAN Code | paper The Pytorch implementation of our CVPR2021 paper "MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generat

TianxiangMa 46 Dec 26, 2022
Official PyTorch implementation of "RMGN: A Regional Mask Guided Network for Parser-free Virtual Try-on" (IJCAI-ECAI 2022)

RMGN-VITON RMGN: A Regional Mask Guided Network for Parser-free Virtual Try-on In IJCAI-ECAI 2022(short oral). [Paper] [Supplementary Material] Abstra

null 27 Dec 1, 2022
Source code for models described in the paper "AudioCLIP: Extending CLIP to Image, Text and Audio" (https://arxiv.org/abs/2106.13043)

AudioCLIP Extending CLIP to Image, Text and Audio This repository contains implementation of the models described in the paper arXiv:2106.13043. This

null 458 Jan 2, 2023
Python package to generate image embeddings with CLIP without PyTorch/TensorFlow

imgbeddings A Python package to generate embedding vectors from images, using OpenAI's robust CLIP model via Hugging Face transformers. These image em

Max Woolf 81 Jan 4, 2023
Official pytorch implementation of paper "Image-to-image Translation via Hierarchical Style Disentanglement".

HiSD: Image-to-image Translation via Hierarchical Style Disentanglement Official pytorch implementation of paper "Image-to-image Translation

null 364 Dec 14, 2022
This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations at CVPR'21. According to some product reasons, we are not planning to release the training/testing codes and models. However, we will release the dataset and the scripts to prepare the dataset.

TransFill-Reference-Inpainting This is the official repo for TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transf

Yuqian Zhou 80 Dec 8, 2022
Official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.

GLIDE This is the official codebase for running the small, filtered-data GLIDE model from GLIDE: Towards Photorealistic Image Generation and Editing w

OpenAI 2.9k Jan 4, 2023
Transfer-Learn is an open-source and well-documented library for Transfer Learning.

Transfer-Learn is an open-source and well-documented library for Transfer Learning. It is based on pure PyTorch with high performance and friendly API. Our code is pythonic, and the design is consistent with torchvision. You can easily develop new algorithms, or readily apply existing algorithms.

THUML @ Tsinghua University 2.2k Jan 3, 2023