VQGAN+CLIP Colab Notebook with user-friendly interface.

Overview

VQGAN+CLIP and other image generation system

VQGAN+CLIP Colab Notebook with user-friendly interface.

Latest Notebook: Open In Colab

Mse regulized zquantize Notebook: Open In Colab

Zooming (Latest release with few addons)(W.I.P): Open In Colab

PixelDrawer: Open In Colab

Pixray Panorama Demo: Open In Colab

Citations

@misc{unpublished2021clip,
    title  = {CLIP: Connecting Text and Images},
    author = {Alec Radford, Ilya Sutskever, Jong Wook Kim, Gretchen Krueger, Sandhini Agarwal},
    year   = {2021}
}
@misc{esser2020taming,
      title={Taming Transformers for High-Resolution Image Synthesis}, 
      author={Patrick Esser and Robin Rombach and Björn Ommer},
      year={2020},
      eprint={2012.09841},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Katherine Crowson - https://github.com/crowsonkb

Public Domain images from Open Access Images at the Art Institute of Chicago - https://www.artic.edu/open-access/open-access-images

Comments
  • I get these errors in pixeldraw:

    I get these errors in pixeldraw:

    iter: 0, loss: 2.74, losses: 0.985, 0.882, 0.876 (-0=>2.743) Oops: runtime error: radix_sort: failed on 1st step: cudaErrorInvalidDeviceFunction: invalid device function Try reducing --num-cuts to save memory

    RuntimeError Traceback (most recent call last) in () 45 settings = pixray.apply_settings() 46 pixray.do_init(settings) ---> 47 pixray.do_run(settings) 48 #@markdown ---

    6 frames /usr/local/lib/python3.7/dist-packages/diffvg-0.0.1-py3.7-linux-x86_64.egg/pydiffvg/render_pytorch.py in backward(ctx, grad_img) 707 use_prefiltering, 708 diffvg.float_ptr(eval_positions.data_ptr()), --> 709 eval_positions.shape[0]) 710 time_elapsed = time.time() - start 711 global print_timing

    RuntimeError: radix_sort: failed on 1st step: cudaErrorInvalidDeviceFunction: invalid device function

    and

    AttributeError Traceback (most recent call last) in () 32 33 # these are good settings for pixeldraw ---> 34 pixray.reset_settings() 35 pixray.add_settings(prompts=prompts, aspect=aspect) 36 pixray.add_settings(quality="better", scale=2.5)

    AttributeError: module 'pixray' has no attribute 'reset_settings'

    opened by AstroNaut244 1
  • Message on downloading models part says

    Message on downloading models part says "Could not resolve host: mirror.io.community" which makes the execution fail.

    Message on downloading models part says "Could not resolve host: mirror.io.community" which makes the execution fail. When it used to work, you'd press download model and it would give you a whole message saying the time left and how much time it has already been working, etc. Now the download finishes in less than a second and the only thing it says is "Message on downloading models part says "Could not resolve host: mirror.io.community" which makes the execution fail." I've tried multiple different versions of the same code, different accounts, reset it multiple times, still to no avail. Almost all of the models don't work besides celebahq I think. I've been doing the same thing for almost a month now, and just a couple days ago it started to not work. I have no idea what's happening. I'm doing everything right. Please fix this!!! Pictures included is the failed execution and the main message I'm talking about. Also my GPU. screenshot 2021-08-22 at 13 13 59

    screenshot 2021-08-22 at 13 13 17 screenshot 2021-08-22 at 13 10 14
    opened by whyiseverythingbad 1
  • Make a video of the results Error

    Make a video of the results Error

    Hi Team,

    I am getting the below error when trying to stitch the images into a video any help would me welcome.


    RuntimeError Traceback (most recent call last) in () 52 "If these steps do not work, please post the traceback in the github." 53 ) ---> 54 raise RuntimeError(stderr) 55 else: 56 print("The video is ready")

    RuntimeError: b"ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers\n built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)\n configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared\n libavutil 55. 78.100 / 55. 78.100\n libavcodec 57.107.100 / 57.107.100\n libavformat 57. 83.100 / 57. 83.100\n libavdevice 57. 10.100 / 57. 10.100\n libavfilter 6.107.100 / 6.107.100\n libavresample 3. 7. 0 / 3...

    opened by TwiistedTank 0
  • invalid output path in pixeldraw

    invalid output path in pixeldraw

    prompts = "Minecraft. #pixelart" #@param {type:"string"} output_path = "/content/output

    correct one:

    prompts = "Minecraft. #pixelart" #@param {type:"string"} output_path = "/content/output"

    opened by AstroNaut244 0
  • Update VQGAN+CLIP(Updated).ipynb

    Update VQGAN+CLIP(Updated).ipynb

    Hello! I noticed the link is incorrect. Also, seems like the URL once fixed is not working as intended to point the right article so I changed the URL as whole.

    opened by Jakeukalane 0
  • Runtime error when trying to create video

    Runtime error when trying to create video

    i know this issue has been posted already, but im doubling down with the hopes it gets noticed

    getting this error when trying to create a video

    b"ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers\n built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)\n configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared\n libavutil 55. 78.100 / 55. 78.100\n libavcodec 57.107.100 / 57.107.100\n libavformat 57. 83.100 / 57. 83.100\n libavdevice 57. 10.100 / 57. 10.100\n libavfilter 6.107.100 / 6.107.100\n libavresample 3. 7. 0 / 3. 7. 0\n libswscale 4. 8.100 / 4. 8.100\n libswresample 2. 9.100 / 2. 9.100\n libpostproc 54. 7.100 / 54. 7.100\nInput #0, image2, from '/content/gdrive/MyDrive/vqgan/steps/%04d.png':\n Duration: 00:00:03.96, start: 0.000000, bitrate: N/A\n Stream #0:0: Video: png, rgb24(pc), 640x352, 25 fps, 25 tbr, 25 tbn, 25 tbc\n[NULL @ 0x55f0bb768c00] Unable to find a suitable output format for '/content/gdrive/MyDrive/vqgan'\n/content/gdrive/MyDrive/vqgan: Invalid argument\n" You may be able to avoid this error by backing up the frames,restarting the notebook, and running only the video synthesis cells,or by decreasing the resolution of the image generation steps. If you restart the notebook, you will have to define the filepath manuallyby adding filepath = 'PATH_TO_THE_VIDEO' to the beginning of this cell. If these steps do not work, please post the traceback in the github.

    RuntimeError Traceback (most recent call last) in () 55 "If these steps do not work, please post the traceback in the github." 56 ) ---> 57 raise RuntimeError(stderr) 58 else: 59 print("The video is ready")

    RuntimeError: b"ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers\n built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)\n configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared\n libavutil 55. 78.100 / 55. 78.100\n libavcodec 57.107.100 / 57.107.100\n libavformat 57. 83.100 / 57. 83.100\n libavdevice 57. 10.100 / 57. 10.100\n libavfilter 6.107.100 / 6.107.100\n libavresample 3. 7. 0 / 3...

    opened by jkgchristmas 0
  • PixelDraw Panorama Demo Notebook - torchtext undefined symbol

    PixelDraw Panorama Demo Notebook - torchtext undefined symbol

    I am currently getting this error from the first frame cell of the pixel draw panorama notebook. My setup was successful, I restarted my runtime, and ran it again. I think this may be caused by a version mismatch on the pytorch side of things as was referenced here: https://github.com/NVIDIA/NeMo/issues/3176. Thank you so much for all your work, these notebooks are amazing!

    Traceback

    ---------------------------------------------------------------------------
    ImportError                               Traceback (most recent call last)
    <ipython-input-3-fc63159cf4cc> in <module>()
         42 
         43 # Simple setup
    ---> 44 import pixray
         45 
         46 # these are good settings for pixeldraw
    
    12 frames
    /usr/local/lib/python3.7/dist-packages/torchtext/vocab/vocab_factory.py in <module>()
          2 from typing import Dict, Iterable, Optional, List
          3 from collections import Counter, OrderedDict
    ----> 4 from torchtext._torchtext import (
          5     Vocab as VocabPybind,
          6 )
    
    ImportError: /usr/local/lib/python3.7/dist-packages/torchtext/_torchtext.so: undefined symbol: _ZTVN5torch3jit6MethodE
    
    opened by thehappydinoa 0
  • filepath not defined

    filepath not defined

    Hi I recently began making videos with VQGAN+clip Zooming - but I keep getting the same error, when I try to create the video in the video creation cell.

    first I get the "filepath not defined" error message:

    NameError Traceback (most recent call last) in () 38 '-preset', 39 'veryslow', ---> 40 filepath 41 ] 42

    NameError: name 'filepath' is not defined


    Then I try to write a line to define the filepath, like this:

    filepath = f'{working_dir}'

    but then I get this error, and I am stuck:

    b"ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers\n built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)\n configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared\n libavutil 55. 78.100 / 55. 78.100\n libavcodec 57.107.100 / 57.107.100\n libavformat 57. 83.100 / 57. 83.100\n libavdevice 57. 10.100 / 57. 10.100\n libavfilter 6.107.100 / 6.107.100\n libavresample 3. 7. 0 / 3. 7. 0\n libswscale 4. 8.100 / 4. 8.100\n libswresample 2. 9.100 / 2. 9.100\n libpostproc 54. 7.100 / 54. 7.100\nInput #0, image2, from '/content/gdrive/MyDrive/AI-ARBEJDE/ZOOMING/ZUCKERBERG-WILLIAMS/steps/%04d.png':\n Duration: 00:00:12.00, start: 0.000000, bitrate: N/A\n Stream #0:0: Video: png, rgb24(pc), 480x256, 25 fps, 25 tbr, 25 tbn, 25 tbc\n[NULL @ 0x55f3afeaec00] Unable to find a suitable output format for '/content/gdrive/MyDrive/AI-ARBEJDE/ZOOMING/ZUCKERBERG-WILLIAMS'\n/content/gdrive/MyDrive/AI-ARBEJDE/ZOOMING/ZUCKERBERG-WILLIAMS: Invalid argument\n" You may be able to avoid this error by backing up the frames,restarting the notebook, and running only the video synthesis cells,or by decreasing the resolution of the image generation steps. If you restart the notebook, you will have to define the filepath manuallyby adding filepath = 'PATH_TO_THE_VIDEO' to the beginning of this cell. If these steps do not work, please post the traceback in the github.

    RuntimeError Traceback (most recent call last) in () 54 "If these steps do not work, please post the traceback in the github." 55 ) ---> 56 raise RuntimeError(stderr) 57 else: 58 print("The video is ready")

    RuntimeError: b"ffmpeg version 3.4.8-0ubuntu0.2 Copyright (c) 2000-2020 the FFmpeg developers\n built with gcc 7 (Ubuntu 7.5.0-3ubuntu1~18.04)\n configuration: --prefix=/usr --extra-version=0ubuntu0.2 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-librsvg --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared\n libavutil 55. 78.100 / 55. 78.100\n libavcodec 57.107.100 / 57.107.100\n libavformat 57. 83.100 / 57. 83.100\n libavdevice 57. 10.100 / 57. 10.100\n libavfilter 6.107.100 / 6.107.100\n libavresample 3. 7. 0 / 3...

    ...

    I don't know what to do.. can anyone help?

    opened by sanssoleilfilm 1
  • PixelDrawer exception when executing

    PixelDrawer exception when executing "Setup"

    P100 with 16 Gigs of GPU Ram. Use: "pixeldraw".

    ` KeyError Traceback (most recent call last) in () 43 # pixray.add_settings(iterations=500, display_every=50) 44 ---> 45 settings = pixray.apply_settings() 46 pixray.do_init(settings) 47 pixray.do_run(settings)

    /content/pixray/pixray.py in apply_settings() 1681 1682 vq_parser = setup_parser(vq_parser) -> 1683 class_table[settings_core.drawer].add_settings(vq_parser) 1684 1685 # TODO: this is slighly sloppy and better would be to add settings of loss functions

    KeyError: 'pixeldraw' `

    Thanks.

    opened by arlojeremy 0
Owner
Justin John
Justin John
Traditional deepdream with VQGAN+CLIP and optical flow. Ready to use in Google Colab

VQGAN-CLIP-Video cat.mp4 policeman.mp4 schoolboy.mp4 forsenBOG.mp4

null 23 Oct 26, 2022
A colab notebook for training Stylegan2-ada on colab, transfer learning onto your own dataset.

Stylegan2-Ada-Google-Colab-Starter-Notebook A no thrills colab notebook for training Stylegan2-ada on colab. transfer learning onto your own dataset h

Harnick Khera 66 Dec 16, 2022
Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized

VQGAN-CLIP-Docker About Zero-Shot Text-to-Image Generation VQGAN+CLIP Dockerized This is a stripped and minimal dependency repository for running loca

Kevin Costa 73 Sep 11, 2022
CLIP + VQGAN / PixelDraw

clipit Yet Another VQGAN-CLIP Codebase This started as a fork of @nerdyrodent's VQGAN-CLIP code which was based on the notebooks of @RiversWithWings a

dribnet 276 Dec 12, 2022
Streamlit Tutorial (ex: stock price dashboard, cartoon-stylegan, vqgan-clip, stylemixing, styleclip, sefa)

Streamlit Tutorials Install pip install streamlit Run cd [directory] streamlit run app.py --server.address 0.0.0.0 --server.port [your port] # http:/

Jihye Back 30 Jan 6, 2023
Text2Art is an AI art generator powered with VQGAN + CLIP and CLIPDrawer models

Text2Art is an AI art generator powered with VQGAN + CLIP and CLIPDrawer models. You can easily generate all kind of art from drawing, painting, sketch, or even a specific artist style just using a text input. You can also specify the dimensions of the image. The process can take 3-20 mins and the results will be emailed to you.

Muhammad Fathy Rashad 643 Dec 30, 2022
An architecture that makes any doodle realistic, in any specified style, using VQGAN, CLIP and some basic embedding arithmetics.

Sketch Simulator An architecture that makes any doodle realistic, in any specified style, using VQGAN, CLIP and some basic embedding arithmetics. See

null 12 Dec 18, 2022
Making a music video with Wav2CLIP and VQGAN-CLIP

music2video Overview A repo for making a music video with Wav2CLIP and VQGAN-CLIP. The base code was derived from VQGAN-CLIP The CLIP embedding for au

Joel Jang | 장요엘 163 Dec 26, 2022
Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab.

CLIP-Guided-Diffusion Just playing with getting CLIP Guided Diffusion running locally, rather than having to use colab. Original colab notebooks by Ka

Nerdy Rodent 336 Dec 9, 2022
A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

A Jupyter notebook to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

Eugenio Herrera 175 Dec 29, 2022
FuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space OptimizationFuseDream: Training-Free Text-to-Image Generationwith Improved CLIP+GAN Space Optimization

FuseDream This repo contains code for our paper (paper link): FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizat

XCL 191 Dec 31, 2022
CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP

CLIP-GEN [简体中文][English] 本项目在萤火二号集群上用 PyTorch 实现了论文 《CLIP-GEN: Language-Free Training of a Text-to-Image Generator with CLIP》。 CLIP-GEN 是一个 Language-F

null 75 Dec 29, 2022
A user-friendly research and development tool built to standardize RL competency assessment for custom agents and environments.

Built with ❤️ by Sam Showalter Contents Overview Installation Dependencies Usage Scripts Standard Execution Environment Development Environment Benchm

SRI-AIC 1 Nov 18, 2021
Try out deep learning models online on Google Colab

Try out deep learning models online on Google Colab

Erdene-Ochir Tuguldur 1.5k Dec 27, 2022
Monitor your ML jobs on mobile devices📱, especially for Google Colab / Kaggle

TF Watcher TF Watcher is a simple to use Python package and web app which allows you to monitor ?? your Machine Learning training or testing process o

Rishit Dagli 54 Nov 1, 2022
ROS-UGV-Control-Interface - Control interface which can be used in any UGV

ROS-UGV-Control-Interface Cam Closed: Cam Opened:

Ahmet Fatih Akcan 1 Nov 4, 2022
Monify: an Expense tracker Program implemented in a Graphical User Interface that allows users to keep track of their expenses

?? MONIFY (EXPENSE TRACKER PRO) ?? Description Monify is an Expense tracker Program implemented in a Graphical User Interface allows users to add inco

Moyosore Weke 1 Dec 14, 2021
NUANCED is a user-centric conversational recommendation dataset that contains 5.1k annotated dialogues and 26k high-quality user turns.

NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions Overview NUANCED is a user-centric conversational recommen

Facebook Research 18 Dec 28, 2021