Generative Art Using Neural Visual Grammars and Dual Encoders

DeepMind

Last update: Jan 5, 2023

Related tags

Deep Learning arnheim

Overview

Generative Art Using Neural Visual Grammars and Dual Encoders

Arnheim 1

The original algorithm from the paper Generative Art Using Neural Visual Grammars and Dual Encoders running on 1 GPU allows optimization of any image using a genetic algorithm. This is much more general but much slower than using Arnheim 2 which uses gradients.

Arnheim 2

A reimplementation of the Arnheim 1 generative architecture in the CLIPDraw framework allowing optimization of its parameters using gradients. Much more efficient than Arnheim 1 above but requires differentiating through the image itself.

Usage

Usage instructions are included in the Colabs which open and run on the free-to-use Google Colab platform - just click the buttons below! Improved performance and longer timeouts are available with Colab Pro.

Arnheim 1

Arnheim 2

Citing this work

If you use this code (or any derived code), data or these models in your work, please cite the relevant accompanying paper.

@misc{fernando2021genart,
      title={Generative Art Using Neural Visual Grammars and Dual Encoders},
      author={Chrisantha Fernando and S. M. Ali Eslami and Jean-Baptiste Alayrac and Piotr Mirowski and Dylan Banarse and Simon Osindero}
      year={2021},
      eprint={2105.00162},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Disclaimer

This is not an official Google product.

Other works may be copyright of the authors of such work.

Comments

CLIPDraw license

The README says:

CLIPDraw provided under license, Copyright 2021 Kevin Frans.

I am slightly confused as to what this means.

Can arnheim_2.ipynb be used under the terms of the Apache 2.0 license? Is CLIPDraw provided under the Apache 2.0 license?

opened by Aspie96 2
Kernel keeps on dying when trying arnheim_3.ipynb

When trying to run arnheim_3.ipynb on local machine, the kernel keeps on dying when executing output = ct.loop(). There no errors before this piece of code. Could you please tell me how to resolve it?

opened by vidit98 1
Tried arnheim2.ipynb - See Error

Cell "Preliminaries" works fine in Colab. The next cell throws an error:

`Torch version: 1.9.0+cu111

AttributeError Traceback (most recent call last)

in () 18 19 import clip ---> 20 import pydiffvg 21 22 os.environ["FFMPEG_BINARY"] = "ffmpeg"

2 frames

/content/diffvg/pydiffvg/render_pytorch.py in RenderFunction() 25 shapes, 26 shape_groups, ---> 27 filter = pydiffvg.PixelFilter(type = diffvg.FilterType.box, 28 radius = torch.tensor(0.5)), 29 output_type = OutputType.color,

AttributeError: module 'diffvg' has no attribute 'FilterType' `

Never mind. Can close. I didn't see step 4. Best if you can get the user to click on a button to restart run time in the cell flow.

opened by metaphorz 1

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

🦩 Flamingo - Pytorch Implementation of Flamingo, state-of-the-art few-shot visual question answering attention net, in Pytorch. It will include the p

630 Dec 28, 2022

PyTorch implementation of Neural Dual Contouring.

NDC PyTorch implementation of Neural Dual Contouring. Citation We are still writing the paper while adding more improvements and applications. If you

140 Dec 26, 2022

Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

Minimal PyTorch implementation of Generative Latent Optimization This is a reimplementation of the paper Piotr Bojanowski, Armand Joulin, David Lopez-

117 Nov 27, 2022

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

CLIP-ViL In our paper "How Much Can CLIP Benefit Vision-and-Language Tasks?", we show the improvement of CLIP features over the traditional resnet fea

310 Dec 28, 2022

PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning This repository is for EMSRDPN introduced in the foll

7 Feb 10, 2022

Memoized coduals - Shows that it is possible to implement reverse mode autodiff using a variation on the dual numbers called the codual numbers

The dual numbers can do efficient autodiff! The codual numbers are a simple meth

2 Dec 19, 2022

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

892 Dec 28, 2022

[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

SADRNet Paper link: SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction Requirements python

Multimedia Computing Group, Nanjing University

99 Dec 30, 2022

[ICCV'2021] Image Inpainting via Conditional Texture and Structure Dual Generation

122 Dec 11, 2022

Generative Art Using Neural Visual Grammars and Dual Encoders

Related tags

Overview

Generative Art Using Neural Visual Grammars and Dual Encoders

Arnheim 1

Arnheim 2

Usage

Citing this work

Disclaimer

You might also like...

Implementation of 🦩 Flamingo, state-of-the-art few-shot visual question answering attention net out of Deepmind, in Pytorch

PyTorch implementation of Neural Dual Contouring.

Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

improvement of CLIP features over the traditional resnet features on the visual question answering, image captioning, navigation and visual entailment tasks.

PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Memoized coduals - Shows that it is possible to implement reverse mode autodiff using a variation on the dual numbers called the codual numbers

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

[TIP 2021] SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction

[ICCV'2021] Image Inpainting via Conditional Texture and Structure Dual Generation

Comments

CLIPDraw license

Kernel keeps on dying when trying arnheim_3.ipynb

Tried arnheim2.ipynb - See Error

Owner

DeepMind

Final project code: Implementing MAE with downscaled encoders and datasets, for ESE546 FA21 at University of Pennsylvania

Catch-all collection of generative art made using processing

This repo holds code for TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation

TSP: Temporally-Sensitive Pretraining of Video Encoders for Localization Tasks

RE3: State Entropy Maximization with Random Encoders for Efficient Exploration

PyTorch Implement of Context Encoders: Feature Learning by Inpainting

[ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

PaddleViT: State-of-the-art Visual Transformer and MLP Models for PaddlePaddle 2.0+

Python code to generate art with Generative Adversarial Network

A fast, dataset-agnostic, deep visual search engine for digital art history