A Deep Learning based project for creating line art portraits.

Related tags

Deep Learning machine-learning ai deep-learning creative-coding colab generative-art movie-poster portraits nogan nogan-training art-portraits
Overview

ArtLine

Open in RunwayML Badge

forthebadge

The main aim of the project is to create amazing line art portraits.

Sounds Intresting,let's get to the pictures!!

Model-(Smooth)

Model-(Quality)

Click on the below image to know more about colab demo, credits to Bhavesh Bhatt for the amazing Youtube video.

Exciting update

Highlights

Example Images

bohemian rhapsody movie , Rami Malek American actor

bohemian

Photo by Maxim from Pexels

Imgur

Friends, TV show.

Friends

Keanu Reeves, Canadian actor.

Keanu

Hrithik Roshan

Hrithik

Alita: Battle Angel

Alita

Virat Kohli, Indian cricketer

Virat

Photo by Anastasiya Gepp from Pexels

Imgur

Interstellar

Interstellar

Pexels Portrait, Model

Imgur

Beyoncé, American singer

Beyoncé

Cartoonize

Lets cartoonize the lineart portraits, Check Toon-Me https://github.com/vijishmadhavan/Toon-Me.

Skrillex , American DJ

Imgur

Tom Hanks, Actor

Imgur

Line Art

The amazing results that the model has produced has a secret sauce to it. The initial model couldn't create the sort of output I was expecting, it mostly struggled with recognizing facial features. Even though (https://github.com/yiranran/APDrawingGAN) produced great results it had limitations like (frontal face photo similar to ID photo, preferably with clear face features, no glasses and no long fringe.) I wanted to break-in and produce results that could recognize any pose. Achieving proper lines around the face, eyes, lips and nose depends on the data you give the model. APDrawing dataset alone was not enough so I had to combine selected photos from Anime sketch colorization pair dataset. The combined dataset helped the model to learn the lines better.

Movie Poster created using ArtLine.

The movie poster was created using ArtLine in no time , it's not as good as it should be but I'm not an artist.

Poster

Poster

Technical Details

Surprise!! No critic,No GAN. GAN did not make much of a difference so I was happy with No GAN.

The mission was to create something that converts any personal photo into a line art. The initial efforts have helped to recognize lines, but still the model has to improve a lot with shadows and clothes. All my efforts are to improve the model and make line art a click away.

Imgur

Dataset

APDrawing dataset

Anime sketch colorization pair dataset

APDrawing data set consits of mostly close-up portraits so the model would struggle to recogonize cloths,hands etc. For this purpose selected images from Anime sketch colorization pair were used.

Going Forward

I hope I was clear, going forward would like to improve the model further as it still struggles with random backgrounds(I'm creating a custom dataset to address this issue). Cartoonizing the image was never part of the project, but somehow it came up and it did okay!! Still lots to improve. Ill release the cartoonize model when it looks impressive enough to show off.

I will be constantly upgrading the project for the foreseeable future.

Getting Started Yourself

The easiest way to get started is to simply try out on Colab: https://colab.research.google.com/github/vijishmadhavan/Light-Up/blob/master/ArtLine(Try_it_on_Colab).ipynb

Installation Details

This project is built around the wonderful Fast.AI library.

  • fastai==1.0.61 (and its dependencies). Please dont install the higher versions
  • PyTorch 1.6.0 Please don't install the higher versions

Limitations

  • Getting great output depends on Lighting, Backgrounds,Shadows and the quality of photos. You'll mostly get good results in the first go but there are chances for issues as well. The model is not there yet, it still needs to be tweaked to reach out to all the consumers. It might be useful for "AI Artisits/ Artists who can bring changes to the final output.

  • The model confuses shadows with hair, something that I'm trying to solve.

  • It does bad with low quality images(below 500px).

  • I'm not a coder, bear with me for the bad code and documentation. Will make sure that I improve with upcoming updates.

Updates

Get more updates on Twitter

Mail me @ [email protected]

Acknowledgments

License

All code in this repository is under the MIT license as specified by the LICENSE file.

Comments
  • No such colab notebook

    No such colab notebook

    Getting Started Yourself

    The easiest way to get started is to simply try out on Colab: https://colab.research.google.com/github/vijishmadhavan/Light-Up/blob/master/ArtLine(Try_it_on_Colab).ipynb

    seems the notebook name changed but it hasn't been updated in the doc above.

    Thanks.

    opened by chengweiv5 3
  • How to load model on Flask app?

    How to load model on Flask app?

    I've got this working on the Google collab document, but am running into issues with Flask with the error AttributeError: Can't get attribute 'FeatureLoss' on <module '__main__' from '/home/ubuntu/anaconda3/bin/flask'>. I believe this is related to https://discuss.pytorch.org/t/error-loading-saved-model/8371 and https://stackoverflow.com/questions/27732354/unable-to-load-files-using-pickle-and-multiple-modules where some solutions have been suggested, but I can't work out how to get it working in this case.

    Basic Flask app (this is of course incomplete, but throws the AttributeError error when trying to load the model):

    from flask import Flask
app = Flask(__name__)

import fastai
import time
from fastai.vision import *
from fastai.utils.mem import *
from fastai.vision import open_image, load_learner, image, torch
import numpy as np
import urllib.request
import PIL.Image
from io import BytesIO
import torchvision.transforms as T
from PIL import Image
import requests
from io import BytesIO
import fastai
from fastai.vision import *
from fastai.utils.mem import *
from fastai.vision import open_image, load_learner, image, torch
import numpy as np
import urllib.request
import PIL.Image
from io import BytesIO
import torchvision.transforms as T

class FeatureLoss(nn.Module):
    def __init__(self, m_feat, layer_ids, layer_wgts):
        super().__init__()
        self.m_feat = m_feat
        self.loss_features = [self.m_feat[i] for i in layer_ids]
        self.hooks = hook_outputs(self.loss_features, detach=False)
        self.wgts = layer_wgts
        self.metric_names = ['pixel',] + [f'feat_{i}' for i in range(len(layer_ids))
              ] + [f'gram_{i}' for i in range(len(layer_ids))]
    def make_features(self, x, clone=False):
        self.m_feat(x)
        return [(o.clone() if clone else o) for o in self.hooks.stored]
    def forward(self, input, target):
        out_feat = self.make_features(target, clone=True)
        in_feat = self.make_features(input)
        self.feat_losses = [base_loss(input,target)]
        self.feat_losses += [base_loss(f_in, f_out)*w
                             for f_in, f_out, w in zip(in_feat, out_feat, self.wgts)]
        self.feat_losses += [base_loss(gram_matrix(f_in), gram_matrix(f_out))*w**2 * 5e3
                             for f_in, f_out, w in zip(in_feat, out_feat, self.wgts)]
        self.metrics = dict(zip(self.metric_names, self.feat_losses))
        return sum(self.feat_losses)
    def __del__(self): self.hooks.remove()

path = Path(".")
learn=load_learner(path, 'ArtLine_650.pkl')

@app.route("/", methods=["POST"])
def home():
    # take user's sent image and turn it into line drawing
    return ''

if __name__ == '__main__':

    app.run(debug = True)

    How would I get this working with Flask?

    opened by wanshun123 3
  • Memory leak bug

    Memory leak bug

    Hello vijishmadhavan:

    After I run step p,img_hr,b = model.predict(img_fast), memory will increase 10M。 As long as you keep running, the memory will keep increasing, so I think it is memory leak.

    Please help me, Thank you very much!

    opened by xiaoxuwei12345 0
  • Packages version are not ok

    Packages version are not ok

    Hello, pip install runway-python fastai==1.0.61 numpy==1.17.2 pandas==1.1.2 torch==1.6.0 torchvision===0.7.0 fails. Some versions need update. Could you fix it please ?

    Thank you.

    opened by fabiencomte 0
  • how to train you model ?

    how to train you model ?

    Sir , Can you tell me which this dataset link ?

    `path = Path('/content/gdrive/My Drive/Apdrawing')

    Blended Facial Features

    path_hr = Path('/content/gdrive/My Drive/Apdrawing/draw tiny') path_lr = Path('/content/gdrive/My Drive/Apdrawing/Tiny Real')

    Portrait Pair

    path_hr3 = Path('/content/gdrive/My Drive/Apdrawing/drawing') path_lr3= Path('/content/gdrive/My Drive/Apdrawing/Real')`

    opened by h4rk8s 2
  • Model results observer

    Model results observer

    Hi!

    I would like to share one small tool that you may found useful. Right now I use it to compare differences between u2net and artline, it allows to quickly switch between two or more models and observe the differences (keys 2, 3).

    Interactive version here

    It uses ffhq as dataset

    opened by peko 1
  • Runnable Flask app to generate artline image

    Runnable Flask app to generate artline image

    Hi, I have built a simple Flask app to generate artline image and open sourced here: https://github.com/jwenjian/artline-demo

    Features:

    1. Can run locally in IDE or python3 app.py
    2. Select original image to process
    3. Show result in html page instead of show in matplotlib
    opened by jwenjian 0
Owner
Vijish Madhavan
Open to work
Vijish Madhavan
GitHub
General Virtual Sketching Framework for Vector Line Art (SIGGRAPH 2021)

General Virtual Sketching Framework for Vector Line Art - SIGGRAPH 2021 Paper | Project Page Outline Dependencies Testing with Trained Weights Trainin

Haoran MO 118 Dec 27, 2022
Deep Text Search is an AI-powered multilingual text search and recommendation engine with state-of-the-art transformer-based multilingual text embedding (50+ languages).

Deep Text Search - AI Based Text Search & Recommendation System Deep Text Search is an AI-powered multilingual text search and recommendation engine w

null 19 Sep 29, 2022
Deep Learning: Architectures & Methods Project: Deep Learning for Audio Super-Resolution

Deep Learning: Architectures & Methods Project: Deep Learning for Audio Super-Resolution Figure: Example visualization of the method and baseline as a

Oliver Hahn 16 Dec 23, 2022
State of the Art Neural Networks for Deep Learning

pyradox This python library helps you with implementing various state of the art neural networks in a totally customizable fashion using Tensorflow 2

Ritvik Rastogi 60 May 29, 2022
tsai is an open-source deep learning package built on top of Pytorch & fastai focused on state-of-the-art techniques for time series classification, regression and forecasting.

Time series Timeseries Deep Learning Pytorch fastai - State-of-the-art Deep Learning with Time Series and Sequences in Pytorch / fastai

timeseriesAI 2.8k Jan 8, 2023
😇A pyTorch implementation of the DeepMoji model: state-of-the-art deep learning model for analyzing sentiment, emotion, sarcasm etc

------ Update September 2018 ------ It's been a year since TorchMoji and DeepMoji were released. We're trying to understand how it's being used such t

Hugging Face 865 Dec 24, 2022
Deepparse is a state-of-the-art library for parsing multinational street addresses using deep learning

Here is deepparse. Deepparse is a state-of-the-art library for parsing multinational street addresses using deep learning. Use deepparse to Use the pr

GRAAL/GRAIL 192 Dec 20, 2022
make ASCII Art by Deep Learning

DeepAA This is convolutional neural networks generating ASCII art. This repository is under construction. This work is accepted by NIPS 2017 Workshop,

OsciiArt 1.4k Dec 28, 2022
Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

Deep Image Search - AI-Based Image Search Engine Deep Image Search is an AI-based image search engine that includes deep transfer learning features Ex

null 139 Jan 1, 2023
Honours project, on creating a depth estimation map from two stereo images of featureless regions

image-processing This module generates depth maps for shape-blocked-out images Install If working with anaconda, then from the root directory: conda e

null 2 Oct 17, 2022
Deep learning (neural network) based remote photoplethysmography: how to extract pulse signal from video using deep learning tools

Deep-rPPG: Camera-based pulse estimation using deep learning tools Deep learning (neural network) based remote photoplethysmography: how to extract pu

Terbe Dániel 138 Dec 17, 2022
Art Project "Schrödinger's Game of Life"

Repo of the project "Team Creative Quantum AI: Schrödinger's Game of Life" Installation new conda env: conda create --name qcml python=3.8 conda activ

ℍ◮ℕℕ◭ℍ ℝ∈ᛔ∈ℝ 2 Sep 15, 2022
This is the unofficial code of Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes. which achieve state-of-the-art trade-off between accuracy and speed on cityscapes and camvid, without using inference acceleration and extra data

Deep Dual-resolution Networks for Real-time and Accurate Semantic Segmentation of Road Scenes Introduction This is the unofficial code of Deep Dual-re

chenjun 113 Dec 23, 2022
A fast, dataset-agnostic, deep visual search engine for digital art history

imgs.ai imgs.ai is a fast, dataset-agnostic, deep visual search engine for digital art history based on neural network embeddings. It utilizes modern

Fabian Offert 5 Dec 14, 2022
Find-Lane-Line - Use openCV library and Python to detect the road-lane-line

Find-Lane-Line This project is to use openCV library and Python to detect the road-lane-line. Data Pipeline Step one : Color Selection Step two : Cann

Kenny Cheng 3 Aug 17, 2022
This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters.

openmc-plasma-source This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters. The OpenMC sources a

Fusion Energy 10 Oct 18, 2022
DeOldify - A Deep Learning based project for colorizing and restoring old images (and video!)

DeOldify - A Deep Learning based project for colorizing and restoring old images (and video!)

Jason Antic 15.8k Jan 4, 2023
LaneDet is an open source lane detection toolbox based on PyTorch that aims to pull together a wide variety of state-of-the-art lane detection models

LaneDet is an open source lane detection toolbox based on PyTorch that aims to pull together a wide variety of state-of-the-art lane detection models. Developers can reproduce these SOTA methods and build their own methods.

TuZheng 405 Jan 4, 2023
A set of tools for creating and testing machine learning features, with a scikit-learn compatible API

Feature Forge This library provides a set of tools that can be useful in many machine learning applications (classification, clustering, regression, e

Machinalis 380 Nov 5, 2022