Paper: De-rendering Stylized Texts

Overview

Paper: De-rendering Stylized Texts

Wataru Shimoda1, Daichi Haraguchi2, Seiichi Uchida2, Kota Yamaguchi1
1CyberAgent.Inc, 2 Kyushu University
Accepted to ICCV2021. [Publication] [Arxiv] [project-page]

Introduction

This repository contains the codes for "De-rendering stylized texts".

Concept

We propose to parse rendering parameters of stylized texts utilizing a neural net.

Demo

The proposed model parses rendering parameters based on famous 2d graphic engine[Skia.org|python implementation], which has compatibility with CSS in the Web. We can export the estimated rendering parameters and edit texts by an off-the-shelf rendering engine.

Installation

Requirements

  • Python >= 3.7
  • Pytorch >= 1.8.1
  • torchvision >= 0.9.1
pip install -r requiements.txt

Font data

  • The proposed model is trained with google fonts.
  • Download google fonts and locate in data/fonts/ as gfonts.
cd data/fonts
git clone https://github.com/google/fonts.git gfonts

Pre-rendered alpha maps

  • The proposed model parses rendering parameters and refines them through the differentiable rendering model, which uses pre-rendered alpha maps.
  • Generate pre-rendered alpha maps.
python -m util_lib.gen_pams

Pre-rendered alpha maps would be generated in data/fonts/prerendered_alpha.

Usage

Test

  • Download the pre-trained weight from this link (weight).
  • Locate the weight file in weights/font100_unified.pth.

Example usage.

python test.py --imgfile=example/sample.jpg

Note

  • imgfile option: path of an input image
  • results would be generated in res/

Data generation

in progress

Train

in progress

Todo

  • Testing codes
  • Codes for the text image generator
  • Training codes
  • Add notebooks for the guide

Reference

@InProceedings{Shimoda_2021_ICCV,
    author    = {Shimoda, Wataru and Haraguchi, Daichi and Uchida, Seiichi and Yamaguchi, Kota},
    title     = {De-Rendering Stylized Texts},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {1076-1085}
}

Contact

This repository is maintained by Wataru shimoda(wataru_shimoda[at]cyberagent.co.jp).

Comments
  • the angle of text is always zero

    the angle of text is always zero

    I find that you use a func named 'sample_text_type_flags' in 'class OffsetSampler(object)' to define the angle. But it always return 0, maybe it is a error? image

    opened by LIYHUI 5
  • name 'get_color' is not defined

    name 'get_color' is not defined

    when I run this python gen.py --bgtype=color, there is a error: [D 220704 06:56:21 handler:78] placement False [D 220704 06:56:21 gen_mode:105] 10, 0, 10000 Traceback (most recent call last): File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/handler.py", line 32, in sample_and_render placed = self.sampler.sample(ih, dh) File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/sampler.py", line 71, in sample style_data = self.style_sampler.sample(ih, dh) File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/sampler_lib/style_sampler.py", line 30, in sample effect_params = self.get_style_params( File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/sampler_lib/style_sampler.py", line 21, in get_style_params grad_param = skp.get_gradation_param( File "/home/yonghui_li/Image_generate/derendering-text/src/skia_lib/skia_paintor.py", line 85, in get_gradation_param color = get_color() NameError: name 'get_color' is not defined Traceback (most recent call last): File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/handler.py", line 32, in sample_and_render placed = self.sampler.sample(ih, dh) File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/sampler.py", line 71, in sample style_data = self.style_sampler.sample(ih, dh) File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/sampler_lib/style_sampler.py", line 30, in sample effect_params = self.get_style_params( File "/home/yonghui_li/Image_generate/derendering-text/src/modules/generator/src/sampler_lib/style_sampler.py", line 21, in get_style_params grad_param = skp.get_gradation_param( File "/home/yonghui_li/Image_generate/derendering-text/src/skia_lib/skia_paintor.py", line 85, in get_gradation_param color = get_color() NameError: name 'get_color' is not defined

    opened by LIYHUI 3
  • error in generate pre-rendered alpha maps for stoke

    error in generate pre-rendered alpha maps for stoke

    generate pre-rendered alpha maps for stroke Traceback (most recent call last): File "C:\ProgramData\Anaconda3\envs\derendering-text-master\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\ProgramData\Anaconda3\envs\derendering-text-master\lib\runpy.py", line 87, in _run_code exec(code, run_globals) File "C:\Users\mowe\Desktop\projects\derendering-text-master\util_lib\gen_pams.py", line 118, in main() File "C:\Users\mowe\Desktop\projects\derendering-text-master\util_lib\gen_pams.py", line 113, in main gen_stroke_pams(char_dict) File "C:\Users\mowe\Desktop\projects\derendering-text-master\util_lib\gen_pams.py", line 94, in gen_stroke_pams alpha_crop = cv2.resize( cv2.error: OpenCV(4.6.0) D:\a\opencv-python\opencv-python\opencv\modules\imgproc\src\resize.cpp:4052: error: (-215:Assertion failed) !ssize.empty() in function 'cv::resize'

    opened by buloseshi 2
  • bug about rotating angles.

    bug about rotating angles.

    image image

    As you can see, some word images are out of the border, I think maybe there is a error in offset_sampler.py. I try to solve it, but there is still errors.

    opened by LIYHUI 1
  • bibtex

    bibtex

    Thank you for publishing the great work!

    There are some mistakes in the bibtex in README.md. The bibtex in the following link will be helpful. https://openaccess.thecvf.com/content/ICCV2021/html/Shimoda_De-Rendering_Stylized_Texts_ICCV_2021_paper.html

    opened by shinya7y 1
  • for prerendering, 0-dimensional alpha_crop causes resize error.

    for prerendering, 0-dimensional alpha_crop causes resize error.

    I appreciate the provided code and tried myself.

    For the prerendering script python -m util_lib.gen_pams

    Following resize code causes error: https://github.com/CyberAgentAILab/derendering-text/blob/0273325a29054cb16a4b4d30466f837df6c62b10/util_lib/gen_pams.py#L95

    This is beause why alpha_crop = crop_alpha_area(alpha) results some (0,n) or (n,0) shaped array, therefore resize is impossible. To deal with this, I temporarily altered code and checked how this issue as following:

    image

    and this is following logs:

    image

    Is this ok? Looks like lots of crop errors occur and result resizing error.

    opened by junhocho 2
Owner
CyberAgent AI Lab
CyberAgent AI Lab
PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos

PyKale is a PyTorch library for multimodal learning and transfer learning as well as deep learning and dimensionality reduction on graphs, images, texts, and videos. By adopting a unified pipeline-based API design, PyKale enforces standardization and minimalism, via reusing existing resources, reducing repetitions and redundancy, and recycling learning models across areas.

PyKale 370 Dec 27, 2022
Generate images from texts. In Russian. In PaddlePaddle

ruDALL-E PaddlePaddle ruDALL-E in PaddlePaddle. Install: pip install rudalle_paddle==0.0.1rc1 Run with free v100 on AI Studio. Original Pytorch versi

AgentMaker 20 Oct 18, 2022
Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts

t5-japanese Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts. The following is a list of models that

Kimio Kuramitsu 1 Dec 13, 2021
Build an Amazon SageMaker Pipeline to Transform Raw Texts to A Knowledge Graph

Build an Amazon SageMaker Pipeline to Transform Raw Texts to A Knowledge Graph This repository provides a pipeline to create a knowledge graph from ra

AWS Samples 3 Jan 1, 2022
This repository contains the source code for the paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks",

DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks Project Page | Video | Presentation | Paper | Data L

Facebook Research 281 Dec 22, 2022
The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

Ren Yurui 261 Jan 9, 2023
The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

Website | ArXiv | Get Start | Video PIRenderer The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic

Ren Yurui 81 Sep 25, 2021
PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021.

IBRNet: Learning Multi-View Image-Based Rendering PyTorch implementation of paper "IBRNet: Learning Multi-View Image-Based Rendering", CVPR 2021. IBRN

Google Interns 371 Jan 3, 2023
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 697 Jan 6, 2023
Official Repo for ICCV2021 Paper: Learning to Regress Bodies from Images using Differentiable Semantic Rendering

[ICCV2021] Learning to Regress Bodies from Images using Differentiable Semantic Rendering Getting Started DSR has been implemented and tested on Ubunt

Sai Kumar Dwivedi 83 Nov 27, 2022
Code for "Layered Neural Rendering for Retiming People in Video."

Layered Neural Rendering in PyTorch This repository contains training code for the examples in the SIGGRAPH Asia 2020 paper "Layered Neural Rendering

Google 154 Dec 16, 2022
NeRViS: Neural Re-rendering for Full-frame Video Stabilization

Neural Re-rendering for Full-frame Video Stabilization

Yu-Lun Liu 9 Jun 17, 2022
Neural Re-rendering for Full-frame Video Stabilization

NeRViS: Neural Re-rendering for Full-frame Video Stabilization Project Page | Video | Paper | Google Colab Setup Setup environment for [Yu and Ramamoo

Yu-Lun Liu 9 Jun 17, 2022
UnsupervisedR&R: Unsupervised Pointcloud Registration via Differentiable Rendering

UnsupervisedR&R: Unsupervised Pointcloud Registration via Differentiable Rendering This repository holds all the code and data for our recent work on

Mohamed El Banani 118 Dec 6, 2022
Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021)

Style-based Point Generator with Adversarial Rendering for Point Cloud Completion (CVPR 2021) An efficient PyTorch library for Point Cloud Completion.

Microsoft 119 Jan 2, 2023
Rendering Point Clouds with Compute Shaders

Compute Shader Based Point Cloud Rendering This repository contains the source code to our techreport: Rendering Point Clouds with Compute Shaders and

Markus Schütz 460 Jan 5, 2023
Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021

AutoInt: Automatic Integration for Fast Neural Volume Rendering CVPR 2021 Project Page | Video | Paper PyTorch implementation of automatic integration

Stanford Computational Imaging Lab 149 Dec 22, 2022
Code for PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Relighting and Material Editing

PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Relighting and Material Editing CVPR 2021. Project page: https://kai-46.github.io/

Kai Zhang 141 Dec 14, 2022
Generative Query Network (GQN) in PyTorch as described in "Neural Scene Representation and Rendering"

Update 2019/06/24: A model trained on 10% of the Shepard-Metzler dataset has been added, the following notebook explains the main features of this mod

Jesper Wohlert 313 Dec 27, 2022