Synthetic Scene Text from 3D Engines

Overview

Introduction

UnrealText is a project that synthesizes scene text images using 3D graphics engine. This repository accompanies our paper: UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World [PDF].

pipeline

We release (Baidu/Google Drive):

Paper

If you find our project helpful in your research, you are highly encouraged to cite our paper:

@article{long2020unrealtext,
  title={UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World},
  author={Long, Shangbang and Yao, Cong},
  journal={arXiv preprint arXiv:2003.10608},
  year={2020}
}
Comments
  • the address of UnrealEngine is invalid

    the address of UnrealEngine is invalid

    when i tried to git pull the UnrealEngine by useing order ''git clone -b 4.22 https://github.com/EpicGames/UnrealEngine.git'' ,it was failed repository 'https://github.com/EpicGames/UnrealEngine.git/' not found

    opened by Instantnoodles-madman 17
  • when i compil UE,it happend wrong

    when i compil UE,it happend wrong

    Failed to download 'http://cdn.unrealengine.com/dependencies/UnrealEngine-4688746-f006827fb0f043339219f8569dbe45f2/023575ba9057aaffd75ea0bd851b2fc6950a9e51': The operation has timed out. (WebException) but i can download this file,How update dependencies manually? may you know the reason for this wrong

    opened by Instantnoodles-madman 16
  • I find that it is difficult to run the demo.

    I find that it is difficult to run the demo.

    When it run to Line 72 in TexPlacingModule.py, error occurred as 'FileNotFoundError: [Errno 2] No such file or directory: '/home/eecbd/github_project/UnrealText_CVPR2020/GeneratedData/DataFraction_14/WordCrops/adjusted_text_box.txt''. We can't find anywhere to save 'adjusted_text_box.txt' in your code, could you please help me solve this problem?

    opened by zhangjx123 9
  • Can you share the word crop code

    Can you share the word crop code

    In the paper : "We crop from the proposed multilingual dataset. We discard images with widths shorter than 32 pixels as they are too blurry, and obtain 4.1M word images in total." But I ended up with more than 7 million text line images.

    opened by wushilian 5
  • Drawing images instead of text

    Drawing images instead of text

    From looking at the code, I was wondering how hard it would be to hijack the StickerTextActor to place small images instead of text, allowing for the generation of a dataset for label detection.

    A bit more context, I am working on building a hazmat label detector (the diamond-shaped sign found on explosive/corrosive/infectious material containers) for a competition and it would be useful to have a lot of synthetic data to save the actual data (of which I do not have a lot) for fine-tuning.

    From glancing at your code, unless I am mistaken you already load the text as PNG and then draw it with the engine, meaning that the text generation is on the Python side and I could simply load my own PNG (that wouldn't be text) and it would work?

    Great project btw, can't wait for your presentation at CVPR :)

    EDIT: Just to be clear, this is not a feature request, I just want to be sure that it is doable.

    opened by Belval 5
  • quite some character level bboxes are of zero width or zero height

    quite some character level bboxes are of zero width or zero height

    thanks again for open sourcing your dataset and generation code. When training a model that requires character level labelling, I found that many of the character level bboxes either have zero width or zero height. May I know a suggested way to clean the data ?

    opened by mmmmmore 4
  • Data annotation problem

    Data annotation problem

    Hi, When I used the dataset (English/Latin) you provided, I found that some json annotations were wrong, such as 749 ~ 753 for sub_48. And why are some coordinates negative?
    Looking forward to your reply.

    opened by Gyann-z 4
  • some imgs seem to be corrupted

    some imgs seem to be corrupted

    Hi, Thanks for your open source dataset. But when I use Unrealtext dataset, I found some imgs corrupted. I referenced mmocr source code to pretrain my model, but it seems loading img meet error:

                with open(img_name, 'rb') as f:
                    img_buff = f.read()
                img = mmcv.imfrombytes(img_buff, IMREAD_COLOR)
    

    accidently, above code would meets error: cv2.error: OpenCV(4.1.2) /io/opencv/modules/imgcodecs/src/loadsave.cpp:730: error: (-215:Assertion failed) !buf.empty() in function 'imdecode_'. It seems the img file is corrupted. But I am not sure which one or ones corrupted (Debugging and finding)

    opened by aiboys 3
  • The character boxes on hook faces seems to be offset

    The character boxes on hook faces seems to be offset

    There seems to be offset of character boxes on hook faces with high probability, shown in the following images. This image is from English dataset Sub39 27.jpg image Only use perspective transformation to get the boxes may not work.

    opened by GaroneHuang 3
  • Can I extract background scene before add text??

    Can I extract background scene before add text??

    I tried to extract image before placing text in the scene by adding self.cleint.SaveImg before 'step 3: place text' in DataGeneratorModule.py

    but text already exists in the extracted scene

    is it possible extract a scene that is the same as the generated image but has no text??

    opened by hsl323 3
  • Can this be run on multiple terminals?

    Can this be run on multiple terminals?

    I want to make the image generation faster by running multiple terminal on the code. Is it feasible to do so? I am stuck cuz only one terminal work and the rest are just not responding

    opened by personwhofloat 2
Owner
Shangbang Long
Shangbang Long
Pytorch implementation of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors

Make-A-Scene - PyTorch Pytorch implementation (inofficial) of Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors (https://arxiv.org/

Casual GAN Papers 259 Dec 28, 2022
Pytorch re-implementation of Paper: SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text Recognition (CVPR 2022)

SwinTextSpotter This is the pytorch implementation of Paper: SwinTextSpotter: Scene Text Spotting via Better Synergy between Text Detection and Text R

mxin262 183 Jan 3, 2023
Unified Interface for Constructing and Managing Workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow.

Couler What is Couler? Couler aims to provide a unified interface for constructing and managing workflows on different workflow engines, such as Argo

Couler Project 781 Jan 3, 2023
Official codebase for Pretrained Transformers as Universal Computation Engines.

universal-computation Overview Official codebase for Pretrained Transformers as Universal Computation Engines. Contains demo notebook and scripts to r

Kevin Lu 210 Dec 28, 2022
This repo contains the code and data used in the paper "Wizard of Search Engine: Access to Information Through Conversations with Search Engines"

Wizard of Search Engine: Access to Information Through Conversations with Search Engines by Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zh

null 19 Oct 27, 2022
QueryFuzz implements a metamorphic testing approach to test Datalog engines.

Datalog is a popular query language with applications in several domains. Like any complex piece of software, Datalog engines may contain bugs. The mo

null 34 Sep 10, 2022
Fuzzing JavaScript Engines with Aspect-preserving Mutation

DIE Repository for "Fuzzing JavaScript Engines with Aspect-preserving Mutation" (in S&P'20). You can check the paper for technical details. Environmen

gts3.org (SSLab@Gatech) 190 Dec 11, 2022
Code for CVPR 2021 oral paper "Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts"

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts The rapid progress in 3D scene understanding has come with growing dem

Facebook Research 182 Dec 30, 2022
[TIP 2020] Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion Code for Multi-Temporal Scene Classification and Scene Ch

Lixiang Ru 33 Dec 12, 2022
Neural Scene Graphs for Dynamic Scene (CVPR 2021)

Implementation of Neural Scene Graphs, that optimizes multiple radiance fields to represent different objects and a static scene background. Learned representations can be rendered with novel object compositions and views.

null 151 Dec 26, 2022
A weakly-supervised scene graph generation codebase. The implementation of our CVPR2021 paper ``Linguistic Structures as Weak Supervision for Visual Scene Graph Generation''

README.md shall be finished soon. WSSGG 0 Overview 1 Installation 1.1 Faster-RCNN 1.2 Language Parser 1.3 GloVe Embeddings 2 Settings 2.1 VG-GT-Graph

Keren Ye 35 Nov 20, 2022
Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

Cheng Zhang 66 Nov 16, 2022
Code and data of the ACL 2021 paper: Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision

MetaAdaptRank This repository provides the implementation of meta-learning to reweight synthetic weak supervision data described in the paper Few-Shot

THUNLP 5 Jun 16, 2022
Official implementation of SynthTIGER (Synthetic Text Image GEneratoR) ICDAR 2021

?? SynthTIGER: Synthetic Text Image GEneratoR Official implementation of SynthTIGER | Paper | Datasets Moonbin Yim1, Yoonsik Kim1, Han-cheol Cho1, Sun

Clova AI Research 256 Jan 5, 2023
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022) https://arxiv.org/abs/2203.09388 Jianqi Ma, Zheto

MA Jianqi, shiki 104 Jan 5, 2023
Learning from Synthetic Shadows for Shadow Detection and Removal [Inoue+, IEEE TCSVT 2020].

Learning from Synthetic Shadows for Shadow Detection and Removal (IEEE TCSVT 2020) Overview This repo is for the paper "Learning from Synthetic Shadow

Naoto Inoue 67 Dec 28, 2022
Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation"

SharinGAN Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation" The official project we

Koutilya PNVR 23 Oct 19, 2022
Synthetic Humans for Action Recognition, IJCV 2021

SURREACT: Synthetic Humans for Action Recognition from Unseen Viewpoints Gül Varol, Ivan Laptev and Cordelia Schmid, Andrew Zisserman, Synthetic Human

Gul Varol 59 Dec 14, 2022
Official PyTorch implementation of "Proxy Synthesis: Learning with Synthetic Classes for Deep Metric Learning" (AAAI 2021)

Proxy Synthesis: Learning with Synthetic Classes for Deep Metric Learning Official PyTorch implementation of "Proxy Synthesis: Learning with Synthetic

NAVER/LINE Vision 30 Dec 6, 2022