AttentionOCR for Arbitrary-Shaped Scene Text Recognition
Introduction
This is the ranked No.1 tensorflow based scene text spotting algorithm on ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (Latin Only, Latin and Chinese), futhermore, the algorithm is also adopted in ICDAR2019 Robust Reading Challenge on Large-scale Street View Text with Partial Labeling and ICDAR2019 Robust Reading Challenge on Reading Chinese Text on Signboard.
Scene text detection algorithm is modified from Tensorpack FasterRCNN, and we only open source code in this repository for scene text recognition. I upload ICDAR2019 ArT competition model to docker hub, please refer to Docker. For more details, please refer to our arXiv technical report.
Our text recognition algorithm not only recognizes Latin and Non-Latin characters, but also supports horizontal and vertical text recognition in one model. It is convenient for multi-lingual arbitrary-shaped text recognition.
Note that the competition model in docker container as described in our technical report is slightly different from the recognition model trained from this updated repository.
Dependencies
python 3
tensorflow-gpu 1.14
tensorpack 0.9.8
pycocotools
Usage
First download and extract multiple text datasets in base text dir, please refer to dataset.py for dataset preprocess and multiple datasets.
Multiple Datasets
$(base_dir)/lsvt
$(base_dir)/art
$(base_dir)/rects
$(base_dir)/icdar2017rctw
You can also synthesize text recognition data for data augmentation, please refer to TextRecognitionDataGenerator. It is helpful for long text recognition and attention-based language model because you can directly synthesize text images from NLP corpus. Then you should rewrite dataset.py for synthetic text dataset.
$(base_dir)/synthetic_text
Train
First, download pretrained inception v4 checkpoint and put it in ./pretrain folder. Then you can modify your gpu lists in config.py for specified gpus and then run:
python train.py
You can visualize your training steps via tensorboard:
tensorboard --logdir='./checkpoint'
Use ICDAR2019-LSVT, ICDAR2019-ArT, ICDAR2019-ReCTS for default training, you can change it with your own training data.
Evaluation
python eval.py --checkpoint_path=$(Your model path)
Use ICDAR2017RCTW for default evaluation with Normalized Edit Distance metric(1-N.E.D specifically), you can change it with your own evaluation data.
Export
Export checkpoint to tensorflow pb model for inference.
python export.py --pb_path=$(Your tensorflow pb model save path) --checkpoint_path=$(Your trained model path)
Test
Load tensorflow pb model for text recognition.
python test.py --pb_path=$(Your tensorflow pb model save path) --img_folder=$(Your test img folder)
Default use ICDAR2019-ArT for test, you can change it with your own test data.
Visualization
Scene text detection and recognition result:
Scene text recognition attention maps:
To learn more about attention mechanism, please refer to Attention Mechanism in Deep Learning.
Docker
I upload ICDAR2019 scene text recognition model include text detection and recognition to Docker Hub.
After nvidia-docker installed, run:
docker pull zhang0jhon/demo:ocr
docker run -it -p 5000:5000 --gpus all zhang0jhon/demo:ocr bash
cd /ocr/ocr
python flaskapp.py
Then you can test with your data via browser:
$(localhost or remote server ip address):5000
595 Dec 27, 2022
777 Jan 9, 2023
89 Aug 4, 2022
572 Jan 5, 2023
155 Dec 6, 2022
1.6k Dec 22, 2022
43 Mar 15, 2022
763 Jan 1, 2023
1000 Dec 27, 2022
170 Dec 26, 2022
115 Dec 12, 2022
1.2k Dec 29, 2022
2.4k Jan 8, 2023
3.2k Jan 4, 2023
684 Jan 6, 2023
27 Jan 8, 2023
121 Oct 15, 2021
102 Jun 29, 2022