Handwritten Line Text Recognition using Deep Learning with Tensorflow
Description
Use Convolutional Recurrent Neural Network to recognize the Handwritten line text image without pre segmentation into words or characters. Use CTC loss Function to train. More read this Medium Post
Why Deep Learning?
Deep Learning self extracts features with a deep neural networks and classify itself. Compare to traditional Algorithms it performance increase with Amount of Data.
Basic Intuition on How it Works.
- First Use Convolutional Recurrent Neural Network to extract the important features from the handwritten line text Image.
- The output before CNN FC layer (512x100x8) is passed to the BLSTM which is for sequence dependency and time-sequence operations.
- Then CTC LOSS Alex Graves is used to train the RNN which eliminate the Alignment problem in Handwritten, since handwritten have different alignment of every writers. We just gave the what is written in the image (Ground Truth Text) and BLSTM output, then it calculates loss simply as
-log("gtText"); aim to minimize negative maximum likelihood path. - Finally CTC finds out the possible paths from the given labels. Loss is given by for (X,Y) pair is:
- Finally CTC Decode is used to decode the output during Prediction.
Detail Project Workflow
- Project consists of Three steps:
- Multi-scale feature Extraction --> Convolutional Neural Network 7 Layers
- Sequence Labeling (BLSTM-CTC) --> Recurrent Neural Network (2 layers of LSTM) with CTC
- Transcription --> Decoding the output of the RNN (CTC decode)
Requirements
- Tensorflow 1.8.0
- Flask
- Numpy
- OpenCv 3
- Spell Checker
autocorrect>=0.3.0pip install autocorrect
Dataset Used
- IAM dataset download from here
- Only needed the lines images and lines.txt (ASCII).
- Place the downloaded files inside data directory
The Trained model is available and download from this link. The trained model CER=8.32% and trained on IAM dataset with some additional created dataset.
To Train the model from scratch
$ python main.py --train
To validate the model
$ python main.py --validate
To Prediction
$ python main.py
Run in Web with Flask
$ python upload.py
Validation character error rate of saved model: 8.654728%
Python: 3.6.4
Tensorflow: 1.8.0
Init with stored values from ../model/snapshot-24
Without Correction clothed leaf by leaf with the dioappoistmest
With Correction clothed leaf by leaf with the dioappoistmest
Prediction output on IAM Test Data 
Prediction output on Self Test Data 
See the project Devnagari Handwritten Word Recognition with Deep Learning for more insights.
Further Improvement
- Using MDLSTM to recognize whole paragraph at once Scan, Attend and Read: End-to-End Handwritten Paragraph Recognition with MDLSTM Attention
- Line segementation can be added for full paragraph text recognition. For line segmentation you can use A* path planning algorithm or CNN model to seperate paragraph into lines.
- Better Image preprocessing such as: reduce backgoround noise to handle real time image more accurately.
- Better Decoding approach to improve accuracy. Some of the CTC Decoder found here
Feel Free to improve this project with pull Request.
This is part of my last semester project of Computer Engineering From Tribhuvan University. July 2019
190 Jan 3, 2023
213 Dec 26, 2022
3 Aug 15, 2022
489 Dec 21, 2022
2 Feb 6, 2022
346 Jan 7, 2023
1000 Dec 27, 2022
80 Dec 30, 2022
14 Aug 18, 2022
356 Dec 8, 2022
2.6k Dec 29, 2022
17 Aug 25, 2022
160 Dec 21, 2022
562 Jan 3, 2023
117 Jan 2, 2023
127 Dec 3, 2022
90 Dec 22, 2022
4 Jun 30, 2022