Extracting Tables from Document Images using a Multi-stage Pipeline for Table Detection and Table Structure Recognition:

Overview

Multi-Type-TD-TSR

Check it out on Open In Colab Source Code of our Paper: Multi-Type-TD-TSR Extracting Tables from Document Images using a Multi-stage Pipeline for Table Detection and Table Structure Recognition

Description

Multi-Type-TD-TSR the Whole Pipeline

As global trends are shifting towards data-driven industries, the demand for automated algorithms that can convert digital images of scanned documents into machine readable information is rapidly growing. Besides the opportunity of data digitization for the application of data analytic tools, there is also a massive improvement towards automation of processes, which previously would require manual inspection of the documents. Although the introduction of optical character recognition (OCR) technologies mostly solved the task of converting human-readable characters from images into machine-readable characters, the task of extracting table semantics has been less focused on over the years. The recognition of tables consists of two main tasks, namely table detection and table structure recognition. Most prior work on this problem focuseson either task without offering an end-to-end solution or paying attention to real application conditions like rotated images or noise artefacts inside the document image. Recent work shows a clear trend towards deep learning approaches coupled with the use of transfer learning for the task of table structure recognition due to the lack of sufficiently large datasets. In this paper we present a multistage pipeline named Multi-Type-TD-TSR, which offers an end-to-end solution for the problem of table recognition. It utilizes state-of-the-art deep learning models for table detection and differentiates between 3 different types of tables based on the tables’ borders. For the table structure recognition we use a deterministic non-data driven algorithm, which works on all table types. We additionally present two algorithms. One for unbordered tables and one for bordered tables, which are the base of the used table structure recognition algorithm. We evaluate Multi-Type-TD-TSR on the ICDAR 2019 table structure recognition dataset and achieve a new state-of-the-art.

Multi-Type-TD-TSR on Fully Bordered Tables

For TSR on fully bordered tables, we use the erosion and dilation operation to extract the row-column grid cell image without any text or characters. The erosion kernels are generally thin vertical and horizontal strips that are longer than the overall font size but shorter than the size of the smallest grid cell and, in particular, must not be wider than the smallest table border width. Using these kernel size constraints results in the erosion operation removing all fonts and characters from the table while preserving the table borders. In order to restore the original line shape, the algorithm applies the dilation operation using the same kernel size on each of the two eroded images, producing an image with vertical and a second with horizontal lines. Finally, the algorithm combines both images by using a bit-wise ```or``` operation and re-inverting the pixel values to obtain a raster cell image. We then use the contours function on the grid-cell image to extract the bounding-boxes for every single grid cell.

Multi-Type-TD-TSR on Unbordered Tables

The TSR algorithm for unbordered tables works similarly to the one for bordered tables but utilizes the erosion operation in a different way. The erosion kernel is in general a thin strip with the difference that the horizontal size of the horizontal kernel includes the full image width and the vertical size of the vertical kernel the full image height. The algorithm slides both kernels independently over the whole image from left to right for the vertical kernel, and from top to bottom for the horizontal kernel. During this process it is looking for empty rows and columns that do not contain any characters or font. The resulting images are inverted and combined by a bit-wise ```and``` operation producing the final output. The output is a grid-cell image similar to the one from TSR for bordered tables, where the overlapping areas of the two resulting images represent the bounding-boxes for every single grid cell.

Multi-Type-TD-TSR on Partially Bordered Tables

The main goal of our algorithms for bordered and unbordered tables is to create a grid cell image by adding borders in the unbordered case and detecting lines in the bordered case. If a table is only partially bordered, then the unbordered algorithm is prevented to add borders in orthogonal direction to the existing borders, while the bordered algorithm can only find the existing borders. Both approaches result in incomplete grid cell images.


TSR for partially bordered tables uses the same erosion algorithm as in bordered tables to detect existing borderes, but without using them to create a grid cell, but to delete the borders from the table image to get an unbordered table. This allows for applying the algorithm for unbordered tables to create the grid-cell image and contours by analogy to the variants discussed above. A key feature of this approach is that it works with both bordered and unbordered tables: it is type-independent.

 

 

 

 

 

Table Structure Recognition Results

ICDAR 19 (Track B2)

IoU IoU IoU IoU Weighted
Team 0.6 0.7 0.8 0.9 Average
CascadeTabNet 0.438 0.354 0.19 0.036 0.232
NLPR-PAL 0.365 0.305 0.195 0.035 0.206
Multi-Type-TD-TSR 0.589 0.404 0.137 0.015 0.253

Instructions

Configurations

The source code is developed under the following library dependencies

  • PyTorch = 1.7.0
  • Torchvision = 0.8.1
  • Cuda = 10.1
  • PyYAML = 5.1

Detectron 2

The table detection model is based on detectron2 follow this installation guide to setup.

Image Alignment Pre-Processing

For the image alignment pre-processing step there is one script available:

  • deskew.py

To apply the image alignment pre-processing algorithm to all images in one folder, you need to execute:

python3 deskew.py

with the following parameters

  • --folder the input folder including document images
  • --output the output folder for the deskewed images

Table Structure Recognition (TSR)

For the table structure recognition we offer a simple script for different approaches

  • tsr.py

To apply a table structure recognition algorithm to all images in one folder, you need to execute:

python3 tsr.py

with the following parameters

  • --folder path of the input folder including table images
  • --type the table structure recognition type type in ["borderd", "unbordered", "partially", "partially_color_inv"]
  • --img_output output folder path for the processed images
  • --xml_output output folder path for the xml files including bounding boxes

Table Detection and Table Structure Recognition (TD & TSR)

To appy the table detection with a followed table structure recogniton

  • tdtsr.py

To apply a table structure recognitio algorithm to all images in one folder, you need to execute:

python3 tdtsr.py

with the following parameters

  • --folder path of the input folder including table images
  • --type the table structure recognition type type in ["borderd", "unbordered", "partially", "partially_color_inv"]
  • --tsr_img_output output folder path for the processed table images
  • --td_img_output output folder path for the produced table cutouts
  • --xml_output output folder path for the xml files for tables and cells including bounding boxes
  • --config path of detectron2 configuration file for table detection
  • --yaml path of detectron2 yaml file for table detection
  • --weights path of detectron2 model weights for table detection

Evaluation

To evaluate the table structure recognition algorithm we provide the following script:

  • evaluate.py

to apply the evaluation the table images and their labels in xml-format have to be the same name and should lie in a single folder. The evaluation could be started by:

python3 evaluate.py

with the following parameter

  • --dataset dataset folder path containing table images and labels in .xml format

Get Data

  • test dataset for table structure recognition including table images and annotations can be downloaded here
  • table detection detectron2 model weights and configuration files can be downloaded here

Citation

@misc{fischer2021multitypetdtsr,
    title={Multi-Type-TD-TSR - Extracting Tables from Document Images using a Multi-stage Pipeline for Table Detection and Table Structure Recognition: from OCR to Structured Table Representations},
    author={Pascal Fischer and Alen Smajic and Alexander Mehler and Giuseppe Abrami},
    year={2021},
    eprint={2105.11021},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
Comments
  • Issue with running table detection and structure recognition

    Issue with running table detection and structure recognition

    Hi, Thanks for the great work. I am trying to run your tdtsr.py script on my custom images but I am running into error with the config file:

    Screen Shot 1400-10-24 at 11 36 49 ' Any idea how to resolve this will be appreciated. Thanks
    opened by Kehindeajayi01 3
  • Possibility of using Multi-Type-TD-TSR locally

    Possibility of using Multi-Type-TD-TSR locally

    Hi, great repo! I am trying out this repo as part of my project and I wish to ask is it possible to use this git repo in local jupyter environment? I faced this error as shown in the screenshot below when trying to run locally, and was unsure how to solve it. Thank you! Screenshot 2021-10-03 at 11 15 39 PM

    opened by chiaaifen 0
  • Error while running colab notebook

    Error while running colab notebook

    Hello,

    When I try to compile the model using your colab notebook, I get the following error:

    The checkpoint state_dict contains keys that are not used by the model:
      pixel_mean
      pixel_std
      proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4}
    

    Any idea what could be going wrong? Cheers!

    opened by ninikolov 1
  • list index out of range error

    list index out of range error

    Hi, I was trying to run the model on the below image: 2 in Colab with following command: !python /content/Multi_Type_TD_TSR/scripts/tsr.py --folder=/content/images --type="partially_color_inv" --img_output=/content/img_output --xml_output=/content/xml_output But I was getting the below error: Traceback (most recent call last): File "/content/Multi_Type_TD_TSR/scripts/tsr.py", line 30, in <module> boxes, img_processed = type_dict[args.type].recognize_structure(img) File "/content/Multi_Type_TD_TSR/scripts/TSR/table_structure_recognition_all.py", line 208, in recognize_structure center = [int(row[index][j][0] + row[index][j][2] / 2) for j in range(len(row[index]))] IndexError: list index out of range

    Not sure though I have provided the correct value of 'type' flag over here.

    opened by shaktiman101 3
  • Onnx conversion of the model

    Onnx conversion of the model

    First of all, thanks a lot for sharing your work. I wanted to ask you if you have tried the onnx conversion of the model or if you could provide some guidance in that direction?

    Thanks

    opened by pshwetank 0
Owner
Pascal Fischer
love machine learning, algorithms, probabilistic approaches, computer vision, natural language processing, robotics, 3D graphics and simulations.
Pascal Fischer
Unofficial implementation of "TableNet: Deep Learning model for end-to-end Table detection and Tabular data extraction from Scanned Document Images"

TableNet Unofficial implementation of ICDAR 2019 paper : TableNet: Deep Learning model for end-to-end Table detection and Tabular data extraction from

Jainam Shah 235 Dec 5, 2022
Motion detector, Full body detection, Upper body detection, Cat face detection, Smile detection, Face detection (haar cascade), Silverware detection, Face detection (lbp), and Sending email notifications

Security camera running OpenCV for object and motion detection. The camera will send email with image of any objects it detects. It also runs a server that provides web interface with live stream video.

Peace 10 Jun 30, 2021
Extract tables from scanned image PDFs using Optical Character Recognition.

ocr-table This project aims to extract tables from scanned image PDFs using Optical Character Recognition. Install Requirements Tesseract OCR sudo apt

Abhijeet Singh 209 Nov 22, 2022
TableBank: A Benchmark Dataset for Table Detection and Recognition

TableBank TableBank is a new image-based table detection and recognition dataset built with novel weak supervision from Word and Latex documents on th

null 842 Dec 3, 2022
End-to-end pipeline for real-time scene text detection and recognition.

Real-time-Scene-Text-Detection-and-Recognition-System End-to-end pipeline for real-time scene text detection and recognition. The detection model use

Fangneng Zhan 89 Aug 4, 2022
Table recognition inside douments using neural networks

TableTrainNet A simple project for training and testing table recognition in documents. This project was developed to make a neural network which reco

Giovanni Cavallin 93 Jul 24, 2022
Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and text alignment capabilities.

Genalog is an open source, cross-platform python package allowing generation of synthetic document images with custom degradations and text alignment capabilities.

Microsoft 231 Nov 29, 2022
Detect textlines in document images

Textline Detection Detect textlines in document images Introduction This tool performs border, region and textline detection from document image data

QURATOR-SPK 70 Jun 30, 2022
Detect textlines in document images

Textline Detection Detect textlines in document images Introduction This tool performs border, region and textline detection from document image data

QURATOR-SPK 70 Jun 30, 2022
Binarize document images

Binarization Binarization for document images Examples Introduction This tool performs document image binarization (i.e. transform colour/grayscale to

QURATOR-SPK 47 Sep 1, 2022
A curated list of resources for text detection/recognition (optical character recognition ) with deep learning methods.

awesome-deep-text-detection-recognition A curated list of awesome deep learning based papers on text detection and recognition. Text Detection Papers

null 2.4k Dec 1, 2022
A tool for extracting text from scanned documents (via OCR), with user-defined post-processing.

The project is based on older versions of tesseract and other tools, and is now superseded by another project which allows for more granular control o

Maxim 32 Jul 24, 2022
A machine learning software for extracting information from scholarly documents

GROBID GROBID documentation Visit the GROBID documentation for more detailed information. Summary GROBID (or Grobid, but not GroBid nor GroBiD) means

Patrice Lopez 1.8k Dec 2, 2022
Document manipulation detection with python

image manipulation detection task: --> tianchi function image segmentation salie

JiaKui Hu 3 Aug 22, 2022
A document scanner application for laptops/desktops developed using python, Tkinter and OpenCV.

DcoumentScanner A document scanner application for laptops/desktops developed using python, Tkinter and OpenCV. Directly install the .exe file to inst

Harsh Vardhan Singh 1 Oct 29, 2021
Erosion and dialation using structure element in OpenCV python

Erosion and dialation using structure element in OpenCV python

Tamzid hasan 2 Nov 11, 2021
A simple document layout analysis using Python-OpenCV

Run the application: python main.py *Note: For first time running the application, create a folder named "output". The application is a simple documen

Roinand Aguila 108 Oct 23, 2022
An interactive document scanner built in Python using OpenCV

The scanner takes a poorly scanned image, finds the corners of the document, applies the perspective transformation to get a top-down view of the document, sharpens the image, and applies an adaptive color threshold to clean up the image.

Kushal Shingote 1 Feb 12, 2022