Official implementation of Character Region Awareness for Text Detection (CRAFT)

I'm trying to export from pth to ONNX format:

import torch
from torch.autograd import Variable
import cv2
import imgproc
from craft import CRAFT

# load net
net = CRAFT()     # initialize
net = net.cuda()
net = torch.nn.DataParallel(net)

net.load_state_dict(torch.load('./craft_mlt_25k.pth'))
net.eval()


# load data
image = imgproc.loadImage('./misc/test.jpg')

# resize
img_resized, target_ratio, size_heatmap = imgproc.resize_aspect_ratio(image, 1280, interpolation=cv2.INTER_LINEAR, mag_ratio=1.5)
ratio_h = ratio_w = 1 / target_ratio

# preprocessing
x = imgproc.normalizeMeanVariance(img_resized)
x = torch.from_numpy(x).permute(2, 0, 1)    # [h, w, c] to [c, h, w]
x = Variable(x.unsqueeze(0))                # [c, h, w] to [b, c, h, w]
x = x.cuda()

# trace export
torch.onnx.export(net,
                  x,
                  'onnx/craft.onnx',
                  export_params=True,
                  verbose=True)

But then encountered this error:

RuntimeError: tuple appears in op that does not forward tuples (VisitNode at /opt/conda/conda-bld/pytorch_1556653114079/work/torch/csrc/jit/passes/lower_tuples.cpp:117)

Followed these issue https://github.com/pytorch/pytorch/issues/5315 and https://github.com/pytorch/pytorch/issues/13397, it turnned out that nn.DataParallel wrapper doesn't support trace export for ONNX.

Is there a workaround for this?

I tested my images first on the web demo, but when I test using the repo locally on my machine I get different results. Is there a difference in the parameters or you are using another weight. I need to know the difference to apply it to my local version.

Website demo result that recognizes line by line[what I need]

Local repo result that recognizes word by word

Loading weights from checkpoint (craft_mlt_25k.pth) Loading weights of refiner from checkpoint (craft_refiner_CTW1500.pth) Traceback (most recent call last):08014857_IMG_0236.jpeg File "test.py", line 169, in file_utils.saveResult(image_path, image[:,:,::-1], polys, dirname=result_folder) File "/home/aiteammor/Downloads/CRAFT-pytorch/file_utils.py", line 62, in saveResult cv2.polylines(np.array(img), [poly.reshape((-1, 1, 2))], True, color=(0, 0, 255), thickness=2) TypeError: Layout of the output array img is incompatible with cv::Mat (step[ndims-1] != elemsize or step[1] != elemsize*nchannels)

This is my reimplementation results. I found that it is not good enough and recall is much lower than your results. Could you give me some advice or matters needing attention. Thanks a lot

First of all thanks very much for your model. I have used your pretrained model for text detection. I have got the bounding box coordinates as txt files. The resulted bounding boxes are shuffled and i could not sort it . When there is a presence of curved text in a same line like in the image below, the order gets shuffled and i need to sort it before passing it to text extraction model.

I have not used POLY mode. For the above image, the model outputs a txt file in which the bb coordinates are as follows. I have added the detected text for better explanation of my problem. In this case the detection order is

146,36,354,34,354,82,146,84 "Australian"

273,78,434,151,411,201,250,129 "Collection"

146,97,250,97,250,150,146,150 "vine"

77,166,131,126,154,158,99,197 "Old"

242,215,361,241,354,273,235,248 "Valley"

140,247,224,219,234,250,150,277 "Eden"

194,298,306,296,307,324,194,325 "Shiraz"

232,406,363,402,364,421,233,426 "Vintage"

152,402,216,405,215,425,151,422 "2008"

124,470,209,480,207,500,122,490 "South"

227,481,387,472,389,494,228,503 "Australia"

222,562,312,564,311,585,222,583 "Gibson"

198,564,217,564,217,584,198,584 "by"

386,570,421,570,421,600,386,600 "750 ml"

But the expected output is Australian->old->vine->collection->Eden->Valley->shiraz->2008->vintage->south->Australia->by->GIBSON->750ml.

Hey, so I got this working on my mac pretty easily. However I just can't seem to get it to work on a docker image.

It seems to crash when executing this line https://github.com/clovaai/CRAFT-pytorch/blob/master/basenet/vgg16_bn.py#L61

Tried to debug through, but don't really have any experience with pdb so got nowhere there. I suspect my Dockerfile isn't setup right. Did anyone manage to get this working or have similar issues? My dockerfile is below.

FROM ubuntu:18.04

ENV LANG=C.UTF-8 LC_ALL=C.UTF-8
 
# Install tools
RUN apt-get update \
        && apt-get --yes install \
        wget \
        curl \
        ca-certificates \
        sudo \
        git \
        bzip2 \
        libx11-6 libglib2.0-0 \
        libsm6 libxext6 libxrender-dev \
        && rm -rf /var/lib/apt/lists/*
 

 # Create a working directory
RUN mkdir /app
WORKDIR /app


# Create a non-root user and switch to it
RUN adduser --disabled-password --gecos '' --shell /bin/bash user \
 && chown -R user:user /app
RUN echo "user ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/90-user
USER user

# All users can use /home/user as their home directory
ENV HOME=/home/user
RUN chmod 777 /home/user

# Install Miniconda
RUN curl -so ~/miniconda.sh https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh \
 && chmod +x ~/miniconda.sh \
 && ~/miniconda.sh -b -p ~/miniconda \
 && rm ~/miniconda.sh
 
# Set environment PATH
ENV PATH=/home/user/miniconda/bin:$PATH
ENV CONDA_AUTO_UPDATE_CONDA=false

RUN conda install --yes python=3.6

COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .

CMD ["python", "test.py"]

Dataset like ICDAR2015 has word level annotation, so that craft can be evaluated easily by IoU.

How can we evaluate the performance of craft with annotation like text region? Such as labeling the entire text area "Hello World!" with a single rectangle.

It is my pleasure to get the answers.

@YoungminBaek Now that Craft have detected the boxes and saved them into a .txt file, can you add a script to extract/crop the detected boxes into images.

Hi, thanks for your work. I have some questions about the detail of generating pseudo ground-truth.

1. In Figure 6, I am confused that why can't we just input the whole image to generate the pseudo ground truth (non-text region area are padded with 0).

2. How to crop the image? As I understand for the paper Figure 4, we should first crop the text region and then feed them to the network. Is that right? However, in the ICDAR2015, the word ground truths aren't the normal rectangle. 1) How do we crop it? (rotate it and then crop?).

In the paper you state "Moreover, with our polygon representation, the curved images can be rectified into straight text images, which are also shown in Fig. 11. We believe this ability for rectification can further be of use for recognition tasks." My question is from a set of polygon points, how can I reconstruct the rectified image? Can you kindly point me towards the correct direction? many thanks in advance

Hi @YoungminBaek. Thanks for the excellent works.

I'm trying to replicate LinkRefiner but not achieve a good result as you did.

May I ask:

1. You said that the width of each line in LinkRefiner ground truth is proportional to the distance between paired control points. How much is it exactly? 40% of the distance between paired control points, 60%, ..etc ?

2. What loss function do you used to train the LinkRefiner? Currently I'm using Focal Loss ;_;

Hope you can answer these question :P

I am extracting the texts box from the image, but one thing that I am confused about is what is the significance of score_text in the following code. bboxes, polys, score_text = test_net(net, image, args.text_threshold, args.link_threshold, args.low_text, args.cuda, args.poly, refine_net)

Is there any confidence threshold for the extracted texts?

While reading the images using the scikit-image skimage.io module i.e img = io.imread(img_file) in imgproc.py python file it throws an error : OSError: image file is truncated (0 bytes not processed) error. How to fix it?

Hi everyone, I am sorry for bothering, maybe that's a trivial question, but I need to "merge" the bounding boxes detected and save them to an image. I have read this issue (https://github.com/clovaai/CRAFT-pytorch/issues/39#issuecomment-526930324) but I believe this code allows to save an image for every bounding box I need an image with more bounding boxes merged. In other terms, with text "Hello World" on the image, I don't want two different cropped images "Hello" and "World", but only one cropped image with "Hello World"

Any suggestions?

Thank you again!

Hi everyone!

craft_mlt_25k.pth uses 3 GB of video ram. I think this is overkill for my work. Do you have any lightweight CRAFT network ?

Thank you!

Batu.

Only for some images the text detection fails with the error: cv2.polylines(img, [poly.reshape((-1, 1, 2))], True, color=(0, 0, 255), thickness=2) cv2.error: OpenCV(4.6.0) :-1: error: (-5:Bad argument) in function 'polylines'

Overload resolution failed:

• Layout of the output array img is incompatible with cv::Mat
• Expected Ptrcv::UMat for argument 'img'

I have looked for a difference between the working and to failed images but cannot seem to find a difference?