I think I found a bug in function one_to_one. Suppose there are one predict and two ground truth, the sigma table is [1,1]^T and tau table is [1,0]^T. This is an many to one case but origin code treat it as one to one case.

Hi, I found that the evaluation in this code run extremely slowly and most time-consuming operation in your code is area/area_of_intersection/iou. These functions are based on mask counting, which depends highly on the size of images(some big-size images can be bottleck of computing). I have replaced mask counting operation with polygon coordinate computing(which uses shapely, a geometry lib written in python) so that it highly boosts the evaluatoin process.
Can I make a PR? Hope for your replying, thx.

Hi. According to standard Detval evaluation protocol, "tr = 0.8, tp = 0.4" (which is also your default setting in the MATLAB-code-Eval.m). But you recommend "tr = 0.7 and tp = 0.6" in your Evaluation_Protocol/README.md file.

We recommend tr = 0.7 and tp = 0.6 threshold for a fairer evaluation with polygon ground-truth and detection format.

I am confused about how to set tr and tp when I want to compare my results with other methods (listed in the Tabel Ranking)

Detection (based on DetEval evaluation protocol, unless stated)

| Method | Precision (%) | Recall (%) | F-measure (%) | Published at | |:--------: | :-----: | :----: | :-----: | :-----: | |MSR [paper] | 85.2 | 73.0 | 78.6 | arXiv:1901.02596 | |FTSN [paper] | 84.7 | 78.0 | 81.3 | ICPR2018 | |TextSnake [paper]| 82.7 | 74.5 | 78.4 | ECCV2018 | |TextField [paper] | 81.2 | 79.9 | 80.6 | TIP2019 | |CTD [paper] | 74.0 | 71.0 | 73.0 | PR2019 | |Mask TextSpotter [paper] | 69.0 | 55.0 | 61.3 | ECCV2018 | |TextNet [paper] | 68.2 | 59.5 | 63.5 | ACCV2018 | |Textboxes [paper] | 62.1 | 45.5 | 52.5 | AAAI2017 | |EAST [paper] | 50.0 | 36.2 | 42.0 | CVPR2017 | |Baseline [paper] | 33.0 | 40.0 | 36.0 | ICDAR2017 | |SegLink [paper] | 30.3 | 23.8 | 26.7 | CVPR2017 |

I have some question about rules of annotation. As for curved text, why do you assign them different number of coordinates? What's the maximum number?

Hi I'm reading the Deteval.py script and I'm confusing about the precision calculation in the many_to_many() method. (line 203)

when you calculating recall, you considered the num_qualified_sigma_candidates, but you don't consider num_qualified_tau_candidates when you calculate precision. Moreover, given the following condition (line 199), I think this method really should be called as many_to_one instead of many_to_many.

In summary, I think if you don't consider num_qualified_tau_candidates when you calculate precision in many_to_many method, and you only check if np.sum(local_tau_table[qualified_sigma_candidates, det_id]) >= tp. This method really should be called as many_to_one, and you probably need another many_to_many method.

Hi, can you explain me some scripts?

What is difference between: T3 annotation tool scripts, Detection_Recognition_Annotation_script.m and the python scripts in Evaluation_Protocol?

Some script is used for generating file with groud truth of text on image (T3 ?), one is used for drawing polygon bounding box (Detection_Recognition_Annotation_script.m?) and the main script where magic is happening is Deteval.py or Pascal_VOC.py ?

With the newest ground truth labels, should gt_reading_mod now look like this? I couldn't ge thte evaluation script to run with the existing version

def gt_reading_mod(gt_dir, gt_id):
    gt_id = gt_id.split('.')[0]
    gt = io.loadmat('%s/gt_%s.mat' % (gt_dir, gt_id))
    gt = `gt['gt']`
    return gt

Hi @ckchng , I add an faster version DetEval.py by polygon computing based on shapely. After testing with prediction uploaded in this comment(I have converted .mat format to .txt format as requested, see no_expand_txt.zip.

the speed boosting is significance:

• slow version: about 10 min for 300 testing image
• faster version: within 1 min

precision: I have uploaded the result of no_expand_txt.zip, see result.txt and ori_result.txt, which have almost the same precision, recall and F1 measure.

By the way, python3 is supported in my version with a few codes adding, hope you like it!

eg: img958.jpg [array(['x:'], dtype='<U2') array([[125, 162, 191, 191, 157, 119]], dtype=int16) array(['y:'], dtype='<U2') array([[627, 631, 626, 637, 639, 636]], dtype=int16) array(['DIENST'], dtype='<U6') array([], dtype='<U1')]

I see there is an example floder in Evaluation_Protocol, but in Python_scripts, I don't know where you read them. Can you give us some simple examples about how to evaluate by Python?
Thank you very much.

Hi, thank you so much for providing such a novel dataset.

I want to do some experiments on it, but I cann't find any explanation for the annotation format and the submission format for evaluation (the meaning each element of the gt stands for).

Could you please help me find that?

I installed Python=3.7 and tensorflow=1.13.2.

By running the ./train.sh, following error appears.

@ckchng @princewang1994 @cs-chan @techkang Kindly enlighten. Thanks