Hi, thanks for the great repo!

• I followed the guide in the README file and want to train the text-line detection model
• I also prepared a dataset with COCO format, same as the MMOCR's repo and psenet_r50_fpnf_600e_pubtabnet.py but got the following error. Seem like it occurs with an empty object but I did not sure the error came from Did you face this issue when training the text-line detection model? Or do you have an idea about how to fix this issue?

Thanks in advance!

您好，按照Install.md安装了mmdetection、mmcv-full、mmocr， 但是运行 sh ./table_recognition/expr/table_recognition_dist_train.sh 报错： keyError:'TABLEMASTER is not in the {registry.name} registry'， 请问有什么建议的解决方法吗？

Hi, thanks for publishing code. when I try to use text recognition checkpoint, find that the alphabet generated by mine is different from yours. This could be due to the fact that I only used part of the data for training master test recongnition model. Could you provide the ./tools/data/alphabet/textline_recognition_alphabet.txt file?

return build_from_cfg(cfg, registry, default_args)

File "/python3.6/site-packages/mmcv/utils/registry.py", line 44, in build_from_cfg f'{obj_type} is not in the {registry.name} registry') KeyError: 'MASTER is not in the detector registry'

我对表格结构识别模型训练一轮后想用test_imgs.py这个脚本来测试一下模型能不能跑通，结果报了一个错，请问这是因为mmcv的版本问题吗，还是什么其他问题？谢谢！（我是用pip install mmcv-full==1.3.4 -f https://download.openmmlab.com/mmcv/dist/cu102/torch1.7.0/index.html这个命令安装mmcv的, mmocr是通过在当前repo的根路径下运行pip install -v -e .安装的，安装后mmocr的版本是0.2.0）

python tools/test_imgs.py configs/textrecog/master/table_master_ResnetExtract_Ranger_0705.py experiments/mmocr_table_recognition/0705/latest.pth /home/datasets/ocr/table/pubtabnet/test /home/datasets/ocr/table/pubtabnet/test.list
Use load_from_local loader
[>>>>>>>>>>>>                                      ] 1/4, 22075.3 task/s, elapsed: 0s, ETA:     0sTraceback (most recent call last):
  File "tools/test_imgs.py", line 165, in <module>
    main()
  File "tools/test_imgs.py", line 151, in main
    result = inference_detector(model, img_path)
  File "/home/xray/homework/table/TableMASTER-mmocr/mmdetection-2.11.0/mmdet/apis/inference.py", line 117, in inference_detector
    test_pipeline = Compose(cfg.data.test.pipeline)
  File "/home/xray/homework/table/TableMASTER-mmocr/mmdetection-2.11.0/mmdet/datasets/pipelines/compose.py", line 22, in __init__
    transform = build_from_cfg(transform, PIPELINES)
  File "/home/anaconda3/envs/pytorch1.7/lib/python3.6/site-packages/mmcv/utils/registry.py", line 44, in build_from_cfg
    f'{obj_type} is not in the {registry.name} registry')
KeyError: 'TableResize is not in the pipeline registry'

Have you tried to convert the MASTER-based model (included textline-recognition & table recognition) to ONNX or TensorRT format?

I followed the tutorial of MMOCR and MMDet to convert those models to ONNX format, here is the output error Do you have any idea for handle this issue? Thanks in advance.

Hi,

I tried to use table-master (alone, not the end2end model) but it takes 18s just to infer on one image from pubtabnet (~500 pixels width) plus it takes about 2GB memory from the GPU (1 tesla K80).

Therefore I wanted to know if it is the normal memory consumption and processing time for table-master and also I wanted to know if there is any mean to reduce the memory consumption and increase the processing speed. Because it is unfeasible to use a model that takes 18s to infer on one image.

Thank you in advance for your answer

After run python data_preprocess.py, I want to train table structure recognition model TableMASTER.

But when I run ./table_recognition/expr/table_recognition_dist_train.sh, it raise KeyError: 'TABLEMASTER is not in the models registry'

def distance_rule_match(end2end_indexes, end2end_bboxes, master_indexes, master_bboxes):
    """
    Get matching between no-match end2end bboxes and no-match master bboxes.
    Use min distance to match.
    This rule will only run (no-match end2end nums > 0) and (no-match master nums > 0)
    It will Return master_bboxes_nums match-pairs.
    :param end2end_indexes:
    :param end2end_bboxes:
    :param master_indexes:
    :param master_bboxes:
    :return: match_pairs list, e.g. [[0,1], [1,2], ...]
    """
    min_match_list = []
    for j, master_bbox in zip(master_indexes, master_bboxes):
        min_distance = np.inf
        min_match = [0, 0]  # i, j
        for i, end2end_bbox in zip(end2end_indexes, end2end_bboxes):
            x_end2end, y_end2end = end2end_bbox[0], end2end_bbox[1]
            x_master, y_master = master_bbox[0], master_bbox[1]
            end2end_point = (x_end2end, y_end2end)
            master_point = (x_master, y_master)
            dist = cal_distance(master_point, end2end_point)
            if dist < min_distance:
                min_match[0], min_match[1] = i, j
                min_distance = dist
        min_match_list.append(min_match)
    return min_match_list

About this function, the output may contain several matches [i, *] for one i. But you want to find only one match for a specific i, should we change order of the two loops here?

when tun table_inference.py with one gpu of 2080ti load epoch_16_0.7767.pth model and config files as follows:

base = [ '../../base/default_runtime.py' ]

alphabet_file = '/tools/data/alphabet/structure_alphabet.txt' alphabet_len = len(open(alphabet_file, 'r').readlines()) max_seq_len = 500

start_end_same = False label_convertor = dict( type='TableMasterConvertor', dict_file=alphabet_file, max_seq_len=max_seq_len, start_end_same=start_end_same, with_unknown=True)

if start_end_same: PAD = alphabet_len + 2 else: PAD = alphabet_len + 3

model = dict( type='TABLEMASTER', backbone=dict( type='TableResNetExtra', input_dim=3, gcb_config=dict( ratio=0.0625, headers=1, att_scale=False, fusion_type="channel_add", layers=[False, True, True, True], ), layers=[1,2,5,3]), encoder=dict( type='PositionalEncoding', d_model=512, dropout=0.2, max_len=5000), decoder=dict( type='TableMasterDecoder', N=3, decoder=dict( self_attn=dict( headers=8, d_model=512, dropout=0.), src_attn=dict( headers=8, d_model=512, dropout=0.), feed_forward=dict( d_model=512, d_ff=2024, dropout=0.), size=512, dropout=0.), d_model=512), loss=dict(type='MASTERTFLoss', ignore_index=PAD, reduction='mean'), bbox_loss=dict(type='TableL1Loss', reduction='sum'), label_convertor=label_convertor, max_seq_len=max_seq_len)

TRAIN_STATE = True img_norm_cfg = dict(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]) train_pipeline = [ dict(type='LoadImageFromNdarrayV2'), dict( type='TableResize', keep_ratio=True, long_size=480), dict( type='TablePad', size=(480, 480), pad_val=0, return_mask=True, mask_ratio=(8, 8), train_state=TRAIN_STATE), dict(type='TableBboxEncode'), dict(type='ToTensorOCR'), dict(type='NormalizeOCR', **img_norm_cfg), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'text', 'scale_factor', 'bbox', 'bbox_masks', 'pad_shape' ]), ]

valid_pipeline = [ dict(type='LoadImageFromNdarrayV2'), dict( type='TableResize', keep_ratio=True, long_size=480), dict( type='TablePad', size=(480, 480), pad_val=0, return_mask=True, mask_ratio=(8, 8), train_state=TRAIN_STATE), dict(type='TableBboxEncode'), dict(type='ToTensorOCR'), dict(type='NormalizeOCR', **img_norm_cfg), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'scale_factor', 'img_norm_cfg', 'ori_filename', 'bbox', 'bbox_masks', 'pad_shape' ]), ]

test_pipeline = [ dict(type='LoadImageFromNdarrayV2'), dict( type='TableResize', keep_ratio=True, long_size=480), dict( type='TablePad', size=(480, 480), pad_val=0, return_mask=True, mask_ratio=(8, 8), train_state=TRAIN_STATE), #dict(type='TableBboxEncode'), dict(type='ToTensorOCR'), dict(type='NormalizeOCR', **img_norm_cfg), dict( type='Collect', keys=['img'], meta_keys=[ 'filename', 'ori_shape', 'img_shape', 'scale_factor', 'img_norm_cfg', 'ori_filename', 'pad_shape' ]), ]

dataset_type = 'OCRDataset' #train_img_prefix = '/pubtabnet/pubtabnet/train' #train_anno_file1 = /StructureLabel_train' train_img_prefix = "pubtabnet/pubtabnet/train" train_anno_file1 = "StructureLabel_train"

train_img_prefix = ''

train_anno_file1 = ''

train1 = dict( type=dataset_type, img_prefix=train_img_prefix, ann_file=train_anno_file1, loader=dict( type='TableMASTERLmdbLoader', repeat=1, max_seq_len=max_seq_len, parser=dict( type='TableMASTERLmdbParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=' ')), pipeline=train_pipeline, test_mode=False)

valid_img_prefix = /pubtabnet/pubtabnet/val'

valid_anno_file1 = /StructureLabel_val'

valid_img_prefix = '/pubtabnet/pubtabnet/val' valid_anno_file1 = '/StructureLabel_val' valid = dict( type=dataset_type, img_prefix=valid_img_prefix, ann_file=valid_anno_file1, loader=dict( type='TableMASTERLmdbLoader', repeat=1, max_seq_len=max_seq_len, parser=dict( type='TableMASTERLmdbParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=' ')), pipeline=valid_pipeline, dataset_info='table_master_dataset', test_mode=True)

test_img_prefix = /pubtabnet/pubtabnet/val'

test_anno_file1 = '/StructureLabel_val'

test_img_prefix = '/pubtabnet/pubtabnet/val' test_anno_file1 = '/StructureLabel_val' test = dict( type=dataset_type, img_prefix=test_img_prefix, ann_file=test_anno_file1, loader=dict( type='TableMASTERLmdbLoader', repeat=1, max_seq_len=max_seq_len, parser=dict( type='TableMASTERLmdbParser', keys=['filename', 'text'], keys_idx=[0, 1], separator=' ')), pipeline=test_pipeline, dataset_info='table_master_dataset', test_mode=True)

data = dict( samples_per_gpu=4, workers_per_gpu=2, train=dict(type='ConcatDataset', datasets=[train1]), val=dict(type='ConcatDataset', datasets=[valid]), test=dict(type='ConcatDataset', datasets=[test]))

optimizer

optimizer = dict(type='Ranger', lr=1e-3) optimizer_config = dict(grad_clip=dict(max_norm=35, norm_type=2))

optimizer_config = dict(grad_clip=None)

learning policy

lr_config = dict( policy='step', warmup='linear', warmup_iters=50, warmup_ratio=1.0 / 3, step=[12, 15]) total_epochs = 17

evaluation

evaluation = dict(interval=1, metric='acc')

fp16

fp16 = dict(loss_scale='dynamic')

checkpoint setting

checkpoint_config = dict(interval=1)

log_config

log_config = dict( interval=100, hooks=[ dict(type='TextLoggerHook')

])

yapf:enable

dist_params = dict(backend='nccl') log_level = 'INFO' load_from = None resume_from = None workflow = [('train', 1)]

if raise find unused_parameters, use this.

find_unused_parameters = True

ret = input.softmax(dim) RuntimeError: CUDA out of memory. Tried to allocate 30.00 MiB (GPU 0; 10.76 GiB total capacity; 444.10 MiB already allocated; 24.56 MiB free; 604.00 MiB reserved in total by PyTorch)

it seems need more gpu memory

Trying to run table recognition demo script in google colab , gdrive mounted and repo is inside gdrive. While runninh the table recognition demoscript i am getting error below >>>>>>>

Use load_from_local loader Traceback (most recent call last): File "./table_recognition/demo/demo_cp.py", line 65, in master_inference = Recognition_Inference(args.master_config, args.master_checkpoint) File "/content/drive/MyDrive/work/TableMASTER-mmocr/table_recognition/table_inference.py", line 99, in init super().init(config_file, checkpoint_file) File "/content/drive/MyDrive/work/TableMASTER-mmocr/table_recognition/table_inference.py", line 38, in init self.model = build_model(config_file, checkpoint_file) File "/content/drive/MyDrive/work/TableMASTER-mmocr/table_recognition/table_inference.py", line 25, in build_model model = init_detector(config_file, checkpoint=checkpoint_file, device=device) File "/content/drive/MyDrive/work/TableMASTER-mmocr/mmdetection-2.11.0/mmdet/apis/inference.py", line 31, in init_detector config = mmcv.Config.fromfile(config) File "/usr/local/lib/python3.8/dist-packages/mmcv/utils/config.py", line 254, in fromfile cfg_dict, cfg_text = Config._file2dict(filename, File "/usr/local/lib/python3.8/dist-packages/mmcv/utils/config.py", line 148, in _file2dict mod = import_module(temp_module_name) File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 1014, in _gcd_import File "", line 991, in _find_and_load File "", line 975, in _find_and_load_unlocked File "", line 671, in _load_unlocked File "", line 843, in exec_module File "", line 219, in _call_with_frames_removed File "/tmp/tmp8emb1yyv/tmptcqvifrk.py", line 6, in FileNotFoundError: [Errno 2] No such file or directory: './tools/data/alphabet/textline_recognition_alphabet.txt' Exception ignored in: <function _TemporaryFileCloser.del at 0x7fe26a3dc940> Traceback (most recent call last): File "/usr/lib/python3.8/tempfile.py", line 579, in del self.close() File "/usr/lib/python3.8/tempfile.py", line 575, in close unlink(self.name) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp8emb1yyv/tmptcqvifrk.py'