An expandable and scalable OCR pipeline

When I use the following command:

nidaba batch -b otsu -l tesseract -o kraken:grc_teubner -p spell_check:polytonic_greek -- /cluster/tufts/perseus_ocr/nidaba/teubner/ammonius_1966/*.tif

I get the following error showing up in the celery log for every page, i.e., every page fails with the same error

[2015-08-13 10:32:52,073: ERROR/MainProcess] Task nidaba.ocr.kraken[00db4d89-a411-4098-9036-9865acabe112] raised unexpected: AttributeError("'NoneType' object has no attribute 'predictString'",)
Traceback (most recent call last):
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/nidaba/tasks/helper.py", line 81, in __call__
    ret = super(NidabaTask, self).__call__(*args, **nkwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/nidaba/plugins/kraken.py", line 123, in ocr_kraken
    for rec in rpred.rpred(rnn, img, [(int(x[0]), int(x[1]), int(x[2]), int(x[3])) for x in lines]):
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/kraken/rpred.py", line 132, in rpred
    pred = network.predictString(line)
AttributeError: 'NoneType' object has no attribute 'predictString'

nidaba.yaml looks like this:

# The home directory for Iris to store files created by OCR jobs. For example,
# tifs, jp2s, meta.xml, and abbyy file downloaded from archive.org are stored
# here. Each new job is automatically placed in a uniquely named directory.
storage_path: /cluster/tufts/perseus_ocr/nidaba/OCR/

# URL to the redis database. May be shared with celery.
redis_url: 'redis://127.0.0.1:6379'

# Spell check configuration. Dictionaries are kept on the common medium (i.e.
# at STORAGE_PATH/tuple[0]/tuple[1]). Each spell checker requires a list of
# valid words ('dictionary') and a dictionary containing all variants of words
# attained by deletion of single characters (see nidaba.lex.make_deldict).
lang_dicts:
  polytonic_greek: {dictionary: [dicts, greek.dic],
                    deletion_dictionary: [dicts, del_greek.dic]}
  latin: {dictionary: [dicts, latin.dic],
                    deletion_dictionary: [dicts, del_latin.dic]}

# Ocropus/kraken models
ocropus_models:
  greek: [models, omnibus-2014-05-31-10-16-00087000.pyrnn.gz]
  grc_teubner: [models, teubner-serif-2013-12-16-11-26-00067000.pyrnn.gz]
  atlantean: [models, atlantean.pyrnn.gz]
  fraktur: [models, fraktur.pyrnn.gz]
  fancy_ligatures: [models, ligatures.pyrnn.gz]

# Models solely working with kraken (i.e. models in HDF5 format).
kraken_models:
  default: [models, en-default.hdf5]


# List of plugins to load. Additional fields in the associative array will be
# handed over to the setup function of the plugin.  Be aware that plugins
# utilizing external components that aren't installed will cause nidaba to
# abort. 
plugins_load:
  tesseract: {implementation: capi, # set to either legacy (hOCR
                                              # output in an *.html file),
                                              # direct (hOCR output in an
                                              # *.hocr file), or capi
                                              # (tesseract version >= 3.02)
             tessdata: /cluster/tufts/perseus_ocr_code/tesseract/tessdata} # location of the tessdata
                                                 # path. May also be a storage
                                                 # tuple.
  #ocropus: {}
  kraken: {}
  #leptonica: {}

  Building wheel for pyxDamerauLevenshtein (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/nidaba/nidaba/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-caa2lwvy/pyxDamerauLevenshtein/setup.py'"'"'; __file__='"'"'/tmp/pip-install-caa2lwvy/pyxDamerauLevenshtein/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-wc90vya7
       cwd: /tmp/pip-install-caa2lwvy/pyxDamerauLevenshtein/

[...]

/home/nidaba/nidaba/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   #warning "Using deprecated NumPy API, disable it with " \
    ^~~~~~~
  pyxdameraulevenshtein/pyxdameraulevenshtein.c: In function ‘__Pyx_GetException’:
  pyxdameraulevenshtein/pyxdameraulevenshtein.c:5209:24: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_type’; did you mean ‘curexc_type’?
       tmp_type = tstate->exc_type;

====

Any idea?

When I use the following command to initialize a nidaba batch

nidaba batch -b nlbin:threshold=0.5,zoom=0.5,escale=1.0,border=0.1,perc=80,range=20,low=5,high=90 -l tesseract -o kraken:greek -p spell_check:polytonic_greek phaistos/OCR/OGL/septuagint-dev/raw_ocr/hvd.swete.1.1901.kraken/*.pbm.png

I get the following error message:

Usage: nidaba batch [OPTIONS] FILES...

Error: Invalid value for "--ocr" / "-o": Positional arguments are deprecated!

Has something changed with the API? This command worked without a problem before. I am using nidaba 0.9.3 and kraken 0.4.2 You can check out my nidaba.yaml file on Homer at /home/mmunson/envs/nidaba/etc/nidaba/nidaba.yaml This is also no longer working on the Tufts cluster where it worked before.

When installing with pip install ., I get the following error message:

Downloading/unpacking pyxDamerauLevenshtein==1.3.1 (from nidaba==0.3.14)
  Downloading pyxDamerauLevenshtein-1.3.1.tar.gz (51kB): 51kB downloaded
  Running setup.py (path:/cluster/home/mmunso01/envs/nidaba/build/pyxDamerauLevenshtein/setup.py) egg_info for package pyxDamerauLevenshtein
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/cluster/home/mmunso01/envs/nidaba/build/pyxDamerauLevenshtein/setup.py", line 28, in <module>
        import numpy
    ImportError: No module named numpy
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/cluster/home/mmunso01/envs/nidaba/build/pyxDamerauLevenshtein/setup.py", line 28, in <module>

    import numpy

ImportError: No module named numpy

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /cluster/home/mmunso01/envs/nidaba/build/pyxDamerauLevenshtein
Storing debug log for failure in /cluster/home/mmunso01/.pip/pip.log

By running pip install numpy first and then running pip install ., I was able to overcome this problem. I also had this problem with PyTables so it could be that pyxDamerauLevenshtein needs to be detached and installed later (or numpy detached and installed earlier).

When I run python setup.py download, I get an invalid command error:

python setup.py download
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'download'

Additionally, trying to download these dependencies manually, accessing http://l.unchti.me/nidaba/MANIFEST results in a 404 not found error.

When I run the command

nidaba batch --binarize sauvola:10,20,30,40 --ocr tesseract:grc+eng --willitblend -- /home/mmunson/ddd/extracted_books/uc1.b4034434/*.png

I get the following error message for every image:

Error in pixSauvolaBinarize: whsize too large for image
[2015-05-07 14:41:31,041: ERROR/MainProcess] Task nidaba.binarize.sauvola[28767f13-0328-4efb-8585-083f56555a25] raised unexpected: NidabaLeptonicaException('Binarization failed for unknownreason.',)
Traceback (most recent call last):
  File "/home/mmunson/envs/nidaba/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/mmunson/envs/nidaba/local/lib/python2.7/site-packages/nidaba/tasks/helper.py", line 61, in __call__
    ret = super(NidabaTask, self).__call__(*args, **nkwargs)
  File "/home/mmunson/envs/nidaba/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/mmunson/envs/nidaba/local/lib/python2.7/site-packages/nidaba/plugins/leptonica.py", line 69, in sauvola
    lept_sauvola(input_path, output_path, whsize, factor)
  File "/home/mmunson/envs/nidaba/local/lib/python2.7/site-packages/nidaba/plugins/leptonica.py", line 111, in lept_sauvola
    raise NidabaLeptonicaException('Binarization failed for unknown'
NidabaLeptonicaException: Binarization failed for unknownreason.

From docs:

Per default no dictionaries and OCR models necessary to runs the tests are installed. To download the necessary files run:

$ python setup.py download Afterwards, the test suite can be run:

$ python setup.py nosetests

python setup.py download tries to load archive http://l.unchti.me/nidaba/tests.tar.bz2, but the link is broken

Is it still possible to send multiple languages to tesseract ocr? When I use

-o tesseract:languages=grc+eng,extended=True

as a switch, I get the error message

File "/cluster/home/mmunso01/envs/nidaba/bin/nidaba", line 11, in <module>
    sys.exit(main())
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/click/core.py", line 700, in __call__
    return self.main(*args, **kwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/click/core.py", line 680, in main
    rv = self.invoke(ctx)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/click/core.py", line 1027, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/click/core.py", line 873, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/click/core.py", line 508, in invoke
    return callback(*args, **kwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/nidaba/cli.py", line 240, in batch
    batch.add_task('ocr', alg[0], **kwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/nidaba/nidaba.py", line 662, in add_task
    task_arg_validator(task.get_valid_args(), **kwargs)
  File "/cluster/home/mmunso01/envs/nidaba/lib/python2.7/site-packages/nidaba/nidaba.py", line 72, in task_arg_validator
    raise NidabaInputException('{} not in list of valid values'.format(val))
nidaba.nidabaexceptions.NidabaInputException: grc+eng not in list of valid values

I see that the languages are supposed to be a list in the documentation, but I am not sure how to get nidaba to recognize a list of languages from the command line.

When I use the following command

nidaba batch -b nlbin:threshold=0.5,zoom=0.5,escale=1.0,border=0.1,perc=80,range=20,low=5,high=90 -l tesseract -o kraken:model=greek -p spell_check:dict=polytonic_greek -- /home/mmunson/phaistos/OCR/in_progress/OCR/hvd.swete.1.1901.kraken/*.png

I get the following response

Preparing filestore             [✗]

And then the job exits. The permissions on the the destination folder (phaistos/OCR/in-progress/OCR) are 777, so it doesn't appear to be a permissions problem.

When I run this command:

nidaba batch -b otsu -l tesseract -o tesseract:grc+eng,extended=True -p spell_check:polytonic_greek -- /cluster/tufts/perseus_ocr/nidaba/teubner/ammonius_1966/*.tif

It produces the filestore and it gets through the rgb-to-gray and the binarization steps, but then it seems to hang and does not produce any OCR output. When I request the status, it says that it is pending. I will send my celery log file by email since I don't know how to attach it to this issue.

This may also be the case with other jobs failing. But I have noticed that as soon as I get a NidabaTesseractException because, I think, Tesseract segmentation craps out on empty pages. At this point, it appears that the segmentation jobs that are already in the queue finish, but the jobs after the segmentation jobs do not even start, even for the pages that did not get a segmentation error.

When I run the command

nidaba batch -b nlbin:threshold=0.5,zoom=0.5,escale=1.0,border=0.1,perc=80,range=20,low=5,high=90 -l tesseract -o kraken:model=migne -o kraken:model=migne-njp -o kraken:model=omnibus -p spell_check:language=polytonic_greek,filter_punctuation=False -f tei2txt -- *.png

on Homer, I get the error:

postprocessing.spell_check (2, n): AttributeError: 'list' object has no attribute 'startswith'

Right now no catalog integration is completely missing, except the rather basic metadata task. Let's think about how to generate MODS/MADS records for batches or TEI metadata from MADS/MODS records or some weird combination of the above.

When using Tesseract, the languages used are designated by the "languages" keyword, whereas with kraken/OCRopus they are designated as models. Perhaps this isn't a big deal, but I think it would be nice to simply have a single keyword here so that I can type tesseract:model=grc and kraken:model=grc and have them both work.

When using Kraken and the spell-checker flag, the spell-checker produces correction candidates even for words that should be in the dictionary, e.g., ἀλλὰ.