The following error message was the result of trying a training (based on fraktur historical + GT):
Error message
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.757118). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.680600). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (2.302509). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (1.631931). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (2.153755). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.732034). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.661815). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.718801). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.743117). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.712587). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.963019). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.834908). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.744983). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.694347). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.802638). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.771759). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.675146). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (1.636742). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (1.748071). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (1.423025). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.787484). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.808512). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (3.172735). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (3.214929). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (3.159862). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.875830). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.835742). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.827941). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.780199). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.717965). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (1.834584). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (2.468633). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (2.007760). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.707687). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.761901). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (19.359528). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (24.881613). Check your callbacks.
WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (7.594938). Check your callbacks.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
save_weights_to_hdf5_group(model_weights_group, model_layers)
save_weights_to_hdf5_group(model_weights_group, model_layers)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 636, in save_weights_to_hdf5_group
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 636, in save_weights_to_hdf5_group
param_dset[:] = val
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
param_dset[:] = val
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/dataset.py", line 708, in __setitem__
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/dataset.py", line 708, in __setitem__
self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5d.pyx", line 222, in h5py.h5d.DatasetID.write
File "h5py/h5d.pyx", line 222, in h5py.h5d.DatasetID.write
File "h5py/_proxy.pyx", line 132, in h5py._proxy.dset_rw
File "h5py/_proxy.pyx", line 132, in h5py._proxy.dset_rw
File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
OSError: Can't write data (file write failed: time = Tue Dec 1 08:28:02 2020
, filename = '/tmp/calamari3umodg4c/fold_2/model_00000453.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0xbb35e60, total write size = 179216, bytes this sub-write = 179216, bytes actually written = 18446744073709551615, offset = 5828608)
During handling of the above exception, another exception occurred:
OSError: Can't write data (file write failed: time = Tue Dec 1 08:28:02 2020
, filename = '/tmp/calamari3umodg4c/fold_4/model_00000438.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0xaba6980, total write size = 265232, bytes this sub-write = 265232, bytes actually written = 18446744073709551615, offset = 5742592)Traceback (most recent call last):
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 753, in on_start
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 753, in on_start
yield
yield
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
total_epochs=epochs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 181, in run_one_epoch
total_epochs=epochs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 181, in run_one_epoch
step += 1
File "/usr/lib/python3.6/contextlib.py", line 88, in exit
step += 1
File "/usr/lib/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 788, in on_batch
next(self.gen)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 788, in on_batch
mode, 'end', step, batch_logs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 239, in _call_batch_hook
mode, 'end', step, batch_logs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 239, in _call_batch_hook
batch_hook(batch, logs)
batch_hook(batch, logs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 528, in on_train_batch_end
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 528, in on_train_batch_end
self.on_batch_end(batch, logs=logs)
self.on_batch_end(batch, logs=logs)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 108, in on_batch_end
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 108, in on_batch_end
self.last_checkpoint = self.make_checkpoint(self.checkpoint_params.output_dir, self.checkpoint_params.output_model_prefix)
self.last_checkpoint = self.make_checkpoint(self.checkpoint_params.output_dir, self.checkpoint_params.output_model_prefix)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 85, in make_checkpoint
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 85, in make_checkpoint
self.model.save(checkpoint_path + '.h5', overwrite=True)
self.model.save(checkpoint_path + '.h5', overwrite=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
signatures, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
signatures, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 120, in save_model_to_hdf5
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 120, in save_model_to_hdf5
f.close()
File "/usr/local/lib/python3.6/dist-packages/h5py/hl/files.py", line 443, in close
f.close()
File "/usr/local/lib/python3.6/dist-packages/h5py/hl/files.py", line 443, in close
h5i.dec_ref(id)
h5i.dec_ref(id)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5i.pyx", line 150, in h5py.h5i.dec_ref
File "h5py/h5i.pyx", line 150, in h5py.h5i.dec_ref
RuntimeError: Problems closing file (file write failed: time = Tue Dec 1 08:28:02 2020
, filename = '/tmp/calamari3umodg4c/fold_4/model_00000438.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0x9ca6b70, total write size = 6144, bytes this sub-write = 6144, bytes actually written = 18446744073709551615, offset = 4096)RuntimeError: Problems closing file (file write failed: time = Tue Dec 1 08:28:02 2020
, filename = '/tmp/calamari3umodg4c/fold_2/model_00000453.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0xb64dbe0, total write size = 6144, bytes this sub-write = 6144, bytes actually written = 18446744073709551615, offset = 4096)
During handling of the above exception, another exception occurred:
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 371, in
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 371, in
main()
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 367, in main
main()
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 367, in main
run(args)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 359, in run
run(args)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 359, in run
progress_bar=not args.no_progress_bars
progress_bar=not args.no_progress_bars
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/trainer.py", line 197, in train
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/trainer.py", line 197, in train
self._run_train(train_net, train_start_time, progress_bar, self.dataset, self.validation_dataset, training_callback)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/trainer.py", line 213, in _run_train
self.run_train(train_net, train_start_time, progress_bar, self.dataset, self.validation_dataset, training_callback)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/trainer.py", line 213, in run_train
train_net.train(train_dataset, val_dataset, checkpoint_params, self.txt_postproc, progress_bar, training_callback)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/tensorflow_model.py", line 332, in train
train_net.train(train_dataset, val_dataset, checkpoint_params, self.txt_postproc, progress_bar, training_callback)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/tensorflow_model.py", line 332, in train
v_cb, es_cb
v_cb, es_cb
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
use_multiprocessing=use_multiprocessing)
use_multiprocessing=use_multiprocessing)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 397, in fit
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 397, in fit
prefix='val')
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
prefix='val')
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 757, in on_start
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 757, in on_start
self.callbacks._call_end_hook(mode)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 262, in _call_end_hook
self.callbacks._call_end_hook(mode)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 262, in _call_end_hook
self.on_train_end()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 379, in on_train_end
self.on_train_end()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 379, in on_train_end
callback.on_train_end(logs)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 74, in on_train_end
callback.on_train_end(logs)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 74, in on_train_end
version='last')
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 85, in make_checkpoint
version='last')
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 85, in make_checkpoint
self.model.save(checkpoint_path + '.h5', overwrite=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
self.model.save(checkpoint_path + '.h5', overwrite=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
signatures, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
signatures, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 92, in save_model_to_hdf5
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 92, in save_model_to_hdf5
f = h5py.File(filepath, mode='w')
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 408, in init
f = h5py.File(filepath, mode='w')
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 408, in init
swmr=swmr)
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 179, in make_fid
swmr=swmr)
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 179, in make_fid
fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 108, in h5py.h5f.create
File "h5py/h5f.pyx", line 108, in h5py.h5f.create
OSError: Unable to create file (file write failed: time = Tue Dec 1 08:28:02 2020
, filename = '/tmp/calamari3umodg4c/fold_2/model_last.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0x7182fa8, total write size = 96, bytes this sub-write = 96, bytes actually written = 18446744073709551615, offset = 0)OSError: Unable to create file (file write failed: time = Tue Dec 1 08:28:02 2020
, filename = '/tmp/calamari3umodg4c/fold_4/model_last.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0x67ecef8, total write size = 96, bytes this sub-write = 96, bytes actually written = 18446744073709551615, offset = 0)
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
save_weights_to_hdf5_group(model_weights_group, model_layers)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 636, in save_weights_to_hdf5_group
param_dset[:] = val
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/dataset.py", line 708, in setitem
self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5d.pyx", line 222, in h5py.h5d.DatasetID.write
File "h5py/_proxy.pyx", line 132, in h5py._proxy.dset_rw
File "h5py/_proxy.pyx", line 93, in h5py._proxy.H5PY_H5Dwrite
OSError: Can't write data (file write failed: time = Tue Dec 1 08:28:09 2020
, filename = '/tmp/calamari3umodg4c/fold_3/model_00000481.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0xb465720, total write size = 49040, bytes this sub-write = 49040, bytes actually written = 18446744073709551615, offset = 2367488)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 753, in on_start
yield
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 342, in fit
total_epochs=epochs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 181, in run_one_epoch
step += 1
File "/usr/lib/python3.6/contextlib.py", line 88, in exit
next(self.gen)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 788, in on_batch
mode, 'end', step, batch_logs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 239, in _call_batch_hook
batch_hook(batch, logs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 528, in on_train_batch_end
self.on_batch_end(batch, logs=logs)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 108, in on_batch_end
self.last_checkpoint = self.make_checkpoint(self.checkpoint_params.output_dir, self.checkpoint_params.output_model_prefix)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 85, in make_checkpoint
self.model.save(checkpoint_path + '.h5', overwrite=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
signatures, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 120, in save_model_to_hdf5
f.close()
File "/usr/local/lib/python3.6/dist-packages/h5py/hl/files.py", line 443, in close
h5i.dec_ref(id)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5i.pyx", line 150, in h5py.h5i.dec_ref
RuntimeError: Problems closing file (file write failed: time = Tue Dec 1 08:28:09 2020
, filename = '/tmp/calamari3umodg4c/fold_3/model_00000481.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0xacbac40, total write size = 6144, bytes this sub-write = 6144, bytes actually written = 18446744073709551615, offset = 4096)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 371, in
main()
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 367, in main
run(args)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/train.py", line 359, in run
progress_bar=not args.no_progress_bars
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/trainer.py", line 197, in train
self._run_train(train_net, train_start_time, progress_bar, self.dataset, self.validation_dataset, training_callback)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/trainer.py", line 213, in run_train
train_net.train(train_dataset, val_dataset, checkpoint_params, self.txt_postproc, progress_bar, training_callback)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/tensorflow_model.py", line 332, in train
v_cb, es_cb
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
use_multiprocessing=use_multiprocessing)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 397, in fit
prefix='val')
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_v2.py", line 757, in on_start
self.callbacks._call_end_hook(mode)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 262, in _call_end_hook
self.on_train_end()
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/callbacks.py", line 379, in on_train_end
callback.on_train_end(logs)
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 74, in on_train_end
version='last')
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/backends/tensorflow_backend/callbacks/earlystopping.py", line 85, in make_checkpoint
self.model.save(checkpoint_path + '.h5', overwrite=True)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
signatures, options)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
model, filepath, overwrite, include_optimizer)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 92, in save_model_to_hdf5
f = h5py.File(filepath, mode='w')
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 408, in init
swmr=swmr)
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 179, in make_fid
fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 108, in h5py.h5f.create
OSError: Unable to create file (file write failed: time = Tue Dec 1 08:28:09 2020
, filename = '/tmp/calamari3umodg4c/fold_3/model_last.ckpt.h5', file descriptor = 5, errno = 28, error message = 'No space left on device', buf = 0x5e02c28, total write size = 96, bytes this sub-write = 96, bytes actually written = 18446744073709551615, offset = 0)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/cross_fold_trainer.py", line 27, in train_individual_model
], args.get("run", None), {"threads": args.get('num_threads', -1)}), verbose=args.get("verbose", False)):
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/utils/multiprocessing.py", line 87, in run
raise Exception("Error: Process finished with code {}".format(process.returncode))
Exception: Error: Process finished with code -11
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/calamari-cross-fold-train", line 33, in
sys.exit(load_entry_point('calamari-ocr==1.0.5', 'console_scripts', 'calamari-cross-fold-train')())
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/scripts/cross_fold_train.py", line 80, in main
temporary_dir=args.temporary_dir, keep_temporary_files=args.keep_temporary_files,
File "/usr/local/lib/python3.6/dist-packages/calamari_ocr-1.0.5-py3.6.egg/calamari_ocr/ocr/cross_fold_trainer.py", line 151, in run
pool.map_async(train_individual_model, run_args).get()
File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
raise self._value
Exception: Error: Process finished with code -11
As a result: no recognition process is running anymore - not for the trained project, nor for the others loaded in ocr4all - with the following error: