I clone the project , changed nothing. except change the train&val.pkl path in the config file , use your images retrain. but i think my result is not good . i check the data , found nothing . here is my trian PS img;

i don't know how to check and find the problem

pure ViT structure

We discussed pure ViT structure at https://github.com/lukas-blecher/LaTeX-OCR/discussions/131 .

1. Initially, I used a pure ViT (6ecc3f4). But the encoder was just not performing very well. The model produced latex code but it has nothing to do with the input image.

And I do come up with same result, the model can't converge. In fact, I would hope that larger pure vit can achieve high performance, it really frustrated me. But in recent days, https://github.com/lukas-blecher/LaTeX-OCR/issues/147#issuecomment-1125837771 give me some idea, because the training loss curve is so familiar like pure vit training curve, so I think the reason why pure vit can't fit maybe due to batch size.

I taken and modified models.py from 844bc219a9469fa7e9dfc8626f74a705bd194d69.

Here is the good news, it's working.

How to use

# for vit
 python -m pix2tex.train --config model/settings/config-vit.yaml --structure vit
# for hybrid, default is hybrid
 python -m pix2tex.train --config model/settings/config.yaml --structure hybrid
 python -m pix2tex.train --config model/settings/config.yaml

Data parallelism

I think multi-GPU training can save more time and a larger batch size, so refer to some documents and blogs and make such changes. Also, it's compatible with one GPU.

How to use

#for one GPU
export CUDA_VISIBLE_DEVICES=6
 python -m pix2tex.train --config model/settings/config-vit.yaml --structure vit
#for multi GPU
export CUDA_VISIBLE_DEVICES=6,7
 python -m pix2tex.train --config model/settings/config-vit.yaml --structure vit

References:

1. Technique 1: Data Parallelism
2. data_parallel_tutorial.ipynb
3. https://github.com/jytime/Mask_RCNN_Pytorch/issues/2#issuecomment-423718196

small modify

I think both hybrid and pure vit work together, why not put them together. so create a folder named as structures.

the branch is based on 720978d8c469780ed070d041d5795c55b705ac1b, please feel free to correct any inappropriate code.😁

When I run latexocr in Windows Terminal, I got this error:

Traceback (most recent call last):
  File "F:\Python\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "F:\Python\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Venv\pytorch_1.11\Scripts\latexocr.exe\__main__.py", line 7, in <module>
  File "F:\Venv\pytorch_1.11\lib\site-packages\pix2tex\gui.py", line 316, in main
    with in_model_path():
  File "F:\Python\lib\contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "F:\Venv\pytorch_1.11\lib\site-packages\pix2tex\utils\utils.py", line 161, in in_model_path
    with path('pix2tex', 'model') as model_path:
  File "F:\Python\lib\contextlib.py", line 119, in __enter__
    return next(self.gen)
  File "F:\Python\lib\importlib\resources.py", line 175, in _path_from_reader
    opener_reader = reader.open_resource(norm_resource)
  File "<frozen importlib._bootstrap_external>", line 1055, in open_resource
PermissionError: [Errno 13] Permission denied: 'F:\\Venv\\pytorch_1.11\\lib\\site-packages\\pix2tex\\model'

Hello!

I am currently on macOS catalina 10.15.7, running python3.7. I went through the installation, and met the same problem as described in #33. I have tried all the fixes, including downgrading x-transformers, tuning the temperature, taking pictures of larger equations, and downloading the weights.pth and image_resizer.pth into the checkpoints folder, yet the program still gives out wrong results. Is there possibly some incompatibility with other distributions for my mac?

Here are my installed packages.

song12301(Jeb) ~ % pip3 list
Package                              Version
------------------------------------ ---------
albumentations                       1.1.0
certifi                              2021.10.8
chardet                              4.0.0
charset-normalizer                   2.0.10
click                                8.0.3
Cython                               0.29.26
einops                               0.4.0
entmax                               1.0
filelock                             3.4.2
idna                                 3.3
imageio                              2.14.0
imagesize                            1.3.0
joblib                               1.1.0
munch                                2.5.0
networkx                             2.6.3
numpy                                1.22.1
opencv-python-headless               4.5.5.62
packaging                            21.3
pandas                               1.4.0
Pillow                               9.0.0
pip                                  21.3.1
pynput                               1.7.6
pyobjc-core                          8.1
pyobjc-framework-ApplicationServices 8.1
pyobjc-framework-Cocoa               8.1
pyobjc-framework-Quartz              8.1
pyparsing                            3.0.7
PyQt5                                5.15.6
PyQt5-Qt5                            5.15.2
PyQt5-sip                            12.9.0
PyQtWebEngine                        5.15.5
PyQtWebEngine-Qt5                    5.15.2
python-dateutil                      2.8.2
python-Levenshtein                   0.12.2
pytz                                 2021.3
PyWavelets                           1.2.0
PyYAML                               6.0
qudida                               0.0.4
regex                                2022.1.18
requests                             2.27.1
sacremoses                           0.0.47
scikit-image                         0.19.1
scikit-learn                         1.0.2
scipy                                1.7.3
screeninfo                           0.8
setuptools                           60.5.0
six                                  1.16.0
threadpoolctl                        3.0.0
tifffile                             2021.11.2
timm                                 0.5.4
tokenizers                           0.9.4
torch                                1.10.1
torchtext                            0.11.1
torchvision                          0.11.2
tqdm                                 4.62.3
typing_extensions                    4.0.1
urllib3                              1.26.8
wheel                                0.37.1
wincertstore                         0.2
x-transformers                       0.12.1

Thanks a lot for your time!

(tf_1.12) root@f15b165683e6:/home/code/LaTeX-OCR# python dataset/dataset.py --equations latex-ocr-data/crohme/CROHME_math.txt --vocab-size 8000 --out crohme-tokenizer.json Generate tokenizer Traceback (most recent call last): File "dataset/dataset.py", line 244, in generate_tokenizer(args.equations, args.out, args.vocab_size) File "dataset/dataset.py", line 228, in generate_tokenizer trainer = BpeTrainer(special_tokens=["[PAD]", "[BOS]", "[EOS]"], vocab_size=vocab_size, show_progress=True) TypeError: 'str' object cannot be interpreted as an integer (tf_1.12) root@f15b165683e6:/home/code/LaTeX-OCR# how to fix it ?

here is my own latex data line ,there is no space . here is you math.txt latex line, there is lots of sapce in the line , i want to konw how to convert my own latex line to you style?

1. normalization influenced by alpha.

grayscale = (data[..., 0]-data[..., 0].min()) / (data[..., 0].max()-data[..., 0].min())*255

2. paste size does not match

padded.paste(im, (0, 0, im.size[0], im.size[1]))

3. pad wrong pixel when text is inverted cause the text has inverted some times, but the padded pixel is hard code to 255

padded = Image.new('L', dims, 255)

I notice this will cause error recognition when the text's pixel is 255 and the pad pixel is also 255, then that pad part will be recognized as text.

Hi, I constantly get a TypeError when trying to run gui.py.

Traceback (most recent call last):
  File "/Applications/LaTeX-OCR-main/gui.py", line 266, in mouseReleaseEvent
    img = ImageGrab.grab(bbox=(x1, y1, x2, y2), all_screens=True)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/PIL/ImageGrab.py", line 42, in grab
    im_resized = im.resize((right - left, bottom - top))
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/PIL/Image.py", line 2040, in resize
    im = im.resize(size, resample, box)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/PIL/Image.py", line 2062, in resize
    return self._new(self.im.resize(size, resample, box))
TypeError: integer argument expected, got float
zsh: abort      '/usr/local/bin/python3.8' '/Applications/LaTeX-OCR-main/gui.py'
phil@Air-von-Phil LaTeX-OCR-main % 

Pix2Tex works, even though I get an unsupported hardware notification. I'm on a M1 MacBook Air. Any help would be great. Kind regards

Hey Buddy, first of all, great work here, I followed the instructions, and it works amazingly for most of the equations. Now I want to train the model for handwritten equations to predict LaTeX. Following README.MD, I'm trying to generate the tokenizer for the CROHME dataset for which I entered the following command:

python dataset/dataset.py --equations dataset/CROHME_math.txt --vocab-size 8000 --out tokenizer.json

I'm getting the following error:

Generate tokenizer
Traceback (most recent call last):
  File "C:\Users\Saad\OneDrive\Desktop\LaTeX-OCR-main\dataset\dataset.py", line 244, in <module>
    generate_tokenizer(args.equations, args.out, args.vocab_size)
  File "C:\Users\Saad\OneDrive\Desktop\LaTeX-OCR-main\dataset\dataset.py", line 228, in generate_tokenizer     
    trainer = BpeTrainer(special_tokens=["[PAD]", "[BOS]", "[EOS]"], vocab_size=vocab_size, show_progress=True)
TypeError: 'str' object cannot be interpreted as an integer

How am I to solve this? I'm assuming that I'll first generate the tokenizer.json file and then train the model on CROHME Dataset. Once I'm done, I'll be able to input handwritten equations and get the corresponding LaTeX. Am I on the right track? Thanks!

Take a snapshot of the clipboard image, if any. Only macOS and Windows are currently supported. https://pillow.readthedocs.io/en/stable/reference/ImageGrab.html#PIL.ImageGrab.grabclipboard

pix2tex.py uses this method but it is not supported in Linux. Need to find a workaround.

When using the --no-cuda argument, it returns an error.

(env) λ python pix2tex.py --no-cuda
Traceback (most recent call last):
  File "H:\pytlat\ocr\pix2tex.py", line 84, in <module>
    args, model, tokenizer = initialize(args)
  File "H:\pytlat\ocr\pix2tex.py", line 33, in initialize
    model.load_state_dict(torch.load(args.checkpoint))
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 853, in _load
    result = unpickler.load()
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 845, in persistent_load
    load_tensor(data_type, size, key, _maybe_decode_ascii(location))
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 834, in load_tensor
    loaded_storages[key] = restore_location(storage, location)
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 175, in default_restore_location
    result = fn(storage, location)
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 151, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "H:\pytlat\env\lib\site-packages\torch\serialization.py", line 135, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

I use torch 1.7.+cpu, cuda version is not installed, and can't use cuda.

When I try to recognize a series of equations, I get an error，

here is the equations image.

here is the result.

Is there any way to fix this error? Thank you for taking the time to look at this issue and reply to it.

Hi all - currently looking into fine-tuning this model and have run into an issue with images of varying different sizes. For this example:

max_height = 192, max_width = 672, patch_size=16

The error causing line is here:

        x += self.pos_embed[:, pos_emb_ind]

(pix2tex.models.hybrid line 25 in CustomVisionTransformer forward_features)

If I have an image of size 522 x 41, this line will throw an error. X consists of 99 patches (+ the cls tokens) making it size [100, 256]

However, the positional embedding indices are only 66 in length. I am currently investigating this issue but don't quite understand the formula used to compute how many positional embedding indicies we are going to need. Right now it is computing 66 different indicies when we should be getting 100 different indicies. I think the issue arises when convolutions from the resnet embedder overlap and the formula doesn't account for this (it requires the image to be divisible by patch_size X patch_size for this formula to work).

If anyone has any thoughts on how to fix this let me know! I'm definitely no computer vision expert but I believe a simple change to account for overlapping convolutions in the embedding may be enough to fix this!

Sorry to bother you again. I found that different resolution of an image will have a big impact on the recognition effect. For example, if the original resolution of some input images is reduced to 80% or enlarged to 120 percent, the recognition effect will change significantly, and the identification results will be too uncertain.

There are some errors in formula recognition. The following are the results of the identification: y=\prod x_{i}^{v}
y=\sum w_{i} x These are the images generated in cli.py:

As you've mentioned, to generate dataset the command should go like this "python -m pix2tex.dataset.dataset --equations path_to_textfile --images path_to_images --out dataset.pkl" and the dataset class basically takes the filename and splits it from "." and takes that first splited part as the line number of the equation file and grab that line as the ground truth data of the image.(Correct me if I'm wrong). My question is how can I generate my own dataset to train the model. Is it necessary to have "000" kind of stuffs before the file name of the image ? Because your google drive data is kind of confusing ! it contains

1. Train Images -> 158480 number of files
2. Valid Images -> 6780 number of files
3. Test Images -> 30637 number of files 195882 Files in total but your ground truth data contains more than that. I found 234484 lines in math.txt file . Can you please explain how can align my own data to generate the dataset.pkl file. Or how can I name the image files and how can I write my math.txt file ?