1 Repositories
Python BPEer Libraries
Train BPE with fastBPE, and load to Huggingface Tokenizer.
BPEer Train BPE with fastBPE, and load to Huggingface Tokenizer. Description The BPETrainer of Huggingface consumes a lot of memory when I am training
1 Dec 23, 2021