Hi,

Thanks for this excellent resource! I've been using BPEmbs in my models since learning about them recently and have found them to work quite well. I'm currently trying to figure out how to use them most effectively with my data which is pre-processed with certain masking tokens for privacy, e.g. <name>, <digit>, etc.

This might be an obvious question, but can you think of a way to extend the vocabulary by adding special tokens to a pre-trained sentencepiece model or is this out of the question? If so, perhaps it would be possible to allow for a certain number of arbitrary special tokens in future iterations of BPEmbs.

Thanks in advance!

It seems that symbols from the sentencepiece model and the associated embeddings are not the same, specifically control symbols are not present in the embedding. For instance, in de.wiki.bpe.op50000.model there are the symbols <s>, </s> and <unk>, but in the associated embedding de.wiki.bpe.vs50000.d300.vectors.bin only <unk> is defined.

Update: Further exploration reveals that the number of common symbols between the aforementioned 2 files are 49631.

Hey, there is an inconsistency between the count of words in the vocabulary file of the English word vectors for 25000 BPE merges and the count of word vectors with a dimensionality of 200. In the vocab file, there is a count of 25000 words, but the .bin file contains 25777 word vectors after loading it into gensim.

Additionally, the order of the word vectors differs from the vocab file. For example, the word "▁explanation" is on position 19387 in the vocab file, but on position 9138 in the .bin file.

It would be very helpful if the index of the vocabulary in all files would be in the same order and have the same count.

Firstly thanks for your efforts in providing the pretrained embeddings.

Unfortunately some of the embeddings are not trained correctly. For instance, the d300 embeddings for the 10k model for English contain 9640 vectors with inf entries, out of 10817 total. It would be great if you can provide your training script, or double check it and upload fixed vectors. The d100 embeddings that you use in the Readme are indeed fine and do not contain any inf values.

Furthermore, while debugging this issue I noticed that the embeddings contain Chinese characters at the following indices[10345, 10451, 10458, 10475, 10514, 10531, 10539, 10541, 10601, 10606, 10609, 10622, 10627, 10632, 10633, 10638, 10657, 10702, 10740, 10750, 10755, 10756, 10762, 10781, 10790, 10791, 10802, 10809, 10810, 10815]. Perhaps it would be sensible to filter out sentences containing Chinese characters from the training corpus?

Dear @all,

I'm trying to load the Dutch BPEmb model with vocabulary size 50k and 100-dimensional embeddings.

bpemb_de = BPEmb(lang="de", vs=50000)

I got an EOFError error:

EOFError Traceback (most recent call last) in 1 import bpemb 2 from bpemb import BPEmb ----> 3 bpemb_de = BPEmb(lang="de", vs=50000)

~/anaconda3/lib/python3.6/site-packages/bpemb/bpemb.py in init(self, lang, vs, dim, cache_dir, preprocess, encode_extra_options, add_pad_emb, vs_fallback, segmentation_only, model_file, emb_file) 188 else: 189 emb_file = self.emb_tpl.format(lang=lang, vs=vs, dim=dim) --> 190 self.emb_file = self._load_file(emb_file, archive=True) 191 self.emb = load_word2vec_file(self.emb_file, add_pad=add_pad_emb) 192 self.most_similar = self.emb.most_similar

~/anaconda3/lib/python3.6/site-packages/bpemb/bpemb.py in _load_file(self, file, archive, cache_dir) 226 file_url = self.base_url + file + suffix 227 print("downloading", file_url) --> 228 return http_get(file_url, cached_file, ignore_tardir=True) 229 230 def repr(self):

~/anaconda3/lib/python3.6/site-packages/bpemb/util.py in http_get(url, outfile, ignore_tardir) 47 import tarfile 48 tf = tarfile.open(fileobj=temp_file) ---> 49 members = tf.getmembers() 50 if len(members) != 1: 51 raise NotImplementedError("TODO: extract multiple files")

~/anaconda3/lib/python3.6/tarfile.py in getmembers(self) 1759 self._check() 1760 if not self._loaded: # if we want to obtain a list of -> 1761 self._load() # all members, we first have to 1762 # scan the whole archive. 1763 return self.members

~/anaconda3/lib/python3.6/tarfile.py in _load(self) 2356 """ 2357 while True: -> 2358 tarinfo = self.next() 2359 if tarinfo is None: 2360 break

~/anaconda3/lib/python3.6/tarfile.py in next(self) 2287 # Advance the file pointer. 2288 if self.offset != self.fileobj.tell(): -> 2289 self.fileobj.seek(self.offset - 1) 2290 if not self.fileobj.read(1): 2291 raise ReadError("unexpected end of data")

~/anaconda3/lib/python3.6/gzip.py in seek(self, offset, whence) 366 elif self.mode == READ: 367 self._check_not_closed() --> 368 return self._buffer.seek(offset, whence) 369 370 return self.offset

~/anaconda3/lib/python3.6/_compression.py in seek(self, offset, whence) 141 # Read and discard data until we reach the desired position. 142 while offset > 0: --> 143 data = self.read(min(io.DEFAULT_BUFFER_SIZE, offset)) 144 if not data: 145 break

~/anaconda3/lib/python3.6/gzip.py in read(self, size) 480 break 481 if buf == b"": --> 482 raise EOFError("Compressed file ended before the " 483 "end-of-stream marker was reached") 484

EOFError: Compressed file ended before the end-of-stream marker was reached

Kindly, any suggestions to fix this issue !!

Hi,

Now that issue https://github.com/bheinzerling/bpemb/issues/34 is sorted out. For the R wrapper of sentencepiece at https://github.com/bnosac/sentencepiece, I'm planning to implement a simple wrapper to download the models you have been providing and next put the package on CRAN. Before I do this, I would like to know the intention of that site: https://nlp.h-its.org Namely

• Who maintains that site
• Will that site persist over time
• Any objections in redistributing these sentencepiece models and glove embeddings? Either license-wise or other.

Thanks for any input.

hi, I'm writing an R wrapper around sentencepiece and tried to load a few of the models and vocabulary provided here. I have found some inconsistencies. In order to make sure this is not related to the version of SentencePiece you used, can you let me know with which version / commit of SentencePiece the models were constructed? I'm making the R wrapper around sentencepiece release v0.1.84 from Oct 12, 2019.

Hi,

First of all, thanks for the great package. Currently, the only way to use my own models with bpemb is to first load another model, and then assign the .spm and .emb attributes manually. This is a bit unwieldy.

I am interested in adding a subclass of BPEmb that overrides the __init__ of BPEmb and simply accepts paths to an spm and emb model/file, from which the other attributes (e.g. size/vs) are derived. Is this something you would accept as a PR? Do you see any problems with this approach?

Thanks! Stéphan

Hi, This may be a dummy question, but when creating a BPEmb with add_pad_emb=True, how do I actually get the padding embedding and what is its ID ? This should maybe figure somewhere in the doc:

from bpemb import BPEmb
bp = BPEmb(lang="en", vs=1000, dim=50, add_pad_emb=True)
print(bp.vs) # prints 1000, was expecting 1000+1 ?

Thanks for the great work, derlin

Hi there, thanks for providing such wonderful pretrained models, I noticed the pretrained model converted all digits/numbers to 0. May I ask why? Numbers are very useful and sometime crucial for downstream NLP tasks such as question answering (e.g. when asking questions related to numbers).

For the german models, there seem to be tokens missing in the vector models (200k merge operations) that are present in the vocab list and sentencepiece models. For example ▁plaisor auens are in the vocab but fail to look up in the model.

The number of tokens in the model len(model.index2word) returns 185712, while in the vocab list there are 200k entries.

Am I missing something here or is there some miss-match between the training of the sentencepiece tokens / glove vectors?

Great resource otherwise, thanks!

I'm working on a machine translation task. When I encode corpus with bpemb, the output is always lower case. Is it possible to retain case information after encode my corpus?

Thank you for making this resource.

It would be nice if we can use the model trained with --model_type=unigram, which is a default mode of sentencepiece.

BPE and Unigram show basically the same BLUE score, but unigram is more flexible and supports subword sampling. https://arxiv.org/abs/1804.10959