In this file

(https://github.com/facebookresearch/LASER/blob/master/tasks/CCMatrix/dl_cc_matrix.py) line 9 (from cc_net.process_wet_file import CCSegmentsReader)

The code raises an error.

It seems there is no CCSegmentsReader to import. (https://github.com/facebookresearch/cc_net/blob/master/cc_net/process_wet_file.py)

Am I doing something wrong?

This problems happens when running multiple concurrent calls to the process of calculating embeddings in TokenLine

def TokenLine(line, lang='en', lower_case=True, romanize=False):
    assert lower_case, 'lower case is needed by all the models'
    roman = lang if romanize else 'none'
    tok = check_output(
            REM_NON_PRINT_CHAR
            + '|' + NORM_PUNC + lang
            + '|' + DESCAPE
            + '|' + MOSES_TOKENIZER + lang
            + ('| python3.6 -m jieba -d ' if lang == 'zh' else '')
            + ('|' + MECAB + '/bin/mecab -O wakati -b 50000 ' if lang == 'ja' else '')
            + '|' + ROMAN_LC + roman,
            input=line,
            encoding='UTF-8',
            shell=True)
    return tok.strip()

Here is the detailed strack trace:

  File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
INFO:default:called LaserEmbeddingHandler...
ERROR:default:[Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/tornado_api/handlers/embeddingLaserHandler.py", line 195, in post
    embeddings = vector_embedding.embedding_line(model=model_laser,lang=lang,bpe_codes=QUALITY_MODEL_PATH + "/93langs.fcodes",input_text=text)
  File "/tornado_api/deeplearning/vector_embedding.py", line 50, in embedding_line
    embeddings.append(t.result()[0].tolist() )
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/tornado_api/deeplearning/similarity_search.py", line 99, in pipeline
    lower_case=lower_case)
  File "/tornado_api/deeplearning/lib/text_processing.py", line 62, in TokenLine
    shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
INFO:default:called LaserEmbeddingHandler...
ERROR:default:[Errno 12] Cannot allocate memory
Traceback (most recent call last):
  File "/tornado_api/handlers/embeddingLaserHandler.py", line 195, in post
    embeddings = vector_embedding.embedding_line(model=model_laser,lang=lang,bpe_codes=QUALITY_MODEL_PATH + "/93langs.fcodes",input_text=text)
  File "/tornado_api/deeplearning/vector_embedding.py", line 50, in embedding_line
    embeddings.append(t.result()[0].tolist() )
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/tornado_api/deeplearning/similarity_search.py", line 99, in pipeline
    lower_case=lower_case)
  File "/tornado_api/deeplearning/lib/text_processing.py", line 62, in TokenLine
    shell=True)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1275, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

Hi, I am trying to use LASER embedding in order to calculate similarity between sentences. I tried with inner product and results are not bad, but maybe others are better. What similarity would you recommend? Thanks

The "pip install" failed but the external tools installer said it was better to build from source. Unfortunately, I haven't been able to find clear documentation for how to do this.

Any help would be appreciated.

While trying to download the corpora, we are receiving 429 errors:

wget https://dl.fbaipublicfiles.com/laser/WikiMatrix/v1/WikiPedia.wuu-zh.tsv
--2019-07-15 20:47:56--  https://dl.fbaipublicfiles.com/laser/WikiMatrix/v1/WikiPedia.wuu-zh.tsv
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 104.20.22.166, 104.20.6.166, 2606:4700:10::6814:6a6, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|104.20.22.166|:443... connected.
HTTP request sent, awaiting response... 429 Too Many Requests
2019-07-15 20:47:56 ERROR 429: Too Many Requests.

Hi, I am trying to get sentence embeddings on Japanese via the docker. However, the output is empty since Mecab is not installed.

Output in python: {'content': JAPANESE_SENTENCE, 'embedding': []}

Output in shell:

 - Encoder: loading /app/LASER/models/bilstm.93langs.2018-12-26.pt
 - Tokenizer: content.txt in language ja
/bin/sh: 1: /app/LASER/tools-external/mecab/bin/mecab: not found
WARNING: No known abbreviations for language 'ja', attempting fall-back to English version...
 - fast BPE: processing tok
 - Encoder: bpe to out.raw
 - Encoder: 0 sentences in 0s

Any idea on how to deal with that? :)

hi,

i noticed that "FastBPE, fast C++ implementation of byte-pair encoding (installed automatically)"

but it seems that there is no fastbpe,

and then i tried to install it manually, but it said that : g++ -std=c++11 -pthread -O3 fastBPE/main.cc -IfastBPE -o fast clang: error: no such file or directory: 'fastBPE/main.cc' clang: error: no input files

could anyone know how to solve this problem ? thanks !

I have downloaded the "v1.0.beta" version of the data and get all the language pairs and I got these numbers:

I don't know why the numbers aren't the same? Is this means that the "v1.0" will include the missing data?

Also when will "v1.0" be available?

Also regardless the missing languages like English, Japanese and Chinese The other language sentences don't fully aligned for example in "ar-fr" we have:

• 3,195,877 Sentences in fr
• 3,134,825 Sentences in ar
• Only 680,546 Sentences are aligned Which is 20% only of the data size Why not the full data aligned? Is this will be solved in "v1.0"?

@gwenzek

Hi

I've downloaded files listed in https://dl.fbaipublicfiles.com/laser/CCMatrix/v1.0.beta/list.txt and extracted the entries in the language information column (e.g., cs-ko/ko) in there.

Then, I've found that there is no English sentence (i.e., "bla-en/en" or "en-bla/en").

Furthermore, some popular language pairs like es-pt (Spanish and Portuguese) are missing.

Is there something wrong or these pairs are not provided intentionally?

I use LASER to achieve this with the combination of cosine similarity and it works fine for near or exact translations, but I get medium percentages (40-50%) between irrelevant translations. I use SentenceEncoder, TokenLine to tokenize each sentence, BPEfastApplyLine, I am calculating embeddings and then cosine similarity.

I'm currently using LASOR to embed my documents. My command is:

./embed.sh ~/source/samsung/vecalign/bleualign_data/overlaps.vi ~/source/samsung/vecalign/bleualign_data/overlaps.vi.emb [vi] 

But I got the error. .

Can anyone face the problem? How can I fix that?

The origin implementation does not accurately get the number of tokens in batches, since it does not take into account of the padded tokens in token count, which is fixed in the PR.

Please add a requirements.txt with compatible third-party versions or some means to create conda evnironments. Installing the latest versions breaks the bucc.sh script and it's tedious to debug and figure out the mismatches of installed libraries.

I have run these commands:

pip install torch==1.0
pip install faiss faiss-gpu
pip install scipy numpy
pip install cython==0.29.6
pip install fairseq==0.12.1
pip install tabulate
pip install pandas
pip install jieba
pip install transliterate==1.10.2
pip install tensorboardX

but I still get an Attribute error when running

from faiss._swigfaiss import delete_FloatVector

Hi, At the end of section 4.3 it says there is a special procedure for high resource languages where only fwd scores are calculated.

in the code if we set "fwd" instead of "max" it is supposed to calculate only the fwd stuff BUT when scoring it need both the x2y_mean AND the y2x_mean, the later is ot being calculated.

So what should it be in the scoring formula ?

Many thanks.

I tried to mine from two wmt news files (en, de) - generated the embeddings with bilstm.93langs.2018-12-26.pt - then mine_bitexts.py with defaults settings ("mine", "ratio") but the results is very bad even though the margin seems good: top lines: any clue ?

1.6953904090950678 Affordable and flexible fee payment structure. Inzwischen sieht das schon ganz anders aus. 1.6196460840402447 Having toured as Beyoncé"s bassist and assistant musical director, she also has her own career as a soloist and songwriter, not to mention a new, much younger audience. Er schreibt: "Mit unglaublicher Trauer muss ich die Nachricht über den Tod meiner wunderschönen Tochter Maia teilen. 1.6135828150470815 As their popularity continues to increase, we will probably end up more focused on what happens after our favourite reality shows than what happens on them. Der Kreativität seien mithin keine Grenzen gesetzt, betont er. 1.566564394910487 With steroid medication making little difference, Karina realised that she was suffering from topical steroid addiction (TSA) and topical steroid withdrawal (TSW), when the skin reacts adversely after long-term use of topical steroids is stopped, Timo Werner kam hingegen nur auf sechs Tore und wurde mit seiner Chancenverschwendung in England phasenweise zur Witzfigur. 1.555322152863677 "I would like to thank the management and backroom team for their unwavering support and commitment and the clubs, supporters, Club Iarmhi and Westmeath County Board. SN/www.picturedesk.com Vizekanzler Werner Kogler und Klubchefin Sigrid Maurer. 1.5525301272930254 The next two corners of finance to feel the invasion of quants will be the corporate bond market - where systematic strategies are now beginning to spread - and private equity, Rattray predicts. Ismaning - Die Dramatik mit den Ismaninger Abschlussversuchen zeichnete sich in der ersten Hälfte noch nicht ab, denn die war nur wenig aufregend. Unter dem Strich hatten die Dachauer etwas mehr vom Spiel, Ismaning lauerte auf Konter und de facto neutralisierten sich alle. Der FCI präsentierte sich im defensiven Verhalten deutlich besser als zuletzt gegen Hallbergmoos und ließ nur Schüsse zu, die Torwart Radic sicher hatte. 1.5299819314567646 He's not a typical freshman, though. Das teilten Polizei und Staatsanwaltschaft am Sonntag gemeinsam mit. 1.5084004481504976 Move over hard seltzer, a new beverage is poised to become the drink of summer. Die gesellschaftliche Spaltung Israels, so viel steht fest, kann mit dieser Wahl nicht überwunden werden. 1.5046071137197825 The store is advising customers to visit its website in the first instance, before making a trip to store, where there is also an opportunity to utilise its click and collect service. If you choose this option, the team will have your items ready for you to collect within one hour. Extreme Änderungen wolle er aber nicht vornehmen: "Jetzt noch mal alles über den Haufen zu schmeißen, wäre auch nicht zielführend". Freilich werde er weiterhin jungen Spielern "eine Plattform geben, sich zu zeigen. 1.4632964789078255 We are psychologists, data scientists and HR consultants who screen, select, develop, and engage talent worldwide. Wem in der Pandemie etwas langweilig geworden und das Rasenmähen oder andere laute Gartenarbeiten eine willkommene Abwechslung ist, sollte nicht übermütig an die Sache herangehen. Denn bei Geräten wie einem motorbetriebenen Rasenmäher müssen die Ruhezeiten eingehalten sowie die Sonntags- und Feiertagsruhe respektiert werden. Haben die Nachbarn es sich nach dem Mittagessen gerade draußen bequem gemacht, sollte man besser noch etwas warten, bevor man den Rasenmäher aufheulen lässt.

For my university FYP project related to text simplification, there's a requirement for me to generate LASER embeddings for a large number of sentences. (15.7 million) However when I try to generate LASER embeddings using the SentenceEncoder in the embed.py, the program stays fully utilized for around 12 hours and then exits without any error (I assume it is because of the high CPU and GPU utilization). I'm using the SentenceEncoder in the following way.

Initialize the SentenceEncoder with the following params. I'm using the pretrained encoder (models/bilstm.93langs.2018-12-26.pt )

SentenceEncoder(encoder_path, max_tokens=3000, cpu=False, verbose=True)

And then generate LASER embeddings as follows.

embeddings = encoder.encode_sentences(read_lines(bpe_filepath))

I tried to execute the setup with above params in a GCP compute engine with 16 cores with 102 GB memory and 1 Nvidia Tesla T4 GPU. The CPU utilization reaches 100% while the GPU utilization is somewhere around 90%. It stays like that for around 12 hours and exits without any error. (no error in nohup.out).

Any idea about what could go wrong ? I'm stucked at this point for several weeks and really appreciate if someone can help me.

cc @hoschwenk