Yet another Python binding for fastText

Overview

pyfasttext

Warning! pyfasttext is no longer maintained: use the official Python binding from the fastText repository: https://github.com/facebookresearch/fastText/tree/master/python

Yet another Python binding for fastText.

The binding supports Python 2.6, 2.7 and Python 3. It requires Cython.

Numpy and cysignals are also dependencies, but are optional.

pyfasttext has been tested successfully on Linux and Mac OS X.
Warning: if you want to compile pyfasttext on Windows, do not compile with the cysignals module because it does not support this platform.

Table of Contents

Installation

To compile pyfasttext, make sure you have the following compiler:

  • GCC (g++) with C++11 support.
  • LLVM (clang++) with (at least) partial C++17 support.

Simplest way to install pyfasttext: use pip

Just type these lines:

pip install cython
pip install pyfasttext

Possible compilation error

If you have a compilation error, you can try to install cysignals manually:

pip install cysignals

Then, retry to install pyfasttext with the already mentioned pip command.

Cloning

pyfasttext uses git submodules.
So, you need to add the --recursive option when you clone the repository.

git clone --recursive https://github.com/vrasneur/pyfasttext.git
cd pyfasttext

Requirements for Python 2.7

Python 2.7 support relies on the future module: pyfasttext needs bytes objects, which are not available natively in Python2.
You can install the future module with pip.

pip install future

Building and installing manually

First, install all the requirements:

pip install -r requirements.txt

Then, build and install with setup.py:

python setup.py install

Building and installing without optional dependencies

pyfasttext can export word vectors as numpy ndarrays, however this feature can be disabled at compile time.

To compile without numpy, pyfasttext has a USE_NUMPY environment variable. Set this variable to 0 (or empty), like this:

USE_NUMPY=0 python setup.py install

If you want to compile without cysignals, likewise, you can set the USE_CYSIGNALS environment variable to 0 (or empty).

Usage

How to load the library?

>>> from pyfasttext import FastText

How to load an existing model?

>>> model = FastText('/path/to/model.bin')

or

>>> model = FastText()
>>> model.load_model('/path/to/model.bin')

Word representation learning

You can use all the options provided by the fastText binary (input, output, epoch, lr, ...).
Just use keyword arguments in the training methods of the FastText object.

Training using Skipgram

>>> model = FastText()
>>> model.skipgram(input='data.txt', output='model', epoch=100, lr=0.7)

Training using CBoW

>>> model = FastText()
>>> model.cbow(input='data.txt', output='model', epoch=100, lr=0.7)

Word vectors

Word vectors access

Vector for a given word

By default, a single word vector is returned as a regular Python array of floats.

>>> model['dog']
array('f', [-1.308749794960022, -1.8326224088668823, ...])
Numpy ndarray

The model.get_numpy_vector(word) method returns the word vector as a numpy ndarray.

>>> model.get_numpy_vector('dog')
array([-1.30874979, -1.83262241, ...], dtype=float32)

If you want a normalized vector (i.e. the vector divided by its norm), there is an optional boolean parameter named normalized.

>>> model.get_numpy_vector('dog', normalized=True)
array([-0.07084749, -0.09920666, ...], dtype=float32)
Words for a given vector

The inverse operation of model[word] or model.get_numpy_vector(word) is model.words_for_vector(vector, k).
It returns a list of the k words closest to the provided vector. The default value for k is 1.

>>> king = model.get_numpy_vector('king')
>>> man = model.get_numpy_vector('man')
>>> woman = model.get_numpy_vector('woman')
>>> model.words_for_vector(king + woman - man, k=1)
[('queen', 0.77121970653533936)]
Get the number of words in the model
>>> model.nwords
500000
Get all the word vectors in a model
>>> for word in model.words:
...   print(word, model[word])
Numpy ndarray

If you want all the word vectors as a big numpy ndarray, you can use the numpy_normalized_vectors member. Note that all these vectors are normalized.

>>> model.nwords
500000
>>> model.numpy_normalized_vectors
array([[-0.07549749, -0.09407753, ...],
       [ 0.00635979, -0.17272158, ...],
       ..., 
       [-0.01009259,  0.14604086, ...],
       [ 0.12467574, -0.0609326 , ...]], dtype=float32)
>>> model.numpy_normalized_vectors.shape
(500000, 100) # (number of words, dimension)

Misc operations with word vectors

Word similarity
>>> model.similarity('dog', 'cat')
0.75596606254577637
Most similar words
>>> model.nearest_neighbors('dog', k=2)
[('dogs', 0.7843924736976624), ('cat', 75596606254577637)]
Analogies

The model.most_similar() method works similarly as the one in gensim.

>>> model.most_similar(positive=['woman', 'king'], negative=['man'], k=1)
[('queen', 0.77121970653533936)]

Text classification

Supervised learning

>>> model = FastText()
>>> model.supervised(input='/path/to/input.txt', output='/path/to/model', epoch=100, lr=0.7)

Get all the labels

>>> model.labels
['LABEL1', 'LABEL2', ...]

Get the number of labels

>>> model.nlabels
100

Prediction

To obtain the k most likely labels from test sentences, there are multiple model.predict_*() methods.
The default value for k is 1. If you want to obtain all the possible labels, use None for k.

Labels and probabilities

If you have a list of strings (or an iterable object), use this:

>>> model.predict_proba(['first sentence\n', 'second sentence\n'], k=2)
[[('LABEL1', 0.99609375), ('LABEL3', 1.953126549381068e-08)], [('LABEL2', 1.0), ('LABEL3', 1.953126549381068e-08)]]

If you want to test a single string, use this:

>>> model.predict_proba_single('first sentence\n', k=2)
[('LABEL1', 0.99609375), ('LABEL3', 1.953126549381068e-08)]

WARNING: In order to get the same probabilities as the fastText binary, you have to add a newline (\n) at the end of each string.

If your test data is stored inside a file, use this:

>>> model.predict_proba_file('/path/to/test.txt', k=2)
[[('LABEL1', 0.99609375), ('LABEL3', 1.953126549381068e-08)], [('LABEL2', 1.0), ('LABEL3', 1.953126549381068e-08)]]
Normalized probabilities

For performance reasons, fastText probabilities often do not sum up to 1.0.

If you want normalized probabilities (where the sum is closer to 1.0 than the original probabilities), you can use the normalized=True parameter in all the methods that output probabilities (model.predict_proba(), model.predict_proba_file() and model.predict_proba_single()).

>>> sum(proba for label, proba in model.predict_proba_single('this is a sentence that needs to be classified\n', k=None))
0.9785203068801335
>>> sum(proba for label, proba in model.predict_proba_single('this is a sentence that needs to be classified\n', k=None, normalized=True))
0.9999999999999898
Labels only

If you have a list of strings (or an iterable object), use this:

>>> model.predict(['first sentence\n', 'second sentence\n'], k=2)
[['LABEL1', 'LABEL3'], ['LABEL2', 'LABEL3']]

If you want to test a single string, use this:

>>> model.predict_single('first sentence\n', k=2)
['LABEL1', 'LABEL3']

WARNING: In order to get the same probabilities as the fastText binary, you have to add a newline (\n) at the end of each string.

If your test data is stored inside a file, use this:

>>> model.predict_file('/path/to/test.txt', k=2)
[['LABEL1', 'LABEL3'], ['LABEL2', 'LABEL3']]

Quantization

Use keyword arguments in the model.quantize() method.

>>> model.quantize(input='/path/to/input.txt', output='/path/to/model')

You can load quantized models using the FastText constructor or the model.load_model() method.

Is a model quantized?

If you want to know if a model has been quantized before, use the model.quantized attribute.

>>> model = FastText('/path/to/model.bin')
>>> model.quantized
False
>>> model = FastText('/path/to/model.ftz')
>>> model.quantized
True

Subwords

fastText can use subwords (i.e. character ngrams) when doing unsupervised or supervised learning.

You can access the subwords, and their associated vectors, using pyfasttext.

Get the subwords

fastText's word embeddings can be augmented with subword-level information. It is possible to retrieve the subwords and their associated vectors from a model using pyfasttext.

To retrieve all the subwords for a given word, use the model.get_all_subwords(word) method.

>>> model.args.get('minn'), model.args.get('maxn')
(2, 4)
>>> model.get_all_subwords('hello') # word + subwords from 2 to 4 characters
['hello', '<h', '<he', '<hel', 'he', 'hel', 'hell', 'el', 'ell', 'ello', 'll', 'llo', 'llo>', 'lo', 'lo>', 'o>']

For fastText, < means "beginning of a word" and > means "end of a word".

As you can see, fastText includes the full word. You can omit it using the omit_word=True keyword argument.

>>> model.get_all_subwords('hello', omit_word=True)
['<h', '<he', '<hel', 'he', 'hel', 'hell', 'el', 'ell', 'ello', 'll', 'llo', 'llo>', 'lo', 'lo>', 'o>']

When a model is quantized, fastText may prune some subwords. If you want to see only the subwords that are really used when computing a word vector, you should use the model.get_subwords(word) method.

>>> model.quantized
True
>>> model.get_subwords('beautiful')
['eau', 'aut', 'ful', 'ul']
>>> model.get_subwords('hello')
['hello'] # fastText will not use any subwords when computing the word vector, only the full word

Get the subword vectors

To get the individual vectors given the subwords, use the model.get_numpy_subword_vectors(word) method.

>>> model.get_numpy_subword_vectors('beautiful') # 4 vectors, so 4 rows
array([[ 0.49022141,  0.13586822,  ..., -0.14065443,  0.89617103], # subword "eau"
       [-0.42594951,  0.06260503,  ..., -0.18182631,  0.34219387], # subword "aut"
       [ 0.49958718,  2.93831301,  ..., -1.97498322, -1.16815805], # subword "ful"
       [-0.4368791 , -1.92924356,  ...,  1.62921488, 1.90240896]], dtype=float32) # subword "ul"

In fastText, the final word vector is the average of these individual vectors.

>>> import numpy as np
>>> vec1 = model.get_numpy_vector('beautiful')
>>> vecs2 = model.get_numpy_subword_vectors('beautiful')
>>> np.allclose(vec1, np.average(vecs2, axis=0))
True

Sentence and text vectors

To compute the vector of a sequence of words (i.e. a sentence), fastText uses two different methods:

  • one for unsupervised models
  • another one for supervised models

When fastText computes a word vector, recall that it uses the average of the following vectors: the word itself and its subwords.

Unsupervised models

For unsupervised models, the representation of a sentence for fastText is the average of the normalized word vectors.

To get the resulting vector as a regular Python array, use the model.get_sentence_vector(line) method.
To get the resulting vector as a numpy ndarray, use the model.get_numpy_sentence_vector(line) method.

>>> vec = model.get_numpy_sentence_vector('beautiful cats')
>>> vec1 = model.get_numpy_vector('beautiful', normalized=True)
>>> vec2 = model.get_numpy_vector('cats', normalized=True)
>>> np.allclose(vec, np.average([vec1, vec2], axis=0)
True

Supervised models

For supervised models, fastText uses the regular word vectors, as well as vectors computed using word ngrams (i.e. shorter sequences of words from the sentence). When computing the average, these vectors are not normalized.

To get the resulting vector as a regular Python array, use the model.get_text_vector(line) method.
To get the resulting vector as a numpy ndarray, use the model.get_numpy_text_vector(line) method.

>>> model.get_numpy_sentence_vector('beautiful cats') # for an unsupervised model
array([-0.20266785,  0.3407566 ,  ...,  0.03044436,  0.39055538], dtype=float32)
>>> model.get_numpy_text_vector('beautiful cats') # for a supervised model
array([-0.20840774,  0.4289546 ,  ..., -0.00457615,  0.52417743], dtype=float32)

Misc utilities

Show the module version

>>> import pyfasttext
>>> pyfasttext.__version__
'0.4.3'

Show fastText version

As there is no version number in fastText, we use the latest fastText commit hash (from HEAD) as a substitute.

>>> import pyfasttext
>>> pyfasttext.__fasttext_version__
'431c9e2a9b5149369cc60fb9f5beba58dcf8ca17'

Show the model (hyper)parameters

>>> model.args
{'bucket': 11000000,
 'cutoff': 0,
 'dim': 100,
 'dsub': 2,
 'epoch': 100,
...
}

Show the model version number

fastText uses a versioning scheme for its generated models. You can retrieve the model version number using the model.version attribute.

version number description
-1 for really old models with no version number
11 first version number added by fastText
12 for models generated after fastText added support for subwords in supervised learning
>>> model.version
12

Extract labels or classes from a dataset

You can use the FastText object to extract labels or classes from a dataset. The label prefix (which is __label__ by default) is set using the label parameter in the constructor.

If you load an existing model, the label prefix will be the one defined in the model.

>>> model = FastText(label='__my_prefix__')
Extract labels

There can be multiple labels per line.

>>> model.extract_labels('/path/to/dataset1.txt')
[['LABEL2', 'LABEL5'], ['LABEL1'], ...]
Extract classes

There can be only one class per line.

>>> model.extract_classes('/path/to/dataset2.txt')
['LABEL3', 'LABEL1', 'LABEL2', ...]

Exceptions

The fastText source code directly calls exit() when something wrong happens (e.g. a model file does not exist, ...).

Instead of exiting, pyfasttext raises a Python exception (RuntimeError).

>>> import pyfasttext
>>> model = pyfasttext.FastText('/path/to/non-existing_model.bin')
Model file cannot be opened for loading!
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/pyfasttext.pyx", line 124, in pyfasttext.FastText.__cinit__ (src/pyfasttext.cpp:1800)
  File "src/pyfasttext.pyx", line 348, in pyfasttext.FastText.load_model (src/pyfasttext.cpp:5947)
RuntimeError: fastext tried to exit: 1

Interruptible operations

pyfasttext uses cysignals to make all the computationally intensive operations (e.g. training) interruptible.

To easily interrupt such an operation, just type Ctrl-C in your Python shell.

>>> model.skipgram(input='/path/to/input.txt', output='/path/to/mymodel')
Read 12M words
Number of words:  60237
Number of labels: 0
... # type Ctrl-C during training
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "src/pyfasttext.pyx", line 680, in pyfasttext.FastText.skipgram (src/pyfasttext.cpp:11125)
  File "src/pyfasttext.pyx", line 674, in pyfasttext.FastText.train (src/pyfasttext.cpp:11009)
  File "src/pyfasttext.pyx", line 668, in pyfasttext.FastText.train (src/pyfasttext.cpp:10926)
  File "src/cysignals/signals.pyx", line 94, in cysignals.signals.sig_raise_exception (build/src/cysignals/signals.c:1328)
KeyboardInterrupt
>>> # you can have your shell back!
Comments
  • The Model just can not be loaded. |

    The Model just can not be loaded. | "RuntimeError: fastext tried to exit: 1"

    The Model just can not be loaded. I've tried every possible approach that I could came up with, nothing worked. Some people discussed about it here: #125, but nothing helpful that I found.

    EMBEDDINGS_MODEL_PATH = '~/fastText/result/fil9.bin'
    self.word_model = FastText(EMBEDDINGS_MODEL_PATH)
    File "src/pyfasttext.pyx", line 137, in pyfasttext.FastText.__cinit__ (src/pyfasttext.cpp:2249)
    File "src/pyfasttext.pyx", line 466, in pyfasttext.FastText.load_model (src/pyfasttext.cpp:7906)
    RuntimeError: fastext tried to exit: 1
    

    Is there any advice someone can give me on this? It would be very helpful.

    Thanks a lot.

    opened by adamliuio 12
  • How does pyfasttext handle sentence inputs?

    How does pyfasttext handle sentence inputs?

    Hi Vincent,

    I had a question about how pyfasttext handles inputs - how do you process sentences, as opposed to individual tokens? For instance, do I need to separately tokenise my input text, or can I provide it without any preprocessing? From my initial attempts it appears that I can, but since you don't mention it your documentation (which is great, by the way), I wasn't sure.

    Thank you for creating this package!

    opened by nsanthanam 11
  • installation failure on Mac OS with gcc 8.2: fatal error: 'random' file not found

    installation failure on Mac OS with gcc 8.2: fatal error: 'random' file not found

    ERROR MESSAGES: src/pyfasttext.cpp:648:10: fatal error: 'random' file not found #include . ^~~~~~~~ . 1 warning and 1 error generated. error: command 'gcc' failed with exit status 1

    Have installed gcc and LLVM... UK-xxxx:vercheng$ gcc --version Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 10.0.0 (clang-1000.10.40.1) Target: x86_64-apple-darwin17.7.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin

    tried this: https://github.com/vrasneur/pyfasttext/issues/24 didn't help wondering if it's due to gcc latest version 8.2 Thanks

    opened by chengwq613 7
  • Error during compilation -

    Error during compilation - "fatal error: 'cstdint' file not found"

    Here's my output:

    ➜  pyfasttext git:(master) python3 setup.py install
    Compiling src/pyfasttext.pyx because it changed.
    [1/1] Cythonizing src/pyfasttext.pyx
    running install
    running build
    running build_ext
    building 'pyfasttext' extension
    creating build
    creating build/temp.macosx-10.7-x86_64-3.6
    creating build/temp.macosx-10.7-x86_64-3.6/src
    creating build/temp.macosx-10.7-x86_64-3.6/src/fastText
    creating build/temp.macosx-10.7-x86_64-3.6/src/fastText/src
    gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -iquote . -include src/custom_exit.h -I. -I/Users/xx/anaconda/include/python3.6m -c src/pyfasttext.cpp -o build/temp.macosx-10.7-x86_64-3.6/src/pyfasttext.o
    In file included from <built-in>:1:
    ./src/custom_exit.h:14:61: error: no member named 'to_string' in namespace 'std'
      throw std::runtime_error("fastext tried to exit: " + std::to_string(status));
                                                           ~~~~~^
    In file included from src/pyfasttext.cpp:488:
    src/fastText/src/vector.h:13:10: fatal error: 'cstdint' file not found
    #include <cstdint>
             ^
    2 errors generated.
    error: command 'gcc' failed with exit status 1
    

    Compiling the "normal" FastText binaries, as well as SaleStock's fastText.py works without any problems. Any thoughts on how to fix this?

    opened by joneidejohnsen 6
  • pip install fails with Cython 0.28

    pip install fails with Cython 0.28

    Cython 0.28 was released yesterday. pip install pyfasttext using this Cython breaks with:

    Collecting pyfasttext
      Downloading pyfasttext-0.4.4.tar.gz (235kB)
        Complete output from command python setup.py egg_info:
        Collecting cysignals
          Downloading cysignals-1.6.9.tar.gz (85kB)
        Building wheels for collected packages: cysignals
          Running setup.py bdist_wheel for cysignals: started
          Running setup.py bdist_wheel for cysignals: finished with status 'done'
          Stored in directory: /root/.cache/pip/wheels/c3/dd/fa/e7a20f8ca22a48bb55b07486dde4e8ed256907192218339b72
        Successfully built cysignals
        Installing collected packages: cysignals
        Successfully installed cysignals-1.6.9
    
        Error compiling Cython file:
        ------------------------------------------------------------
        ...
              ret['label'] = self.label
              return ret
    
            cdef size_t index = 0
            args = get_fasttext_args(self.ft)
            args_map = get_args_map(args)
                                   ^
        ------------------------------------------------------------
    
        src/pyfasttext.pyx:264:28: Cannot assign type 'shared_ptr[const Args]' to 'const shared_ptr[const Args]'
        Compiling src/pyfasttext.pyx because it depends on /usr/local/lib/python2.7/dist-packages/cysignals/signals.pxd.
        [1/1] Cythonizing src/pyfasttext.pyx
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-build-_kbz9r/pyfasttext/setup.py", line 88, in <module>
            'FASTTEXT_VERSION': get_fasttext_commit_hash()}),
          File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 1026, in cythonize
            cythonize_one(*args)
          File "/usr/local/lib/python2.7/dist-packages/Cython/Build/Dependencies.py", line 1146, in cythonize_one
            raise CompileError(None, pyx_file)
        Cython.Compiler.Errors.CompileError: src/pyfasttext.pyx
    
        ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_kbz9r/pyfasttext/
    

    If I do pip install Cython==0.27.3 first I can successfully install pyfasttext

    The above was run using Python 2.7.12 on Ubuntu 16.0.4.3

    opened by flawaetz 3
  • pyfasttext import error

    pyfasttext import error

    Hi I successfully installed pyfasttext in python 3.6 but at import I have this error:

    ImportError Traceback (most recent call last) in () ----> 1 from pyfasttext import FastText

    ImportError: /home/stephane.mbatchou/anaconda3/lib/python3.6/site-packages/pyfasttext.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZTINSt6thread6_StateE

    Thanks

    opened by bananemure 3
  • Build fails with macOS 10.12

    Build fails with macOS 10.12

    With Python 2.7.14 the compilation fails on macOS 10.12. Requirements (cython, cysignals, future and numpy) are satisfied. Compilation fails both using pip and manual conversion.

    C-Compiler is:

    Apple LLVM version 9.0.0 (clang-900.0.38)
    Target: x86_64-apple-darwin16.7.0
    Thread model: posix
    InstalledDir: /Library/Developer/CommandLineTools/usr/bin
    
    src/fasttext_access.cpp:54:48: error: 'find' is a private member of 'fasttext::Dictionary'
    ALLOW_CONST_METHOD_ACCESS(Dictionary, int32_t, find, const std::string&);
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~
    
    opened by MarcoNiemann 3
  • Quantization error

    Quantization error

    I am not able to pass other options such as qnorm while trying to quantize the model

    model.quantize(input=train_path, output = model_out_path, qnorm=True, retrain=True, cutoff=100000

    Gets the below stacktrace

    rc/pyfasttext.pyx in pyfasttext.FastText.quantize (src/pyfasttext.cpp:11347)()
    
    src/pyfasttext.pyx in pyfasttext.FastText.train (src/pyfasttext.cpp:10844)()
    
    RuntimeError: fastext tried to exit: 1
    
    opened by whiletruelearn 2
  • Installs fails with pip10

    Installs fails with pip10

    Summary

    Install currently fails with pip >= 10. This is because this line uses pip's .main API which is no longer present in the package.

    See for for more details: https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html https://github.com/pypa/pip/issues/5191

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-5ikhpkeg/pyfasttext/setup.py", line 18, in <module>
        ret = pip.main(['install', 'cysignals'])
    AttributeError: module 'pip' has no attribute 'main'
    
    opened by DomHudson 2
  • Quantization error: Model file cannot be opened for loading!

    Quantization error: Model file cannot be opened for loading!

    Great library! I have an issue with quantization.

    When I try to run the quantization example:

    from pyfasttext import FastText
    
    model = FastText()
    model.quantize(input='data/input_data.txt', output='artifacts/quantized_model',
                   epoch=40, lr=0.3,
                   dim=80, minn=2,
                   maxn=3,
                   label='__label__',
                   wordNgrams=2)
    
    

    The python process exits with code 139 and the following error message:

    Model file cannot be opened for loading!
    ------------------------------------------------------------------------
    0   signals.cpython-35m-darwin.so       0x0000000110fbcbb8 sigdie + 120
    1   signals.cpython-35m-darwin.so       0x0000000110fbcaef cysigs_signal_handler + 351
    2   libsystem_platform.dylib            0x00007fff5c0d4f5a _sigtramp + 26
    3   libc++abi.dylib                     0x00007fff59efe467 GCC_except_table51 + 119
    4   pyfasttext.cpython-35m-darwin.so    0x00000001100f6519 _ZL39__pyx_pf_10pyfasttext_8FastText_36trainP31__pyx_obj_10pyfasttext_FastTextP7_objectS2_ + 6905
    5   pyfasttext.cpython-35m-darwin.so    0x00000001100f0c8f _ZL39__pyx_pw_10pyfasttext_8FastText_37trainP7_objectS0_S0_ + 111
    6   python                              0x000000010f7de8fe PyCFunction_Call + 62
    7   pyfasttext.cpython-35m-darwin.so    0x00000001100e1f71 _ZL19__Pyx_PyObject_CallP7_objectS0_S0_ + 97
    8   pyfasttext.cpython-35m-darwin.so    0x00000001100f1573 _ZL42__pyx_pw_10pyfasttext_8FastText_45quantizeP7_objectS0_S0_ + 227
    9   python                              0x000000010f7de8fe PyCFunction_Call + 62
    10  python                              0x000000010f85f357 PyEval_EvalFrameEx + 23159
    11  python                              0x000000010f8624ab _PyEval_EvalCodeWithName + 3115
    12  python                              0x000000010f85988c PyEval_EvalCode + 44
    13  python                              0x000000010f88881d PyRun_FileExFlags + 205
    14  python                              0x000000010f887d88 PyRun_SimpleFileExFlags + 280
    15  python                              0x000000010f8a00e6 Py_Main + 2982
    16  python                              0x000000010f78f128 main + 232
    17  libdyld.dylib                       0x00007fff5be53115 start + 1
    18  ???                                 0x0000000000000002 0x0 + 2
    ------------------------------------------------------------------------
    Unhandled SIGSEGV: A segmentation fault occurred.
    This probably occurred because a *compiled* module has a bug
    in it and is not properly wrapped with sig_on(), sig_off().
    Python will now terminate.
    ------------------------------------------------------------------------
    
    opened by saxelsen 2
  • add Trove classifier for POSIX

    add Trove classifier for POSIX

    As of pyfasttext == 0.4.4, the relevant operating systems are not indicated on PyPI, except for the comment:

    Warning: pyfasttext does not currently compile on Windows because the cysignals module does not support this platform.

    in the README.

    (Thanks for mentioning this in the README, it made me aware of cysignals requirements.) Also, it may be worth noting that cysignals appears to work using Cygwin. See also https://github.com/sagemath/cysignals/pull/64.

    Also, it is possible to support platforms where cysignals is unavailable via conditional compilation:

    IF HAVE_CYSIGNALS:
        from cysignals.signals cimport sig_on, sig_off, sig_check
    ELSE:
        # for non-POSIX systems
        noop = lambda: None
        sig_on = noop
        sig_off = noop
        sig_check = noop
    

    at:

    https://github.com/vrasneur/pyfasttext/blob/35bbbf4e80fc65851b37e1bb988d8d396f1b34cb/src/pyfasttext.pyx#L38

    as outlined here. (In this case, a try:...cimport...except ImportError: raises errors when compiling with Cython, so an IF...ELSE is needed).

    opened by johnyf 2
  • pyhdc.cpp:5:10: fatal error: 'cstdint' file not found ,  1 error generated. error: command 'gcc' failed with exit status 1

    pyhdc.cpp:5:10: fatal error: 'cstdint' file not found , 1 error generated. error: command 'gcc' failed with exit status 1

    I am using MACOS High Sierra 10.13.6 i tried to install pyhdc, but I got the following error bb$ sudo python3 setup.py install running install running build running build_ext building 'pyhdc' extension gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/eman/anaconda3/include -arch x86_64 -I/Users/eman/anaconda3/include -arch x86_64 -DHNAME=permutations_8160.h -I/Users/eman/anaconda3/lib/python3.7/site-packages/numpy/core/include -I. -I/Users/eman/anaconda3/include/python3.7m -c pyhdc.cpp -o build/temp.macosx-10.7-x86_64-3.7/pyhdc.o -std=c++11 pyhdc.cpp:5:10: fatal error: 'cstdint' file not found #include ^~~~~~~~~ 1 error generated. error: command 'gcc' failed with exit status 1

    I installed gcc and Std libraries, but it does not work. Any advice?

    opened by emfhasan 0
  • pyhdc.cpp:5:10: fatal error: 'cstdint' file not found.  error generated. error: command 'gcc' failed with exit status 1

    pyhdc.cpp:5:10: fatal error: 'cstdint' file not found. error generated. error: command 'gcc' failed with exit status 1

    I am using MACOS High Sierra 10.13.6 i tried to install pyhdc, but I got the following error eman$ sudo python3 setup.py install running install running build running build_ext building 'pyhdc' extension gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/eman/anaconda3/include -arch x86_64 -I/Users/eman/anaconda3/include -arch x86_64 -DHNAME=permutations_8160.h -I/Users/eman/anaconda3/lib/python3.7/site-packages/numpy/core/include -I. -I/Users/eman/anaconda3/include/python3.7m -c pyhdc.cpp -o build/temp.macosx-10.7-x86_64-3.7/pyhdc.o -std=c++11 pyhdc.cpp:5:10: fatal error: 'cstdint' file not found #include ^~~~~~~~~ 1 error generated. error: command 'gcc' failed with exit status 1

    I installed gcc and Std libraries, but it does not work. Any advice?

    opened by emfhasan 0
  • Failed to install pyfasttext on MacOS 10.14.6 (Mojave) via pip

    Failed to install pyfasttext on MacOS 10.14.6 (Mojave) via pip

    I want to install pyfasttext into my project virtual environment on MacOS 10.14.6 (Mojave) via pip.

    Some dependencies have been installed (like cython and cysignals), but I still got the error as below.

    Tried re-install XCode command line tools, but did not work.


    gcc -DNDEBUG -g -fwrapv -O3 -Wall -iquote . -include src/custom_exit.h -Isrc -I/Users/duan/IDMED/zeta_search/venv/lib/python3.6/site-packages/cysignals -I. -Isrc/variant/include -I/Users/duan/IDMED/zeta_search/venv/include -I/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m -I/Users/duan/IDMED/zeta_search/venv/lib/python3.6/site-packages/numpy/core/include -c src/fasttext_access.cpp -o build/temp.macosx-10.9-x86_64-3.6/src/fasttext_access.o -Wno-sign-compare -std=c++0x src/fasttext_access.cpp:43:1: error: non-type template argument is not a pointer to member constant ALLOW_METHOD_ACCESS(FastText, bool, checkModel, std::istream&); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/private_access.h:55:38: note: expanded from macro 'ALLOW_METHOD_ACCESS' template struct rob<Only_##MEMBER, (RET_TYPE(CLASS::*)(__VA_ARGS__))(&CLASS::MEMBER)> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/fasttext_access.cpp:54:1: error: non-type template argument is not a pointer to member constant ALLOW_CONST_METHOD_ACCESS(Dictionary, int32_t, find, const std::string&); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/private_access.h:59:38: note: expanded from macro 'ALLOW_CONST_METHOD_ACCESS' template struct rob<Only_##MEMBER, (RET_TYPE(CLASS::*)(__VA_ARGS__) const)(&CLASS::MEMBER)> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/fasttext_access.cpp:55:1: error: non-type template argument is not a pointer to member constant ALLOW_CONST_METHOD_ACCESS(Dictionary, void, pushHash, std::vector<int32_t>&, int32_t); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/private_access.h:59:38: note: expanded from macro 'ALLOW_CONST_METHOD_ACCESS' template struct rob<Only_##MEMBER, (RET_TYPE(CLASS::*)(__VA_ARGS__) const)(&CLASS::MEMBER)> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/fasttext_access.cpp:56:1: error: non-type template argument is not a pointer to member constant ALLOW_METHOD_ACCESS(Dictionary, void, initTableDiscard, ); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/private_access.h:55:38: note: expanded from macro 'ALLOW_METHOD_ACCESS' template struct rob<Only_##MEMBER, (RET_TYPE(CLASS::*)(__VA_ARGS__))(&CLASS::MEMBER)> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/fasttext_access.cpp:57:1: error: non-type template argument is not a pointer to member constant ALLOW_METHOD_ACCESS(Dictionary, void, initNgrams, ); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ src/private_access.h:55:38: note: expanded from macro 'ALLOW_METHOD_ACCESS' template struct rob<Only_##MEMBER, (RET_TYPE(CLASS::*)(__VA_ARGS__))(&CLASS::MEMBER)> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 5 errors generated. error: command 'gcc' failed with exit status 1 ---------------------------------------- ERROR: Command errored out with exit status 1: /Users/duan/IDMED/zeta_search/venv/bin/python3.6 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/ls/1mmwn3p11nd2j32fbw3704ww0000gn/T/pip-install-itzgumsa/pyfasttext/setup.py'"'"'; __file__='"'"'/private/var/folders/ls/1mmwn3p11nd2j32fbw3704ww0000gn/T/pip-install-itzgumsa/pyfasttext/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/ls/1mmwn3p11nd2j32fbw3704ww0000gn/T/pip-record-haa1t52z/install-record.txt --single-version-externally-managed --compile --install-headers /Users/duan/IDMED/zeta_search/venv/include/site/python3.6/pyfasttext Check the logs for full command output.

    opened by Kungreye 4
  • Fails install on windows 10 with python 3

    Fails install on windows 10 with python 3

    Hi, I tried to install pyfasttext on windows 10 under python 3.6. I have g++ from mingw, clang LLVM 7.0.1 and put both of them on environment variable. I set USE_CYSIGNALS=0 because I'm on windows. But when I run the command: python setup.py install I got the following error:

    File "setup.py", line 66, in build_extensions
        if 'clang' in self.compiler.compiler[0]:
    AttributeError: 'MSVCCompiler' object has no attribute 'compiler'
    

    Can anyone help me on this, I want to be able to use pyfasttext on windows.

    opened by Vonisoa 0
  • official fastText  example is very stranger, have anybody know how to used.

    official fastText example is very stranger, have anybody know how to used.

    official Python binding from the fastText repository: https://github.com/facebookresearch/fastText/tree/master/python , open this website , only little example , compare pyfasttext document , I cannot understand official document , have anybody know how to understand official fasttext example , In my mind pyfasttext document better than official fasttext document

    opened by chenbaicheng 0
  • bayesopt import error

    bayesopt import error

    I tried to run the example given in the repo. But i met the bayesopt import error. I couldn't find this in PIP too. Can you guide me, what I have missed?

    opened by giriannamalai 2
Releases(0.4.6)
Owner
Vincent Rasneur
Vincent Rasneur
scikit-learn wrappers for Python fastText.

skift scikit-learn wrappers for Python fastText. >>> from skift import FirstColFtClassifier >>> df = pandas.DataFrame([['woof', 0], ['meow', 1]], colu

Shay Palachy 233 Sep 9, 2022
scikit-learn wrappers for Python fastText.

skift scikit-learn wrappers for Python fastText. >>> from skift import FirstColFtClassifier >>> df = pandas.DataFrame([['woof', 0], ['meow', 1]], colu

Shay Palachy 209 Feb 17, 2021
🌸 fastText + Bloom embeddings for compact, full-coverage vectors with spaCy

floret: fastText + Bloom embeddings for compact, full-coverage vectors with spaCy floret is an extended version of fastText that can produce word repr

Explosion 222 Dec 16, 2022
Blazing fast language detection using fastText model

Luga A blazing fast language detection using fastText's language models Luga is a Swahili word for language. fastText provides a blazing fast language

Prayson Wilfred Daniel 18 Dec 20, 2022
Yet Another Sequence Encoder - Encode sequences to vector of vector in python !

Yase Yet Another Sequence Encoder - encode sequences to vector of vectors in python ! Why Yase ? Yase enable you to encode any sequence which can be r

Pierre PACI 12 Aug 19, 2021
Yet Another Compiler Visualizer

yacv: Yet Another Compiler Visualizer yacv is a tool for visualizing various aspects of typical LL(1) and LR parsers. Check out demo on YouTube to see

Ashutosh Sathe 129 Dec 17, 2022
Yet Another Neural Machine Translation Toolkit

YANMTT YANMTT is short for Yet Another Neural Machine Translation Toolkit. For a backstory how I ended up creating this toolkit scroll to the bottom o

Raj Dabre 121 Jan 5, 2023
YACLC - Yet Another Chinese Learner Corpus

汉语学习者文本多维标注数据集YACLC V1.0 中文 | English 汉语学习者文本多维标注数据集(Yet Another Chinese Learner

BLCU-ICALL 47 Dec 15, 2022
pyMorfologik MorfologikpyMorfologik - Python binding for Morfologik.

Python binding for Morfologik Morfologik is Polish morphological analyzer. For more information see http://github.com/morfologik/morfologik-stemming/

Damian Mirecki 18 Dec 29, 2021
Just Another Telegram Ai Chat Bot Written In Python With Pyrogram.

OkaeriChatBot Just another Telegram AI chat bot written in Python using Pyrogram. Requirements Python 3.7 or higher.

Wahyusaputra 2 Dec 23, 2021
This is the Alpha of Nutte language, she is not complete yet / Essa é a Alpha da Nutte language, não está completa ainda

nutte-language This is the Alpha of Nutte language, it is not complete yet / Essa é a Alpha da Nutte language, não está completa ainda My language was

catdochrome 2 Dec 18, 2021
A high-level yet extensible library for fast language model tuning via automatic prompt search

ruPrompts ruPrompts is a high-level yet extensible library for fast language model tuning via automatic prompt search, featuring integration with Hugg

Sber AI 37 Dec 7, 2022
Telegram bot to auto post messages of one channel in another channel as soon as it is posted, without the forwarded tag.

Channel Auto-Post Bot This bot can send all new messages from one channel, directly to another channel (or group, just in case), without the forwarded

Aditya 128 Dec 29, 2022
ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

ANTLR (ANother Tool for Language Recognition) is a powerful parser generator for reading, processing, executing, or translating structured text or binary files.

Antlr Project 13.6k Jan 5, 2023
Python module (C extension and plain python) implementing Aho-Corasick algorithm

pyahocorasick pyahocorasick is a fast and memory efficient library for exact or approximate multi-pattern string search meaning that you can find mult

Wojciech Muła 763 Dec 27, 2022
Python module (C extension and plain python) implementing Aho-Corasick algorithm

pyahocorasick pyahocorasick is a fast and memory efficient library for exact or approximate multi-pattern string search meaning that you can find mult

Wojciech Muła 579 Feb 17, 2021
Python-zhuyin - An open source Python library that provides a unified interface for converting between Chinese pinyin and Zhuyin (bopomofo)

Python-zhuyin - An open source Python library that provides a unified interface for converting between Chinese pinyin and Zhuyin (bopomofo)

null 2 Dec 29, 2022
Ελληνικά νέα (Python script) / Greek News Feed (Python script)

Ελληνικά νέα (Python script) / Greek News Feed (Python script) Ελληνικά English Το 2017 είχα υλοποιήσει ένα Python script για να εμφανίζει τα τωρινά ν

Loren Kociko 1 Jun 14, 2022