A Python implementation of LightFM, a hybrid recommendation algorithm.

Lyst

Last update: Jan 2, 2023

Related tags

Overview

LightFM

Build status
Linux
OSX (OpenMP disabled)
Windows (OpenMP disabled)

LightFM is a Python implementation of a number of popular recommendation algorithms for both implicit and explicit feedback, including efficient implementation of BPR and WARP ranking losses. It's easy to use, fast (via multithreaded model estimation), and produces high quality results.

It also makes it possible to incorporate both item and user metadata into the traditional matrix factorization algorithms. It represents each user and item as the sum of the latent representations of their features, thus allowing recommendations to generalise to new items (via item features) and to new users (via user features).

For more details, see the Documentation.

Need help? Contact me via email, Twitter, or Gitter.

Installation

Install from pip:

pip install lightfm

or Conda:

conda install -c conda-forge lightfm

Quickstart

Fitting an implicit feedback model on the MovieLens 100k dataset is very easy:

from lightfm import LightFM
from lightfm.datasets import fetch_movielens
from lightfm.evaluation import precision_at_k

# Load the MovieLens 100k dataset. Only five
# star ratings are treated as positive.
data = fetch_movielens(min_rating=5.0)

# Instantiate and train the model
model = LightFM(loss='warp')
model.fit(data['train'], epochs=30, num_threads=2)

# Evaluate the trained model
test_precision = precision_at_k(model, data['test'], k=5).mean()

Articles and tutorials on using LightFM

How to cite

Please cite LightFM if it helps your research. You can use the following BibTeX entry:

@inproceedings{DBLP:conf/recsys/Kula15,
  author    = {Maciej Kula},
  editor    = {Toine Bogers and
               Marijn Koolen},
  title     = {Metadata Embeddings for User and Item Cold-start Recommendations},
  booktitle = {Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender
               Systems co-located with 9th {ACM} Conference on Recommender Systems
               (RecSys 2015), Vienna, Austria, September 16-20, 2015.},
  series    = {{CEUR} Workshop Proceedings},
  volume    = {1448},
  pages     = {14--21},
  publisher = {CEUR-WS.org},
  year      = {2015},
  url       = {http://ceur-ws.org/Vol-1448/paper4.pdf},
}

Development

Pull requests are welcome. To install for development:

Clone the repository: git clone [email protected]:lyst/lightfm.git
Setup a virtual environment: cd lightfm && python3 -m venv venv && source ./venv/bin/activate
Install it for development using pip: pip install -e . && pip install -r test-requirements.txt
You can run tests by running ./venv/bin/py.test tests.
LightFM uses black (version 18.6b4) to enforce code formatting.

When making changes to the .pyx extension files, you'll need to run python setup.py cythonize in order to produce the extension .c files before running pip install -e ..

Comments

LightFM always produces same precision@6 result metric

I'm running parameter optimization on a LightFM implicit factorization model using "warp" loss. When I run np.mean(precision_at_k(...)) on the test data, it is virtually always the same result, 0.316850870848, out to 12 digits. Is this expected for some reason? If not, any ideas how to figure out what might be wrong?

Here is sample output showing the param combinations and the output:

2017-01-20 21:10:02,363 INFO __main__: Starting training iteration 1, params:
{
    "alpha": 0.0041233076328981919,
    "epochs": 45,
    "learning_rate": 0.50174314850490254,
    "no_components": 184
}
2017-01-20 21:10:02,363 INFO lightfm_uma: Training model...
2017-01-20 21:25:20,518 INFO lightfm_uma: Finished precision@6 = 0.316850870848
2017-01-20 21:25:21,453 INFO __main__: Starting training iteration 2, params:
{
    "alpha": 0.0064873564172718886,
    "epochs": 63,
    "learning_rate": 0.50406151543722921,
    "no_components": 180
}
2017-01-20 21:25:21,453 INFO lightfm_uma: Training model...
2017-01-20 21:44:36,565 INFO lightfm_uma: Finished precision@6 = 0.316850870848
2017-01-20 21:44:37,495 INFO __main__: Starting training iteration 3, params:
{
    "alpha": 0.020473212242717205,
    "epochs": 62,
    "learning_rate": 0.74135691825946459,
    "no_components": 156
}
2017-01-20 21:44:37,496 INFO lightfm_uma: Training model...
2017-01-20 22:00:45,661 INFO lightfm_uma: Finished precision@6 = 0.316850870848

opened by scottee 28

How to create mapping for item_features and build item_features if the feature contains multiple values for each item?

Hi,

I was following the example of Building datasets and tried to build the movielens dataset myself. However, when I tries to build the mapping for item and item features, I realized that the feature I choose, which is the genres in movies.csv, has multiple values in one cell (eg. Action|Fantasy|Comedy) instead of just one value. How should I use dataset.fit_partial to create the mapping in these case? And the same goes for building interactions. Thank you so much!

opened by Ivanclj 26
Options for only updating user (or item) parameters while keeping the other fixed?

This would be useful for strong generalization evaluation.

I think this should be doable but probably will need to refactor a lot of Cython code to separate user and item parameters update.

(Trying to rush for a paper submission, otherwise would love to contribute)

opened by dawenl 25
fails to install conda install lightfm
d:\Recommender systems\code>conda install lightfm Fetching package metadata ......... Solving package specifications: . PackageNotFoundError: Package not found: '' Package missing in current win-64 ch annels:

lightfm

You can search for packages on anaconda.org with

anaconda search -t conda lightfm
opened by Sandy4321 23
User / item embeddings Nan with large training set

Apologies if this is user error, but I appear to be getting Nan embeddings from LightFM and I'm unsure what I could have done wrong. I followed the documentation, and have raise the issue on SO.

Basically I have a large data-set where collaborative filtering is working fine, but where user / item embeddings are provided, the model produces nan embeddings.

http://stackoverflow.com/questions/40967226/lightfm-user-item-producing-nan-embeddings

opened by lesterlitch 23
Training questions
Hi, I have two questions,

I try to build a recommender 'online'. If my input data(interaction, user features) has some new actions, and I want to update the model has trained before. Should I retrain a new model? or I can just use method fit_partial like this model.fit_partial(updated_interaction, updated_user_features, item_features) to update my model?

I observed the model training with num_thread=4 on linux server but it seems only use one CPU and the other three use 3% - 5.5% only. I saw that LightFM uses OpenMP but I'm not run it on OSX or Windows. Also I reinstall LightFM and not see any warning like LightFM was compiled without OpenMP support., so OpenMP should be enable? How to run my training to use four CPUs with 100%? Is there something wrong? How can I check it?

Sorry for long description about my questions, thank you!
opened by aldll 17
Segmentation fault (core dump) at the prediction

I'm using the WARP to create my model. To fit it, I've no problem. But for the prediction, I've a segmentation fault (core dump).

How can I solve it ?

Thx

I'm using Ubuntu 16.04 with python 2.7.

opened by phenric 17
Add support for saving and loading models

Is this something you would want in the project @maciejkula ? This is just a rough outline of some code I have laying around to do something like this, so it might need to be fixed up a bit and whatnot, but I'd rather do that after I know your opinion :)

And if it already exists, it's a bit embarrassing but I didn't find it! :)

opened by NegatioN 16
Building datasets
Hello !

Thank you for this open source package, it help a lot and your work is amazing.

I just a have a silly question about dataset construction. I followed the example for my data: user (160.000 x 300) and item (4000 x 4).

dataset = Dataset() dataset.fit(users=(x['id_user'] for x in user), items=(x['id_item'] for x in item), user_features=((x['id_user'], [[x[col] for col in list_columns_user]]) for x in user), item_features=((x['id_item'], [[x[col] for col in list_columns_item]]) for x in item))

But when I try dataset.user_features_shape() I get (160000, 160000). shouldn't I rather have this (160000, 300) ?

Indeed, we can read in the documentation :

Returns ------- (num user ids, num user features): tuple of ints

and my num user features is 300. So there is an error in what I did?

Sorry for the stupid question!
opened by ghost 16
Build features vectors
Hi,

About building users/items features vectors, I took a look at your examples in the repo and also at the post http://blog.ethanrosenthal.com/2016/11/07/implicit-mf-part-2/, but I have a question about it.

I saw that you commonly used "tags" metadata about an item (or user), where you have several tags values for a same item.

But let's take the case where we have, in the original metadata, only one tag value per item, and in total 100 different tags values. It will make a 100-dimensions vector, one-hot-encoded, with only one '1'. Is it still a relevant feature item vector if it is that sparse ?

If I have several kind of tags (for example : item category, color, size type..), is it correct to concatenate my binary feature vectors together ? Like this :

| | tag type A | tag type B | ... | tag type Z | |:------:|:-----------------:|:-----------:|:----:|:-----------:| | item 1 | 0 1 0 0 0 0 0 0 0 | 0 0 0 0 0 1 | ... | 0 0 1 0 0 0 | | item 2 | 0 0 0 0 0 0 1 0 0 | 0 1 0 0 0 0 | ... | 0 0 0 0 1 0 | | ... | ... | ... | ... | ... |

Each 0 / 1 representing a different value encoded for each tag type

Thanks in advance
opened by jwheatp 16

Unintuitive behaviour with cross_validate.random_train_test_split

The random_train_test_split makes is easy to split the interactions matrix into train and test dataset but if you have data with weights you will have to apply random_train_test_split twice with the same random_state parameter. My concern is that it would be intuitive to do something like:

from lightfm.data import Dataset
from lightfm.cross_validation import random_train_test_split

users = np.random.choice([0., 1., 2.], (10, 1))
items = np.random.choice([0., 1., 2.], (10, 1))
weight = np.random.rand(10,1)
data = np.concatenate((users, items, weight), axis=1)

dataset = Dataset()
dataset.fit(users=np.unique(data[:, 0]), items=np.unique(data[:, 1]))
interactions, weight = dataset.build_interactions((i[0], i[1], i[2]) for i in data)

test_percentage = 0.2
random_state = np.random.RandomState(seed=1)

train, test = random_train_test_split(
    interactions=interactions,
    test_percentage=test_percentage,
    random_state=random_state
)
train_weight, test_weight = random_train_test_split(
    interactions=weight,
    test_percentage=test_percentage,
    random_state=random_state
)

np.array_equal(train.row, train_weight.row)
np.array_equal(train.col, train_weight.col)
np.array_equal(test.row, test_weight.row)
np.array_equal(test.col, test_weight.col)

>>> False
>>> False
>>> False
>>> False

This will result in an incorrect split because the state of the random_state changes after the first call to random_state.shuffle. For the above example to work as intended you need to make separate but identical RandomStates:

random_state_interaction = np.random.RandomState(seed=1)
random_state_weight = np.random.RandomState(seed=1)

train, test = random_train_test_split(
    interactions=interactions,
    test_percentage=test_percentage,
    random_state=random_state_interaction
)
train_weight, test_weight = random_train_test_split(
    interactions=weight,
    test_percentage=test_percentage,
    random_state=random_state_weight
)

np.array_equal(train.row, train_weight.row)
np.array_equal(train.col, train_weight.col)
np.array_equal(test.row, test_weight.row)
np.array_equal(test.col, test_weight.col)

>>> True
>>> True
>>> True
>>> True

It works but I think it's a little awkward. Two possible solutions/suggestions:

Only require a seed parameter and create a RandomState in cross_validate._shuffle method. This has the added benefit of fitting in with the larger libraries that only require seed and not a RandomState generator. I also don't see any additional flexibility by passing in a generator instead of a simple integer.
Make a copy of random_state before applying shuffle in cross_validate._shuffle.

Thoughts?

opened by RothNRK 15

Meaning of the recommendation scores

Hello, what is the meaning of the recommendation score for user-item pairs? The ones that are given by the predict() function. I mean, is there some kind of range? How it is calculated?

Thank you!

opened by apadros01 0
lightfm 1.16 installation problem_11.12.2022

Hello! Ran into a problem installing lightfm 1.16 on Linux-5.15.0-56-generic-x86_64-with-glibc2.35/Python 3.10.6. The error log is in the attachment, I tried to solve it using the available methods provided on the Internet, but it did not help. Could you suggest how to overcome this problem? Thanks in advance.

Collecting lightfm Using cached lightfm-1.16.tar.gz (310 kB) Preparing metadata (setup.py) ... done Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from lightfm) (1.23.5) Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.10/dist-packages (from lightfm) (1.9.3) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from lightfm) (2.28.1) Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from lightfm) (1.2.0) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->lightfm) (2022.12.7) Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->lightfm) (1.26.13) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->lightfm) (3.4) Requirement already satisfied: charset-normalizer<3,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->lightfm) (2.1.1) Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->lightfm) (1.2.0) Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->lightfm) (3.1.0) Building wheels for collected packages: lightfm Building wheel for lightfm (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [176 lines of output] /usr/local/lib/python3.10/dist-packages/setuptools/dist.py:770: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead warnings.warn( running bdist_wheel running build running build_py creating build creating build/lib.linux-x86_64-cpython-310 creating build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/cross_validation.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/init.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/lightfm.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/data.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/_lightfm_fast.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/evaluation.py -> build/lib.linux-x86_64-cpython-310/lightfm creating build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/init.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/movielens.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/_common.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/stackexchange.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/_lightfm_fast_no_openmp.c -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/_lightfm_fast_openmp.c -> build/lib.linux-x86_64-cpython-310/lightfm running build_ext building 'lightfm._lightfm_fast_openmp' extension creating build/temp.linux-x86_64-cpython-310 creating build/temp.linux-x86_64-cpython-310/lightfm x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.10 -c lightfm/_lightfm_fast_openmp.c -o build/temp.linux-x86_64-cpython-310/lightfm/_lightfm_fast_openmp.o -ffast-math -march=native -fopenmp lightfm/_lightfm_fast_openmp.c: In function ‘__pyx_tp_dealloc_array’: lightfm/_lightfm_fast_openmp.c:24868:5: error: lvalue required as increment operand 24868 | ++Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c:24870:5: error: lvalue required as decrement operand 24870 | --Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c: In function ‘__pyx_tp_dealloc_memoryview’: lightfm/_lightfm_fast_openmp.c:25179:5: error: lvalue required as increment operand 25179 | ++Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c:25181:5: error: lvalue required as decrement operand 25181 | --Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c: In function ‘__pyx_tp_dealloc__memoryviewslice’: lightfm/_lightfm_fast_openmp.c:25429:5: error: lvalue required as increment operand 25429 | ++Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c:25431:5: error: lvalue required as decrement operand 25431 | --Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c: In function ‘__Pyx_InitGlobals’: lightfm/_lightfm_fast_openmp.c:26184:1: warning: ‘PyEval_InitThreads’ is deprecated [-Wdeprecated-declarations] 26184 | PyEval_InitThreads(); | ^~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.10/Python.h:130, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/ceval.h:122:37: note: declared here 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void); | ^~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c: In function ‘__Pyx_ParseOptionalKeywords’: lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c: In function ‘__Pyx_decode_c_string’: lightfm/_lightfm_fast_openmp.c:28218:9: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations] 28218 | return PyUnicode_FromUnicode(NULL, 0); | ^~~~~~ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here 551 | Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject) PyUnicode_FromUnicode( | ^~~~~~~~~~~~~~~~~~~~~ error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for lightfm Running setup.py clean for lightfm Failed to build lightfm Installing collected packages: lightfm Running setup.py install for lightfm ... error error: subprocess-exited-with-error

× Running setup.py install for lightfm did not run successfully. │ exit code: 1 ╰─> [178 lines of output] /usr/local/lib/python3.10/dist-packages/setuptools/dist.py:770: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead warnings.warn( running install /usr/local/lib/python3.10/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( running build running build_py creating build creating build/lib.linux-x86_64-cpython-310 creating build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/cross_validation.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/init.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/lightfm.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/data.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/_lightfm_fast.py -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/evaluation.py -> build/lib.linux-x86_64-cpython-310/lightfm creating build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/init.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/movielens.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/_common.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/datasets/stackexchange.py -> build/lib.linux-x86_64-cpython-310/lightfm/datasets copying lightfm/_lightfm_fast_no_openmp.c -> build/lib.linux-x86_64-cpython-310/lightfm copying lightfm/_lightfm_fast_openmp.c -> build/lib.linux-x86_64-cpython-310/lightfm running build_ext building 'lightfm._lightfm_fast_openmp' extension creating build/temp.linux-x86_64-cpython-310 creating build/temp.linux-x86_64-cpython-310/lightfm x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.10 -c lightfm/_lightfm_fast_openmp.c -o build/temp.linux-x86_64-cpython-310/lightfm/_lightfm_fast_openmp.o -ffast-math -march=native -fopenmp lightfm/_lightfm_fast_openmp.c: In function ‘__pyx_tp_dealloc_array’: lightfm/_lightfm_fast_openmp.c:24868:5: error: lvalue required as increment operand 24868 | ++Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c:24870:5: error: lvalue required as decrement operand 24870 | --Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c: In function ‘__pyx_tp_dealloc_memoryview’: lightfm/_lightfm_fast_openmp.c:25179:5: error: lvalue required as increment operand 25179 | ++Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c:25181:5: error: lvalue required as decrement operand 25181 | --Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c: In function ‘__pyx_tp_dealloc__memoryviewslice’: lightfm/_lightfm_fast_openmp.c:25429:5: error: lvalue required as increment operand 25429 | ++Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c:25431:5: error: lvalue required as decrement operand 25431 | --Py_REFCNT(o); | ^~ lightfm/_lightfm_fast_openmp.c: In function ‘__Pyx_InitGlobals’: lightfm/_lightfm_fast_openmp.c:26184:1: warning: ‘PyEval_InitThreads’ is deprecated [-Wdeprecated-declarations] 26184 | PyEval_InitThreads(); | ^~~~~~~~~~~~~~~~~~ In file included from /usr/include/python3.10/Python.h:130, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/ceval.h:122:37: note: declared here 122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void); | ^~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c: In function ‘__Pyx_ParseOptionalKeywords’: lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:26990:21: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 26990 | (PyUnicode_GET_SIZE(**name) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject *op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘PyUnicode_AsUnicode’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:580:45: note: declared here 580 | Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE *) PyUnicode_AsUnicode( | ^~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c:27006:25: warning: ‘_PyUnicode_get_wstr_length’ is deprecated [-Wdeprecated-declarations] 27006 | (PyUnicode_GET_SIZE(**argname) != PyUnicode_GET_SIZE(key)) ? 1 : | ^ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:446:26: note: declared here 446 | static inline Py_ssize_t _PyUnicode_get_wstr_length(PyObject op) { | ^~~~~~~~~~~~~~~~~~~~~~~~~~ lightfm/_lightfm_fast_openmp.c: In function ‘__Pyx_decode_c_string’: lightfm/_lightfm_fast_openmp.c:28218:9: warning: ‘PyUnicode_FromUnicode’ is deprecated [-Wdeprecated-declarations] 28218 | return PyUnicode_FromUnicode(NULL, 0); | ^~~~~~ In file included from /usr/include/python3.10/unicodeobject.h:1046, from /usr/include/python3.10/Python.h:83, from lightfm/_lightfm_fast_openmp.c:20: /usr/include/python3.10/cpython/unicodeobject.h:551:42: note: declared here 551 | Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject) PyUnicode_FromUnicode( | ^~~~~~~~~~~~~~~~~~~~~ error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1 [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: legacy-install-failure

× Encountered error while trying to install package. ╰─> lightfm

note: This is an issue with the package mentioned above, not pip. hint: See above for output from the failure.

opened by Abricovich 1
How to validate the Dataset.mapping
Using the same example from https://making.lyst.com/lightfm/docs/examples/dataset.html#building-the-id-mappings

How could I validade the mapping of each item feature?

When I call user_id_map, u_f_map, item_id_map, i_f_map = dataset.mapping()

after this item_features = dataset.build_item_features(((x['ISBN'], [x['Book-Author']]) for x in get_book_features()))

I got this result from item_id_map

{'034545104X': 0, '0155061224': 1, '0446520802': 2, '052165615X': 3...}

Now, I know that '034545104X' is mapped as 0. Looking at item_features built before I got this

(0, 0) 0.5 (0, 343789) 0.5 (1, 1) 0.5 (1, 428522) 0.5 (2, 2) 0.5 (2, 341954) 0.5 (3, 3) 0.5 .......

Looking at book_features as a pandas dataframe

As seen, getting the item mapped as 0 which is the item 034545104X. And filtering the dataframe, for this item, the book-author is Flesh Tones: A Novel

But When I do this

item_feature_inverse_map = {v:k for k, v in i_f_map.items()} print(item_feature_inverse_map[343789])

The result is 'M. J. Rose' which is different of Flesh Tones: A Novel.
opened by Marcoslsl 0
How to deploy model using node js

I created a recommendation system using lightFM on Google Colab. I want to deploy this model using Node JS. I read this article and tried the method recommended here. It recommended using tensorflow.js. For that the model needed to be saved and this is where the problem arose since I wrote my model using lightFM.

This gave rise to following error: ` AttributeError Traceback (most recent call last) in ----> 1 model.save("model.h5")

AttributeError: 'LightFM' object has no attribute 'save' `

What is the alternative to model.save in lightfm? How can we deploy model written in lightfm using node js?

opened by s-bh 0
Using LightFM for Retail Customer Recommendation based on purchased products

Hi, I am looking at using LightFM for recommending products to customers in retail stores based on the products the customer has bought in past. In the customer buying history, I have data regarding the products along with the purchase quantities for each product the customer has bought. The customer might have bought products with quantities ranging from 1 to 100 or more. I am looking for suggestions on how I can convert these product-wise purchased quantities to the rating score of 1 to 5. Looking for your suggestion. Thank you.

opened by gundewardv 0

Add new user/item ids or features

Hello,

Thanks for this nice work, very efficient and designed for real wrold use cases.

Concerning the cold-start issue, you indicate in the documentation to call fit_partial method of the lightfm.data.Dataset class, and to "resize your LightFM model to be able to use the new features". What does "resize your LightFM model to be able to use the new features" really means ?

First I train the model

from lightfm import LightFM
from lightfm.data import Dataset
from lightfm.evaluation import auc_score, precision_at_k, recall_at_k, reciprocal_rank

dataset = Dataset(user_identity_features=False, item_identity_features=True)
dataset.fit(users=train_users_df.index.unique(), 
            items=train_items_df.index, 
            item_features=train_tag_labels)

train_item_features = dataset.build_item_features(train_item_features_)
train_interactions, train_weights = dataset.build_interactions(train_users_df["MatchId"].items())

recommender = LightFM(loss='warp')

recommender = recommender.fit(interactions=train_interactions,
                              item_features=train_item_features,
                              sample_weight=train_weights,
                              epochs=NUM_EPOCHS,
                              num_threads=NUM_THREADS)

After, I would like to predict using this model but for new items, with new features, unseen during this 1st fit. I fit again partially my Dataset without issue ...

dataset.fit_partial(users=test_users_df.index.unique(), 
                    items=test_items_df.index, 
                    item_features=test_tag_labels)

test_item_features_ = build_item_features(test_items_df)
test_item_features = dataset.build_item_features(test_item_features_)

... but I get an error when predicting with the model

recommender.predict_rank(test_interactions=next_items,
                         train_interactions=past_items,
                         item_features=test_item_features)

and I get the following error: "ValueError: The item feature matrix specifies more features than there are estimated feature embeddings: 3623 vs 4985" where 3623 is the number of item features saw during the fit and 4985 is the number of features after adding new items with new features.

Then, is there a way to "resize the model" as suggested in the documentation ?

Thanks

opened by PaulSteffen-betclic 1

Releases(1.16)

1.16(Nov 27, 2020)
Addded

Set the LIGHTFM_NO_CFLAGS environment variable when building LightFM to prevent it from setting -ffast-math or -march=native compiler flags.

Changed

predict now returns float32 predictions.

Source code(tar.gz)
Source code(zip)
1.15(May 26, 2018)
Added

Added a check that there is no overlap between test and train in predict_ranks (thanks to @artdgn).

Added dataset builder functionality.

Fixed

Fixed error message when item features have the wrong dimensions.

Predict now checks for overflow in inputs to predict.

WARP fitting is now numerically stable when there are very few items to draw negative samples from (< max_sampled).

Source code(tar.gz)
Source code(zip)
1.14(Nov 18, 2017)
Added

added additional input checks for non-normal inputs (NaNs, infinites) for features

added additional input checks for non-normal inputs (NaNs, infinites) for interactions

cross validation module with dataset splitting utilities

Changed

LightFM model now raises a ValueError (instead of assertion) when the number of supplied features exceeds the number of estimated feature embeddings.

Warn and delete downloaded file when Movielens download is corrputed. This happens in the wild cofuses users terribly.

Source code(tar.gz)
Source code(zip)
1.13(May 20, 2017)
Added

added get_{user/item}_representations functions to facilitate extracting the latent representations out of the model.

Fixed

recall_at_k and precision_at_k now work correctly at k=1 (thanks to Zank Bennett).

Moved Movielens data to data release to prevent grouplens server flakiness from affecting users.

Fix segfault when trying to predict from a model that has not been fitted.

Source code(tar.gz)
Source code(zip)
1.12(Jan 27, 2017)
Changed

Ranks are now computed pessimistically: when two items are tied, the positive item is assumed to have higher rank. This will lead to zero precision scores for models that predict all zeros, for example.

The model will raise a ValueError if, during fitting, any of the parameters become non-finite (NaN or +/- infinity).

Added mid-epoch regularization when a lot of regularization is used. This reduces the likelihood of numerical instability at high regularization rates.

Source code(tar.gz)
Source code(zip)
1.11(Dec 26, 2016)

Source code(tar.gz)
Source code(zip)
1.10(Nov 25, 2016)

Source code(tar.gz)
Source code(zip)
1.9(May 25, 2016)

Source code(tar.gz)
Source code(zip)