Implement the recommender evaluation techniques: MAE, RMSE, F1-Score, Precision and Recall.

Evaluating collaborative filtering recommender systems References: http://dl.acm.org/citation.cfm?id=963772

Evaluation of recommender systems: new approach

http://ec.iem.cyut.edu.tw/drupal/sites/default/files/Evaluation%20of%20recommender%20systems%20A%20new%20approach.pdf

Create the FileDataModel that will receive as input the *.txt or *.csv or any text file and parse it and store it as internal structure.

Use a sample database to test the FileDataModel.

Work with the DictDataModel Assigned developer to choose the best internal structure to store the user matrix ratings/preferences.

In section 2.1.5 of http://muricoca.github.com/crab/tutorial.html, sample code

>>> from crab.recommenders.knn import UserBasedRecommender

should be

>>> from scikits.crab.recommenders.knn import UserBasedRecommender

Check and implement several cross validation techniques for evaluate the recommenders.

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/cross_validation.py

Currently I get this when trying to install crab:

error in scikits.crab setup command: Distribution contains no modules or packages for namespace package 'scikits'

My installation attempts included

pip install git+git://github.com/muricoca/crab.git

and

pip install -e git+git://github.com/muricoca/crab.git#egg=crab

Add the implementation of Recommender Evaluator which will evaluate the recommender metrics using the cross-validation functions available for the data set.

The current data models lack the representation as strings by using repr.

The goal is when you do print model , it will show a brief representation of the ratings matrix (based on the current ones) such as:

print model MatrixDataModel (3 by 3) red orange green apple 2.000000 --- 1.000000 orange --- 2.000000 --- celery --- --- 1.000000

Based on the paper of Sarwar :

Sarwar, B.; Karypis, G.; Konstan, J.; Riedl, J. (2000), Application of Dimensionality Reduction in Recommender System A Case Study

and

NCREMENTAL SINGULAR VALUE. DECOMPOSITION ALGORITHMS FOR. HIGHLY SCALABLE RECOMMENDER. SYSTEMS (SARWAR ET AL)

As setup.py does not define a file-specific encoding, it's encoding defaults to ASCII (per PEP-236). The descr string in setup.py contains the non-ASCII ligature fl, so executing setup.py results in SyntaxError: Non-ASCII character '\xef' in file .... The attached patch fixes that by replacing the ligature with two separate glyphs.

It is necessary to set the params in the recommend BaseRecommender method by calling self.__set_params(**params) from BaseEstimator.

How to put this in the BaseRecommender without calling super in the Child Class?

If you run python setup.py install you will not get the dataset files copied into your site-packages. This can be fixed updating package_data attribute in the setup.py file.

There is a small typo in scikits/crab/metrics/pairwise.py.

Should read distance rather than distrance.

Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

• When installing from a Colaboratory (Jupyter) Notebook, I had issues generated from the file book_crossing.py. They where related to : the print function (missing parenthesis that are mandatory in Python 3), and the the comma (",") in an Exception statement, that has to be replaced by "as" keyword.

• There was an other error, in the same file, related to a "path" in an import: **from base import Bunch**instead of **from .base import Bunch**

• The errors where corrected and the import from scikits.crab import datasets` worked. But there are still other issues.

from scikits.crab import datasets

movies = datasets.load_sample_movies() songs = datasets.load_sample_songs() from scikits.crab.models import MatrixPreferenceDataModel

Build the model

model = MatrixPreferenceDataModel(movies.data)

from scikits.crab.metrics import pearson_correlation from scikits.crab.similarities import UserSimilarity

Build the similarity

similarity = UserSimilarity(model, pearson_correlation)

from scikits.crab.recommenders.knn import UserBasedRecommender

Build the User based recommender

recommender = UserBasedRecommender(model, similarity, with_preference=True)

Recommend items for the user 5 (Toby)

recommender.recommend(5)

https://github.com/muricoca/crab/blob/beb355538acc419b82beae3c6845d1e1cff5d26b/scikits/crab/metrics/metrics.py#L319-L323

I doubt that the calculation of precision and recall is wrong.

The definition of precision is the proportion of the recommendation list (the y_pred) which is contained in the true list(the y_real).

So, I think the calculation of precision should be precision[i] = (intersection_size / float(len(y_items_pred)))
if len(y_items_pred) else 0.0

and similarly, the calculation of recall should be recall[i] = (intersection_size / float(len(y_real[i])))
if len(y_real[i]) else 0.0

ImportError: No module named crab.metrics.classes

installed crab . but unable import.

from pprint import pprint import csv from scikits.crab.models import MatrixPreferenceDataModel, MatrixBooleanPrefDataModel from scikits.crab.metrics import pearson_correlation, euclidean_distances, jaccard_coefficient, cosine_distances, manhattan_distances, spearman_coefficient from scikits.crab.similarities import ItemSimilarity, UserSimilarity from scikits.crab.recommenders.knn import ItemBasedRecommender, UserBasedRecommender from scikits.crab.recommenders.knn.neighborhood_strategies import NearestNeighborsStrategy from scikits.crab.recommenders.knn.item_strategies import ItemsNeighborhoodStrategy from scikits.crab.recommenders.svd.classes import MatrixFactorBasedRecommender from scikits.crab.metrics.classes import CfEvaluator

""" import random

fieldnames = ['user_id', 'item_id', 'star_rating'] with open('dataset-recsys.csv', "w") as myfile: # writing data to new csv file writer = csv.DictWriter(myfile, delimiter = ',', fieldnames = fieldnames)
writer.writeheader()

for x in range(1, 21):
    items = random.sample(list(range(1, 41)), 20)
    random.randint(1,5)
    for item in items:        
        writer.writerow({'user_id': x, 'item_id': item, 'star_rating': random.randint(1, 5)})

"""

dataset = {} with open('sample_movielens_data.txt') as myfile:
reader = csv.DictReader(myfile, delimiter=',')
i = 0
for line in reader:
i += 1 if (i == 1): continue

    if (int(line['user_id']) not in dataset):
        dataset[int(line['user_id'])] = {}
        
    dataset[int(line['user_id'])][int(line['item_id'])] = float(line['star_rating'])

model = MatrixPreferenceDataModel(dataset)

User-based Similarity

similarity = UserSimilarity(model, cosine_distances) neighborhood = NearestNeighborsStrategy() recsys = UserBasedRecommender(model, similarity, neighborhood)

Item-based Similarity

similarity = ItemSimilarity(model, cosine_distances) nhood_strategy = ItemsNeighborhoodStrategy() recsys = ItemBasedRecommender(model, similarity, nhood_strategy, with_preference=False)

recsys = MatrixFactorBasedRecommender(model=model, items_selection_strategy=nhood_strategy, n_features=10, n_interations=1)

evaluator = CfEvaluator()

rmse = evaluator.evaluate(recsys, 'rmse', permutation=False) mae = evaluator.evaluate(recsys, 'mae', permutation=False) nmae = evaluator.evaluate(recsys, 'nmae', permutation=False) precision = evaluator.evaluate(recsys, 'precision', permutation=False) recall = evaluator.evaluate(recsys, 'recall', permutation=False) f1score = evaluator.evaluate(recsys, 'f1score', permutation=False)

all_scores = evaluator.evaluate(recsys, permutation=False) #all_scores = evaluator.evaluate(boolean_recsys, permutation=False)

result = evaluator.evaluate(recsys, None, permutation=False, at=10, sampling_ratings=0.7)

Cross Validation

result = evaluator.evaluate_on_split(recsys, 'rmse', permutation=False, at=10, cv=5, sampling_ratings=0.7)

pprint (result)