This is the code used in the paper "Entity Embeddings of Categorical Variables".

Thanks for sharing your code serving it as a reference to learn from and understand the use case of embedding better.

Our dataset is also tabular and have high cardinality columns (~6 million) like IP Address. Besides that 85% of IP Addresses appear only once in the dataset. What are your thoughts on using this technique to convert such a categorical column into euclidean space?

Dear Entron, I downloaded the kaggle branch and trained with test_models.py file (default option: 1 network, train_ratio = 0.97). But I cannot see the meaningful embeddings as yours. I attached state_embedding with all states in the same distance with each other. Can you tell me why it is ? Thank you very much ! P/S: I used keras 1.2.2, tensorflow r0.10 with GPU. I got "Result on validation data: 0.10472426564821177" and I saved trained model by keras.save() (I could not save by pickle.dump because of error as in issue #9)

I'm sorry I asked such a question,I used keras2.2.4,It was found 'Merge' that it could not be introduced in 'keras.layers.core ',But I found 'merge' in 'keras.layers',Unfortunately, the following error occurred Thank you very much !

Hi entron,

really very awesome and learnt alot from your work. However, i do have a question and i think this is the most appropriate place to ask.

Looking at your script, what is the methodology to decide to embed a variable or dense a variable. For example u chose to make variable like promo as a dense layer ...etc . Is there a reason for those that were chosen as a dense layer?

In case of XGBoost I could find in this repo any functions that would use embeddings. As I understood, entity embeddings are produced by __build_keras_model(), which seems to be used only for deep learning here, even though the paper shows its results in Tables III and IV also for KNN, RF, and XGB.

Helping others to reproduce accuracy gains from the use of embeddings in XGBoost is important because XGBoost baseline accuracy is difficult to improve using any neural network-based method, not just embeddings.

Hi @entron I was wondering if you tried lagged variables? Did they end up just not being that useful?

What was your experience trying to incorporate classic time series features into this dataset such as sales trends etc?

Thanks for your insights.

Hi entron,

I got a problem when I pickle.dump the models.

Traceback (most recent call last): File "train_test_model.py", line 90, in pickle.dump(models, f) _pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup module on builtins failed

Cloud you give me some ideas? very thanks.

Should I use all training data for embedding and then attach the embedded feature to train/test and then train XGB? Or I need to split the training into two set and learn the embedding from on set and then use the other set to train XGB?

Hi entron. This is more like conceptual question. I'm reading your source and docs and I can see that you have categorized many features alone by themselves but you haven't, for example, combined several of them into single embedding. By this I mean to use very similar features by nature in an embedding group whose resulting vector would represent "behaviour" of the group, maybe better than each feature solo categorized. Did you try something like that? If yes, can you share your conclusions? Thank you.

Hi, I'm trying to learn your paper but something wrong with my code. I try to locate the error and I found the file named "embeddings.pickle" was everywhere but I just cannot find how to create it. I noticed the comment "# Use plot_embeddings.ipynb to create" but still I cannot find out the answer. I'm wondering if you can give me a hand, thanks a lot.

Hi, Thanks for your code. I notice that the features for training are as:

return [store_open,
            store_index,
            day_of_week,
            promo,
            year,
            month,
            day,
            store_data[store_index - 1]['State']
            ]

but there are lots of other features like in store.csv:

Assortment | CompetitionDistance | CompetitionOpenSinceMonth | CompetitionOpenSinceYear

Could you share the consideration why you not using these as features? Thx!

Hello,

I have a general rudimentary question ( sorry in advance).

I have reviewed (not fully) many parts of the codes in here. I'd like to test the proposed embedding on a new data, but am not sure where to begin.

I have a simple 2-column data: first col is patient id (assume 1M unique patients) second col is ICD10 diag code (assume 10K categories). We have repeated measurements in data, meaning that diagnoses can be repeated within a given patient and across many patients.

I tested Multiple Correspondance Analysis with categorical data from this link, but the results are not very useful.

Similar to the German States example in the repo, my goal is to perform (unsupervised) dimensionality reduction ( such as the ones you'd see in denoising AE with minimizing reconstruction error).

• Where should I start? Do I need to run one-hot beforehand?
• What funcs should I use after loading my raw data to generate such embedding?

Appreciate any words of wisdom you may be able to share.

There is a problem:

TypeError: init() missing 1 required positional argument: 'target_shape'

and the code is:

model_store.add(Reshape(dims=(50,)))

dims??? I can't understand this paramter.

Thank you very much~