LOFO (Leave One Feature Out) Importance calculates the importances of a set of features based on a metric of choice,

I don't know how to feature request XD.

It would be great if you can add the categorical_feature parameter in your Dataset just like in the lightgbm docs. Thanks!!

Median and IQR could be more robust and useful if distribution of importances is not normal.

Something like this

importance_df["importance_md"] = lofo_cv_scores_normalized.median(axis=1)
importance_df["importance_iqr"] = stats.iqr(lofo_cv_scores_normalized, axis=1)

Also for plot_importance there could be a choice between error and 95%CI;

For std it would be

importance_df.plot(x="feature", 
y="importance_mean", 
xerr=1.96 * importance_df.importance_std,
kind='barh', 
color=importance_df["color"], 
figsize=figsize)

and for iqr

importance_df.plot(x="feature", 
y="importance_md", 
xerr=1.57 * importance_df.importance_iqr / np.sqrt(n), # num_sampling for flofo and num of folds for lofo
kind='barh', 
color=importance_df["color"], 
figsize=figsize)

from lofo import Dataset doesn't work and gives error of

cannot import name 'Dataset' from 'lofo'

But if I write from lofo.dataset import Dataset then it works fine.

I think it has something to do with init.py

Hi, Consider the following code:

import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import KFold
from lofo import LOFOImportance

brca = load_breast_cancer()
df = pd.DataFrame(brca["data"], columns=brca["feature_names"]).assign(target=brca["target"])

lofo = LOFOImportance(df, brca["feature_names"], "target", 
                      cv=KFold(n_splits=5, random_state=4), 
                      scoring="roc_auc")
importance_df = lofo.get_importance()

image1

This produces a importance ranking of features. There are some positive means and some negative means in the dataset.

I will now select all features, whose mean is above zero and calculate the importance again.

new_features = importance_df.query("importance_mean > 0")["feature"].tolist()
lofo = LOFOImportance(df, new_features, "target", 
                      cv=KFold(n_splits=5, random_state=4), 
                      scoring="roc_auc")
importance_df2 = lofo.get_importance()

image2

Again, there are some positive means and some negative means in the dataset. This is not what I was expecting. I removed the supposedly unnecessary features after the first step, why are there new unnecessary features after the second step?

Hi,

I have 1673 features. When I tried using LOFO importance, the result is the following:

Are the features showing up one on top of the other because the plot isn't long enough? What would you suggest to fix this problem?

Thank you

The code below is okay to get importance_df.

import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_breast_cancer, load_iris
from sklearn.model_selection import KFold
from lofo import LOFOImportance, Dataset, plot_importance

data = load_breast_cancer(as_frame=True)# load as dataframe
df = data.data
df['target']=data.target.values

# model
model = RandomForestClassifier()
# dataset
dataset = Dataset(df=df, target="target", features=[col for col in df.columns if col != 'target'])
# get feature importance
cv = KFold(n_splits=5, shuffle=True, random_state=666)
lofo_imp = LOFOImportance(dataset, cv=cv, scoring="f1",model=model)
importance_df = lofo_imp.get_importance()
print(importance_df)

But if we modify load_breast_cancer to load_iris, the importance_df values are all NaN.

Is the lofo-importance only support binary classification?

This is not an issue, but rather a quick question for clarification.

From the brief definition of the method, it is a little hard to tell how LOFO and RFE/Backward Selection differ from each other. Could you please compare & contrast?

Thank you again for sharing this lib with the community!

Serdar

Is there any way to pass a sample weight column into a Dataset object, or is there any plan to add this functionality? It should fit pretty smoothly into the sklearn framework I would expect, but maybe I'm missing something.

Thanks for the wonderful package!

When there are many features, the task takes a long time and is easy to collapse. Therefore, I think it is necessary to add the logging function and breakpoint recovery function in lofo.

Thanks for putting this repo together. Who developed this method? Is it the same as LOCO? https://www.tandfonline.com/doi/full/10.1080/01621459.2017.1307116?casa_token=HAl_ErrKi18AAAAA:YyDJybfbzaLMDU1Zlzq8D4OQnZmUeEwukWJFagcsFB7_JA-W-7ifcINhc8N0FTbtbImLjezWESo

Thanks, Daniel

Thanks for the very cool package. I was wondering if it is possible to pass a group_id somehow in order to do Groupkfold or Groupshufflesplit cross validation? I don't see anywhere obvious, but wanted to check in case I'm missing anything.

Thanks.

Hi,

For neural network if you change the number of features, you need to change the input dimension and therefore the number of neurons. So, we could have an option like:

• leaving='drop' for current behaviour
• leaving='replace' for NN

What do you think?

Indeed, in the first example here, the target attribute does not exist and should be instead "HasDetections" or defined before creating the model

I have created pull request here

The algorithm doesn't support multiclass classification. In infer_model function the classification task is defined only for two unique values of the target.