Lime: Explaining the predictions of any machine learning classifier

Overview

lime

Build Status Binder

This project is about explaining what machine learning classifiers (or models) are doing. At the moment, we support explaining individual predictions for text classifiers or classifiers that act on tables (numpy arrays of numerical or categorical data) or images, with a package called lime (short for local interpretable model-agnostic explanations). Lime is based on the work presented in this paper (bibtex here for citation). Here is a link to the promo video:

KDD promo video

Our plan is to add more packages that help users understand and interact meaningfully with machine learning.

Lime is able to explain any black box classifier, with two or more classes. All we require is that the classifier implements a function that takes in raw text or a numpy array and outputs a probability for each class. Support for scikit-learn classifiers is built-in.

Installation

The lime package is on PyPI. Simply run:

pip install lime

Or clone the repository and run:

pip install .

We dropped python2 support in 0.2.0, 0.1.1.37 was the last version before that.

Screenshots

Below are some screenshots of lime explanations. These are generated in html, and can be easily produced and embedded in ipython notebooks. We also support visualizations using matplotlib, although they don't look as nice as these ones.

Two class case, text

Negative (blue) words indicate atheism, while positive (orange) words indicate christian. The way to interpret the weights by applying them to the prediction probabilities. For example, if we remove the words Host and NNTP from the document, we expect the classifier to predict atheism with probability 0.58 - 0.14 - 0.11 = 0.31.

twoclass

Multiclass case

multiclass

Tabular data

tabular

Images (explaining prediction of 'Cat' in pros and cons)

Tutorials and API

For example usage for text classifiers, take a look at the following two tutorials (generated from ipython notebooks):

For classifiers that use numerical or categorical data, take a look at the following tutorial (this is newer, so please let me know if you find something wrong):

For image classifiers:

For regression:

Submodular Pick:

The raw (non-html) notebooks for these tutorials are available here.

The API reference is available here.

What are explanations?

Intuitively, an explanation is a local linear approximation of the model's behaviour. While the model may be very complex globally, it is easier to approximate it around the vicinity of a particular instance. While treating the model as a black box, we perturb the instance we want to explain and learn a sparse linear model around it, as an explanation. The figure below illustrates the intuition for this procedure. The model's decision function is represented by the blue/pink background, and is clearly nonlinear. The bright red cross is the instance being explained (let's call it X). We sample instances around X, and weight them according to their proximity to X (weight here is indicated by size). We then learn a linear model (dashed line) that approximates the model well in the vicinity of X, but not necessarily globally. For more information, read our paper, or take a look at this blog post.

Contributing

Please read this.

Comments
  • LIME explain instance resulting in empty graph

    LIME explain instance resulting in empty graph

    Hi. I'm working with a spark data frame and to be able to make use of LIME we had to make some modifications:

    def new_predict_fn(data):
        sdf = map(lambda x: (int(x[0]), Vectors.dense(x[0:])), data)
        sdf = spark.createDataFrame(sdf, schema=["id", "features"]).select("features")
        predictions = cv_model.transform(sdf).select("prediction")
        return predictions.toPandas()["prediction"].values.reshape(-1)
    
    lime_df_test = nsdf.select('features').toPandas()
    lime_df_test = pd.DataFrame.from_records(lime_df_test['features'].tolist())
    
    exp = explainer.explain_instance(lime_df_test.iloc[10].values, new_predict_fn, num_features=20)
    display(exp.as_pyplot_figure())
    

    However, when it results in an "empty" explanation:

    [('3575 <= 2199.13', 0.0), ('3981 <= 2189.88', 0.0), ('3987 <= 2189.88', 0.0), ('4527 <= 93.00', 0.0), ('4003 <= 1.00', 0.0), ('4528 <= 0.00', 0.0), ('3824 <= 14000000.00', 0.0), ('4256 <= 2199.73', 0.0), ('3685 <= 2190.45', 0.0), ('3579 <= 2199.13', 0.0)]

    We are looking for some reason for this to happen. A simple test with the modifications mentioned above worked well, but using real data (with more than 3000) columns, we faced that problem. The only idea that comes to my mind is that LIME is not being able to explain an instance locally (?). But I'm not sure if that makes sense. I'm also wondering (now) if it's not just a case that the weights are plotted with 1 decimal place of precision and if (how) I could change that.

    Thanks.

    opened by paulaceccon 21
  • How to interpret LIME results?

    How to interpret LIME results?

    I am considering using LIME, and I am having some struggle to understand what exactly it outputs.

    I posed a question on stack exchange with a MCVE, but maybe this is more suitable here.

    Consider the following code, that uses logistic regression to fit a logistic process, and uses LIME for a new example.

    import numpy as np
    import lime.lime_tabular
    from sklearn.linear_model import LogisticRegression
    
    # generate a logistic latent variable from `a` and `b` with coef. 1, 1
    data = []
    for t in range(100000):
        a = 1 - 2 * np.random.random()
        b = 1 - 2 * np.random.random()
        noise = np.random.logistic()
        c = int(a + b + noise > 0)  # to predict
        data.append([a, b, c])
    data = np.array(data)
    
    x = data[:, :-1]
    y = data[:, -1]
    
    # fit Logistic regression without regularization (C=inf)
    classifier = LogisticRegression(C=1e10)
    classifier.fit(x, y)
    
    print(classifier.coef_)
    
    # "explain" with LIME
    explainer = lime.lime_tabular.LimeTabularExplainer(
                    x, mode='classification',
                    feature_names=['a', 'b'])
    
    explanation = explainer.explain_instance(np.array([1, 1]), classifier.predict_proba, num_samples=100000)
    print(explanation.as_list())
    

    output:

    [[ 0.9981159   0.99478328]]  # print(classifier.coef_)
    [('a > 0.50', 0.219), ('b > 0.50', 0.219)] # print(explanation.as_list())
    

    the ~[[1, 1]] is because we are doing logistic regression to a Logistic process with these coefficients.

    What do the values 0.219... mean? Are they relatable to any quantity of this example?

    opened by jorgecarleitao 19
  • Regression support

    Regression support

    I took @aikramer2's work in #52 and made the following minor changes to satisfy flake8:

    • synced the branch with the latest master from the base project
    • new LimeError class
    • lots of formatting (mostly lines that were >79
    • removed two unused labels variables.
    opened by Aylr 18
  • numerical data (categoric & continuous) explanation on SVC, and NN

    numerical data (categoric & continuous) explanation on SVC, and NN

    I follow your example from https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20continuous%20and%20categorical%20features.html for continuous and categorical data and give it a try with different models. I used SVC (from sklearn) and NN (from keras). somehow both of the method I used get crashed and restart the kernel when it try to get the explanation (exp=...), code below.

    ` #1 using SVC predict_fn = lambda x: svm_linear.predict_proba(encoder.transform(x)) explainer = lime.lime_tabular.LimeTabularExplainer(train ,feature_names = feature_names,class_names=class_names, categorical_features=categorical_features, categorical_names=categorical_names, kernel_width=3)

    all_explains = {} for i in range(test.shape[0]): exp = explainer.explain_instance(test[i], predict_fn, num_features=5) all_explains[i]=exp`

    ` #2 using NN from keras.models import Sequential from keras.layers import Dense, Dropout, Activation from keras.optimizers import SGD

    model = Sequential() model.add(Dense(32, input_dim=encoded_train.shape[1], activation='relu')) model.add(Dense(32,activation='relu')) model.add(Dense(1, activation='sigmoid'))

    model.compile(loss='binary_crossentropy',optimizer='rmsprop',metrics=['accuracy'])

    model.fit(encoded_train_toa, labels_train, epochs=30, batch_size=128) score = model.evaluate(encoded_test_toa, labels_test, batch_size=128)

    def trans(x): x = encoder.transform(x).toarray() return model.predict_proba(x)

    import lime from lime import lime_tabular explainer = lime.lime_tabular.LimeTabularExplainer(train ,feature_names = feature_names,class_names=class_names, categorical_features=categorical_features, categorical_names=categorical_names) all_explains={} predict_fn = lambda x: trans(x) for i in range(test.shape[0]): temp = test[i,:] exp = explainer.explain_instance(temp, predict_fn, num_features=5) all_explains[i]=exp ` is SVM and NN not supported yet for numerical data? Because I have no problem using it on tree-based classifiers.

    opened by yusufazishty 18
  • Found input variables with inconsistent numbers of samples: [5000, 1]

    Found input variables with inconsistent numbers of samples: [5000, 1]

    Not sure if a new version of scikit-learn is messing this up or not but I get this error when trying to run an explanation:

    Found input variables with inconsistent numbers of samples: [5000, 1]

    The outer-error occurs in lime_base.py here: https://github.com/marcotcr/lime/blob/master/lime/lime_base.py#L75

    The inner error is thrown in scikit-learn here: https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/validation.py#L180

    I have tried to follow the multi-class notebook example as closely as I could but I do not see anything I could change to make my data look any more like the one in the example. That is, all of my classifier outputs look exactly like what's given in the example.

    Any suggestions?

    Thanks!

    opened by courageon 16
  • LIME vs feature importance

    LIME vs feature importance

    Hi,

    I have a question regarding the feature importance vs LIME.

    For the adult data set, we can see the feature importance of my model -

    importance

    However, when I set plotting various lime plots, I will post a few:

    1

    2

    3

    I ran around 20 plots and mostly we can see for example the variable marital status to be used in the decision. However, for the feature importance, it is slightly low. Is there a reason for this?

    feature importance let us know that the more important features are on the higher nodes for splitting. For the LIME, it is ordered by values. Is it correct to understand that more important features does not necessarily means that it will result in larger gain/loss in the LIME?

    opened by germayneng 14
  • Added RecurrentTabularExplainer

    Added RecurrentTabularExplainer

    This addresses issue #56, and adds a class to handle tabular data when the model is a stateless keras-style recurrent neural network. In that case, the model expects data with shape (n_samples, n_timesteps, n_features). This mostly uses the machinery of the existing LimeTabularExplainer, but reshapes data in a few places so that the user doesn't have to worry about reshaping themselves.

    opened by kgullikson88 14
  • Does the code work for regressors yet?

    Does the code work for regressors yet?

    Or is it only scikit-learn classifiers for now? I am interested in using LIME for some work that I am doing that uses SVRs. Would it be a lot of work to extend LIME to work with regression models? I am happy to help.

    opened by desilinguist 14
  • Add PySpark MLlib support

    Add PySpark MLlib support

    "All we require is that the classifier implements a function that takes in raw text or a numpy array and outputs a probability for each class."

    Necessarily to operate within the context in where it acts, Spark MLlib takes more complicated data structures as input. Would it be possible to add support for Spark? If it is, I might do it, but I'm not sure it is possible.

    opened by rjurney 13
  • Is it possible to do incremental training on LimeTabularExplainer?

    Is it possible to do incremental training on LimeTabularExplainer?

    Hi, I have a data, I fit a model, store the model, later I get new data, I don't want to retrain with full data, so I fit the new data, may I know is it possible to create explainer as a incremental fit for the new data.

    data = [[1, 2], [0.5, 6], [0, 10], [1, 18]]
    scaler = MinMaxScaler()
    scaler.partial_fit(data)
    sc_data = scaler.transform(data)
    model1 = IForest(contamination=0.1).fit(sc_data)
    explainer = lime.lime_tabular.LimeTabularExplainer(sc_data, 
                                                          mode='classification',
                                                          feature_names=feature_names,
                                                          kernel_width=5,
                                                          random_state=42,
                                                          discretize_continuous=False)
    

    I store the model, scaler, explainer for serving purpose, after some time i get more data, so fit the new data to the same model, is it possible for explainer as well?

    data2 = [[15, 12], [15, 16], [0, 11], [1, 18]]
    scaler = load(scaler)
    loaded_model = load(model1)
    
    scaler.partial_fit(data2)
    sc_data2 = scaler.transform(data2)
    model2 = loaded_model.fit(sc_data2)
    explainer = lime.lime_tabular.LimeTabularExplainer(????????????????)
    

    Thanks in advance for the inputs.

    opened by hanzigs 12
  • Interpreting Fine-tuned Bert model using LIME

    Interpreting Fine-tuned Bert model using LIME

    Thanks for this amazing work. I am trying to interpret Fine-tuned BERT model using Transformer framework. It seems there is tokenization issue, when I try to use LIME with BERT. Here is the error that i am getting:

    Traceback (most recent call last):
      File "src/predict.py", line 351, in <module>
        exp = explainer.explain_instance(s, prediction.predictor, num_features=6)
      File "/home/ramesh/.virtualenvs/transformer-env/lib/python3.6/site-packages/lime/lime_text.py", line 417, in explain_instance
        distance_metric=distance_metric)
      File "/home/ramesh/.virtualenvs/transformer-env/lib/python3.6/site-packages/lime/lime_text.py", line 484, in __data_labels_distances
        labels = classifier_fn(inverse_data)
      File "src/predict.py", line 297, in predictor
        input_ids, input_mask, segment_ids = self.convert_text_to_features(text)
      File "src/predict.py", line 135, in convert_text_to_features
        tokens_a = self.tokenizer.tokenize(text_a)
      File "/home/ramesh/.virtualenvs/transformer-env/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 649, in tokenize
        tokenized_text = split_on_tokens(added_tokens, text)
      File "/home/ramesh/.virtualenvs/transformer-env/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 637, in split_on_tokens
        if sub_text not in self.added_tokens_encoder \
    TypeError: unhashable type: 'list'
    

    Here is my code:

    def predictor(self, text):
    
            max_seq_length=128
            input_ids, input_mask, segment_ids = self.convert_text_to_features(text)
            self.model.to(self.device)
    
            with torch.no_grad():
                outputs = self.model(input_ids, input_mask, segment_ids)
    
            logits = outputs[0]
            logits = F.softmax(logits, dim=1)
    
            return logits.numpy()
    
    def convert_text_to_features(self, text_a, text_b=None):
    
            features = []
            cls_token = self.tokenizer.cls_token
            sep_token = self.tokenizer.sep_token
            cls_token_at_end = False
            sequence_a_segment_id = 0
            sequence_b_segment_id = 1
            cls_token_segment_id = 1
            pad_token_segment_id = 0
            mask_padding_with_zero = True
            pad_token = 0
            tokens_a = self.tokenizer.tokenize(text_a)
            tokens_b = None
    
            self._truncate_seq_pair(tokens_a, self.max_seq_length - 2)
    
            tokens = tokens_a + [sep_token]
            segment_ids = [sequence_a_segment_id] * len(tokens)
    
            if tokens_b:
                tokens += tokens_b + [sep_token]
                segment_ids += [sequence_b_segment_id] * (len(tokens_b) + 1)
    
    
            tokens = [cls_token] + tokens
            segment_ids = [cls_token_segment_id] + segment_ids
    
            input_ids = self.tokenizer.convert_tokens_to_ids(tokens)
    
            input_mask = [1 if mask_padding_with_zero else 0] * len(input_ids)
            padding_length = self.max_seq_length - len(input_ids)
    
    
            input_ids = input_ids + ([pad_token] * padding_length)
            input_mask = input_mask + ([0 if mask_padding_with_zero else 1] * padding_length)
            segment_ids = segment_ids + ([pad_token_segment_id] * padding_length)
    
            assert len(input_ids) == self.max_seq_length
            assert len(input_mask) == self.max_seq_length
            assert len(segment_ids) == self.max_seq_length
    
            input_ids = torch.tensor([input_ids], dtype=torch.long).to(self.device)
            input_mask = torch.tensor([input_mask], dtype=torch.long).to(self.device)
            segment_ids = torch.tensor([segment_ids], dtype=torch.long).to(self.device)
            return input_ids, input_mask, segment_ids
    
    
    if __name__ == '__main__':
    
        model_path = "models/mrpc"
        bert_model_class = "bert"
        prediction = Prediction(bert_model_class, model_path, lower_case=True, seq_length=128)
        label_names = [0, 1]
        explainer = LimeTextExplainer(class_names=label_names)
        train_df = pd.read_csv("data/train.tsv", sep = '\t')
    
        for example in train_df["string"]:
            exp = explainer.explain_instance(example, prediction.predictor, num_features=6)
            print(exp.as_list())
    

    I have checked this issue356, but still i cannot figure out my problem.

    Any leads will be appreciated.

    Thank you :)

    opened by rameshjes 12
  • Can the sampling around an instance method be part of the utils?

    Can the sampling around an instance method be part of the utils?

    I have noticed that sampling is done by a private method __data_inverse of the LimeTabularExplainer class. Is there a way to access the sampling algorithm without initializing the class object?

    opened by HardikPrabhu 0
  • Problems with implementing LIME, can't get binary 0 or 1 outputs, only probabilities

    Problems with implementing LIME, can't get binary 0 or 1 outputs, only probabilities

    Hope everyone is holding on tight, we're soon at Christmas!

    Hi, as I have already made a post about the issue in detail over at stackexchange, I'll just post the link here and a quick summary of my problem:

    Link here: https://datascience.stackexchange.com/questions/116799/problems-with-implementing-lime?noredirect=1#comment117787_116799

    My problem is that I want to show my random forest score through the predict.proba(x_test)[:, 1] method. However, it gives me this error code when I try use LIME to show the score using that prediction:

    Cell In [31], line 3 1 i = 49 ----> 3 exp = explainer.explain_instance(x_test.iloc[i], predict_fn_rf, num_features=5) 4 exp.show_in_notebook(show_table=True)

    File ~\OneDrive\Documents\Anaconda\lib\site-packages\lime\lime_tabular.py:361, in LimeTabularExplainer.explain_instance(self, data_row, predict_fn, labels, top_labels, num_features, num_samples, distance_metric, model_regressor) 359 if self.mode == "classification": 360 if len(yss.shape) == 1: --> 361 raise NotImplementedError("LIME does not currently support " 362 "classifier models without probability " 363 "scores. If this conflicts with your " 364 "use case, please let us know: " 365 "https://github.com/datascienceinc/lime/issues/16") 366 elif len(yss.shape) == 2: 367 if self.class_names is None:

    NotImplementedError: LIME does not currently support classifier models without probability scores. If this conflicts with your use case, please let us know: https://github.com/datascienceinc/lime/issues/16

    opened by Ostlimpa 0
  • Add predict_fn_accept_dense_only to explain_instance when input is sparse but model takes dense

    Add predict_fn_accept_dense_only to explain_instance when input is sparse but model takes dense

    Add predict_fn_accept_dense_only to explain_instance.

    When data_row (of explain_instance) is scipy.sparse.matrix but the model was NOT trained with scipy.sparse.matrix, the predict_fn_accept_dense_only convert inverse to dense before call predict_fn (and then back to sparse after call predict_fn)

    Bin [email protected]

    opened by dbinlbl 1
  • background/filter dataset in LIME

    background/filter dataset in LIME

    Hi LIME community, Thanks for the help or any lead in advance.

    One thing we are looking for is to provide a background data for LIME at the time of explanation of every instance. We are not sure if it is available in LIME? More specifically, it looks like the shap.KernelExplainer' data parameter. https://shap-lrjball.readthedocs.io/en/latest/generated/shap.KernelExplainer.html

    One possible method is to provide feature_selection (e.g, highest_weights) for LimeTabularExplainer or explain_instance_with_data. But, it looks like LimeTabularExplainer needs to build the explainer for every instance, and explain_instance_with_data needs users to provide neighborhood_data etc.

    Bests, Bin

    opened by dbinlbl 1
  • Create text classification tutorial using tensorflow

    Create text classification tutorial using tensorflow

    I've noticed that a few people are talking about using LIME with TensorFlow models. There was an issue, which was addressed before, but I believe this tutorial can provide a guide for users since it uses a very popular dataset (Kaggle's tweets classification).

    In this notebook, I've used the Universal Sentence Encoder from TensorFlow Hub to classify the text data.

    Hopefully, it will be helpful.

    opened by Nour-Aldein2 0
Releases(0.2.0.0)
Owner
Marco Tulio Correia Ribeiro
Marco Tulio Correia Ribeiro
Algorithms for monitoring and explaining machine learning models

Alibi is an open source Python library aimed at machine learning model inspection and interpretation. The focus of the library is to provide high-qual

Seldon 1.9k Dec 30, 2022
treeinterpreter - Interpreting scikit-learn's decision tree and random forest predictions.

TreeInterpreter Package for interpreting scikit-learn's decision tree and random forest predictions. Allows decomposing each prediction into bias and

Ando Saabas 720 Dec 22, 2022
Using / reproducing ACD from the paper "Hierarchical interpretations for neural network predictions" 🧠 (ICLR 2019)

Hierarchical neural-net interpretations (ACD) ?? Produces hierarchical interpretations for a single prediction made by a pytorch neural network. Offic

Chandan Singh 111 Jan 3, 2023
A game theoretic approach to explain the output of any machine learning model.

SHAP (SHapley Additive exPlanations) is a game theoretic approach to explain the output of any machine learning model. It connects optimal credit allo

Scott Lundberg 18.3k Jan 8, 2023
Visualizer for neural network, deep learning, and machine learning models

Netron is a viewer for neural network, deep learning and machine learning models. Netron supports ONNX (.onnx, .pb, .pbtxt), Keras (.h5, .keras), Tens

Lutz Roeder 20.9k Dec 28, 2022
Visualizer for neural network, deep learning, and machine learning models

Netron is a viewer for neural network, deep learning and machine learning models. Netron supports ONNX, TensorFlow Lite, Keras, Caffe, Darknet, ncnn,

Lutz Roeder 16.3k Sep 27, 2021
A data-driven approach to quantify the value of classifiers in a machine learning ensemble.

Documentation | External Resources | Research Paper Shapley is a Python library for evaluating binary classifiers in a machine learning ensemble. The

Benedek Rozemberczki 187 Dec 27, 2022
Visual analysis and diagnostic tools to facilitate machine learning model selection.

Yellowbrick Visual analysis and diagnostic tools to facilitate machine learning model selection. What is Yellowbrick? Yellowbrick is a suite of visual

District Data Labs 3.9k Dec 30, 2022
FairML - is a python toolbox auditing the machine learning models for bias.

======== FairML: Auditing Black-Box Predictive Models FairML is a python toolbox auditing the machine learning models for bias. Description Predictive

Julius Adebayo 338 Nov 9, 2022
A library that implements fairness-aware machine learning algorithms

Themis ML themis-ml is a Python library built on top of pandas and sklearnthat implements fairness-aware machine learning algorithms. Fairness-aware M

Niels Bantilan 105 Dec 30, 2022
Interpretability and explainability of data and machine learning models

AI Explainability 360 (v0.2.1) The AI Explainability 360 toolkit is an open-source library that supports interpretability and explainability of datase

null 1.2k Dec 29, 2022
A collection of research papers and software related to explainability in graph machine learning.

A collection of research papers and software related to explainability in graph machine learning.

AstraZeneca 1.9k Dec 26, 2022
L2X - Code for replicating the experiments in the paper Learning to Explain: An Information-Theoretic Perspective on Model Interpretation.

L2X Code for replicating the experiments in the paper Learning to Explain: An Information-Theoretic Perspective on Model Interpretation at ICML 2018,

Jianbo Chen 113 Sep 6, 2022
JittorVis - Visual understanding of deep learning model.

JittorVis is a deep neural network computational graph visualization library based on Jittor.

null 182 Jan 6, 2023
Lime: Explaining the predictions of any machine learning classifier

lime This project is about explaining what machine learning classifiers (or models) are doing. At the moment, we support explaining individual predict

Marco Tulio Correia Ribeiro 10.3k Jan 1, 2023
A library for debugging/inspecting machine learning classifiers and explaining their predictions

ELI5 ELI5 is a Python package which helps to debug machine learning classifiers and explain their predictions. It provides support for the following m

null 2.6k Dec 30, 2022
Wandb-predictions - WANDB Predictions With Python

WANDB API CI/CD Below we capture the CI/CD scenarios that we would expect with o

Anish Shah 6 Oct 7, 2022
Static Features Classifier - A static features classifier for Point-Could clusters using an Attention-RNN model

Static Features Classifier This is a static features classifier for Point-Could

ABDALKARIM MOHTASIB 1 Jan 25, 2022
Using Bert as the backbone model for lime, designed for NLP task explanation (sentence pair text classification task)

Lime Comparing deep contextualized model for sentences highlighting task. In addition, take the classic explanation model "LIME" with bert-base model

JHJu 2 Jan 18, 2022
Algorithms for monitoring and explaining machine learning models

Alibi is an open source Python library aimed at machine learning model inspection and interpretation. The focus of the library is to provide high-qual

Seldon 1.9k Dec 30, 2022