This PR adds explanations for Keras models that are used to classify images. Specifically we implement Grad-CAM.

For example, the following piece of code:

import keras
import numpy as np
import eli5

# load model
xception = keras.applications.xception.Xception(include_top=True, weights='imagenet', classes=1000)

# load image
im = keras.preprocessing.image.load_img('../eli5_examples/motorcycle.jpg', target_size=(299, 299))
doc = keras.preprocessing.image.img_to_array(im)
doc = np.expand_dims(doc, axis=0)
keras.applications.xception.preprocess_input(doc)

# explain
eli5.show_prediction(xception, doc)

produces this explanation for the class 'motor scooter':

The following features are added:

• Add keras package with explain_prediction_keras() and formatters.image module with format_as_image() (requires matplotlib and Pillow).
• Add .image attribute to base.Explanation and .heatmap to base.TargetExplanation.
• Make ipython.show_prediction() dispatch to an image display function for image explanations.

TODO items before this PR is finalized:

• Resolve reviews
• ~~Coverage~~
• ~~Pass CI~~
• ~~Mypy type annotations~~
• ~~Docs (formatting, tutorial)~~
• ~~Integration and unit tests~~

Hello there. There seems to be a constant feature names mismatch error when I use xgboost and pandas along with eli5.

Potential reasons:

1. eli5 does not access the feature names from within xgboost which can cause potential fallouts if the features are not in the same order as they were when they were put into xgboost or they are re-ordered once they enter eli5.

Would really appreciate help debugging and will push in a patch if fixed.

Towards fixing #15, allowing explain_weights for Pipelines. Does not handle explain_predictions; tracking the provenance of features back to parts of text may be tricky in a convoluted pipeline/union structure.

No tests yet.

Introduces new eli5.transform.transform_feature_names singledispatch, which could do with more implementations.

I think the exact forms of the names should remain experimental... or indeed we could refuse to provide default implementations for anything but vectorizers, selectors, Pipeline and FeatureUnion, letting users register their own.

Rubbish example:

from sklearn import *
from eli5 import explain_weights
from eli5.sklearn.transform import register_experimental_feature_names
from IPython.display import display
register_experimental_feature_names()
est = pipeline.make_pipeline(pipeline.FeatureUnion([('words', pipeline.make_pipeline(feature_extraction.text.CountVectorizer(),
                                                                                     feature_extraction.text.TfidfTransformer())),
                                                    ('chars', feature_extraction.text.CountVectorizer(analyzer='char'))]),
                             feature_selection.SelectKBest(feature_selection.chi2, k=10),
                             linear_model.LogisticRegression())
bunch = datasets.fetch_20newsgroups(categories=['alt.atheism', 'talk.religion.misc', 'rec.autos'])
est.fit(bunch.filenames, bunch.target)

display(explain_weights(est, target_names=bunch.target_names))

I'm not sure I'll be able to complete this patch any time soon, though I am keen to have something like this merged! Help is welcome.

Features with whitespaces in front get these whitespaces removed in HTML.

Compare:

+2.837  spa 
+2.805   spa

and

I think whitespaces should be replaced with   for HTML display. It could also make sense to use another background for text, in order to show whitespaces in the end.

This PR adds eli5.formatters.format_as_html. format_as_text is moved into eli5.formatters.text, but is accessible by the old name eli5.formatters.format_as_text. HTML formatting is done with Jinja2. This seems to be the most popular standalone templating library. But maybe just generating xml with etree from the standard library would be better (no need to learn another language) - what do you think @kmike ?

There is one thing that I punted on here: caveats and detailed explanations are rendered in a <pre> block. Probably a proper solution would be to represent them in some structured way, and then format for terminal or html display separately. I didn't do it because rendering in <pre> is easy and looks not too bad IMO.

html created during test runs are saved to .html folder, with names that include test module and the caller.

I was following this tutorial

https://github.com/TeamHG-Memex/eli5/blob/master/docs/source/_notebooks/debug-sklearn-crfsuite.rst

and got an error when trying to get the weights and features of the trained CRF model.

I can't get apply explain_weights neither show_weights for sklearn_crfsuite.CRF object.

I'm running all inside a python Jupyter notebook:

eli5.explain_weights_sklearn_crfsuite(crf)

AttributeError                            Traceback (most recent call last)
<ipython-input-75-2efd5261dc5d> in <module>()
      1 import eli5
----> 2 eli5.explain_weights_sklearn_crfsuite(crf)

~/virtual_envs/python3/lib/python3.6/site-packages/eli5/sklearn_crfsuite/explain_weights.py in explain_weights_sklearn_crfsuite(crf, top, target_names, targets, feature_re, feature_filter)
     26     ``feature_re`` and ``feature_filter`` parameters.
     27     """
---> 28     feature_names = np.array(crf.attributes_)
     29     state_coef = crf_state_coef(crf).todense().A
     30     transition_coef = crf_transition_coef(crf)

~/virtual_envs/python3/lib/python3.6/site-packages/sklearn_crfsuite/estimator.py in attributes_(self)
    463         A list of known attributes.
    464         """
--> 465         if self._info is None:
    466             return None
    467 

~/virtual_envs/python3/lib/python3.6/site-packages/sklearn_crfsuite/estimator.py in _info(self)
    497             return None
    498         if self._info_cached is None:
--> 499             self._info_cached = self.tagger_.info()
    500         return self._info_cached
    501 

pycrfsuite/_pycrfsuite.pyx in pycrfsuite._pycrfsuite.Tagger.info (pycrfsuite/_pycrfsuite.cpp:9581)()

pycrfsuite/_pycrfsuite.pyx in pycrfsuite._pycrfsuite.Tagger.info (pycrfsuite/_pycrfsuite.cpp:9498)()

~/virtual_envs/python3/lib/python3.6/site-packages/pycrfsuite/_dumpparser.py in feed(self, line)
     60             self.state = None
     61         else:
---> 62             getattr(self, 'parse_%s' % self.state)(line)
     63 
     64     def parse_FILEHEADER(self, line):

~/virtual_envs/python3/lib/python3.6/site-packages/pycrfsuite/_dumpparser.py in parse_STATE_FEATURES(self, line)
     83     def parse_STATE_FEATURES(self, line):
     84         m = re.match(r"\(\d+\) (.+) --> (.+): ([+-]?\d+\.\d+)", line)
---> 85         attr, label = m.group(1), m.group(2)
     86         assert attr in self.result.attributes
     87         assert label in self.result.labels

AttributeError: 'NoneType' object has no attribute 'group'

and with show_weigths I have a similar error

eli5.show_weights(crf, top=30)

AttributeError                            Traceback (most recent call last)
<ipython-input-76-ed63fb41bd58> in <module>()
----> 1 eli5.show_weights(crf, top=30)

~/virtual_envs/python3/lib/python3.6/site-packages/eli5/ipython.py in show_weights(estimator, **kwargs)
    116     """
    117     format_kwargs, explain_kwargs = _split_kwargs(kwargs)
--> 118     expl = explain_weights(estimator, **explain_kwargs)
    119     html = format_as_html(expl, **format_kwargs)
    120     return HTML(html)

~/virtual_envs/python3/lib/python3.6/site-packages/singledispatch.py in wrapper(*args, **kw)
    208 
    209     def wrapper(*args, **kw):
--> 210         return dispatch(args[0].__class__)(*args, **kw)
    211 
    212     registry[object] = func

~/virtual_envs/python3/lib/python3.6/site-packages/eli5/sklearn_crfsuite/explain_weights.py in explain_weights_sklearn_crfsuite(crf, top, target_names, targets, feature_re, feature_filter)
     26     ``feature_re`` and ``feature_filter`` parameters.
     27     """
---> 28     feature_names = np.array(crf.attributes_)
     29     state_coef = crf_state_coef(crf).todense().A
     30     transition_coef = crf_transition_coef(crf)

~/virtual_envs/python3/lib/python3.6/site-packages/sklearn_crfsuite/estimator.py in attributes_(self)
    463         A list of known attributes.
    464         """
--> 465         if self._info is None:
    466             return None
    467 

~/virtual_envs/python3/lib/python3.6/site-packages/sklearn_crfsuite/estimator.py in _info(self)
    497             return None
    498         if self._info_cached is None:
--> 499             self._info_cached = self.tagger_.info()
    500         return self._info_cached
    501 

pycrfsuite/_pycrfsuite.pyx in pycrfsuite._pycrfsuite.Tagger.info (pycrfsuite/_pycrfsuite.cpp:9581)()

pycrfsuite/_pycrfsuite.pyx in pycrfsuite._pycrfsuite.Tagger.info (pycrfsuite/_pycrfsuite.cpp:9498)()

~/virtual_envs/python3/lib/python3.6/site-packages/pycrfsuite/_dumpparser.py in feed(self, line)
     60             self.state = None
     61         else:
---> 62             getattr(self, 'parse_%s' % self.state)(line)
     63 
     64     def parse_FILEHEADER(self, line):

~/virtual_envs/python3/lib/python3.6/site-packages/pycrfsuite/_dumpparser.py in parse_STATE_FEATURES(self, line)
     83     def parse_STATE_FEATURES(self, line):
     84         m = re.match(r"\(\d+\) (.+) --> (.+): ([+-]?\d+\.\d+)", line)
---> 85         attr, label = m.group(1), m.group(2)
     86         assert attr in self.result.attributes
     87         assert label in self.result.labels

AttributeError: 'NoneType' object has no attribute 'group'

Nevertheless if I call crf.tagger_.dump(filename="crf_tagger.txt") I get a file crf_tagger.txt with all the information about features, state transitions, etc.

Related to #196 Add format_as_dataframe and format_as_dataframes, as discussed here https://github.com/TeamHG-Memex/eli5/issues/196#issuecomment-305909187

TODO:

• [x] fix class order for transition features
• [x] documentation

It would be good if Explanation could be exported not just to HTML, but to Pandas dataframes or a similar tabular format. This would enable further slicing and dicing, alternative methods of highlighting through DataFrame.style, and the ability to export to other on-disk formats.

• Python 3.5
• XGBoost '0.7.post3'
• sklearn '0.19.1'
• Pandas '0.22.0'
• ELI5 '0.8'

I'm working on a regression problem in insurance. XGB outperforms sklearn's tree ensembles. ELI5's show_weights works fine if I use XGBRegressor or RandomForestRegressor, if I use PermutationImportance then RandomForestRegressor works but XGBRegressor throws an error. I have a workaround noted below, I'm not sure where the problem lies. Any guidance would be happily received.

Estimator:

XGBRegressor(base_score=0.5, booster='gbtree', colsample_bylevel=1,
       colsample_bytree=1, gamma=0, learning_rate=0.1, max_delta_step=0,
       max_depth=2, min_child_weight=1, missing=None, n_estimators=119,
       n_jobs=1, nthread=None, objective='reg:tweedie', random_state=0,
       reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
       silent=True, subsample=1, tweedie_variance_power=1.9)

X_test_dbg.shape (1759, 3) 
y_test_dbg.shape (1759,)

type(X_test_dbg) # pandas.core.frame.DataFrame

If I train est on the DataFrame version of the data - training, testing and ELI5's show_weights are fine but PermutationImportance throws a feature-name error:

perm = PermutationImportance(est).fit(X_test_dbg, y_test_dbg)

ValueError: feature_names mismatch: ['myfeature1', 'myfeature2', 'myfeature3'] ['f0', 'f1', 'f2']
expected myfeature1, myfeature3, myfeature2 in input data
training data did not have the following fields: f1, f0, f2

If I change the type of the training data to an ndarray then PermutationImportance no longer throws an error:

X_train_dbg = X_train_dbg.as_matrix()
y_train_dbg = y_train_dbg.as_matrix()
X_test_dbg = X_test_dbg.as_matrix()
y_test_dbg = y_test_dbg.as_matrix()
type(X_train_dbg) # numpy.ndarray for all 4 variables
and then 
est.fit(X_train_dbg, y_train_dbg) 
perm = PermutationImportance(est).fit(X_test_dbg, y_test_dbg)

I'll note that with sklearn's RandomForestRegressor I can use Pandas DataFrames in PermutationImportance without problem, it is only an issue with XGBoost.

My features are two continuous values and one binary indicator, I have no categorical features. This is a smaller set of features from a larger problem (where the problem originated).

I got the clue to try as_matrix in this XGBoost bug report on sparse matrices - this might be a red herring as I'm not using any sparse matrices at all: https://github.com/dmlc/xgboost/issues/1238#issuecomment-275855469

Possibly related - this ELI5 bug report on Pandas and XGBoost has linked code, that code has a similar error (In[79] with show_predictions) to my error: https://github.com/TeamHG-Memex/eli5/issues/166 https://github.com/nareshshah139/titanic_rebranded/blob/master/ELI5_Example.ipynb but potentially as @kmike noted get_dummies is called separately on the train and validation set, so maybe they don't have matching feature names in this example?
​

Hi,

I was trying to execute the exact code mentioned at eli5 documentation over here..

xgboost tutorial : http://eli5.readthedocs.io/en/latest/tutorials/xgboost-titanic.html

but during below step...I am getting an error.. Code: from eli5 import show_prediction show_prediction(clf, valid_xs[1], vec=vec, show_feature_values=True)

Error Message:

Let me know if I am missing anything over here...

My Configuration :- Windows 7, Python 2.7.13

Any help would be really appreciated !

Using mypy instead of mypy-lang. Had to work about some existing mypy issues and filed one new. At least they are all known :) A couple of fixes were actual bugs in types: missing variants, extra commas, misplaced type annotations. Also mypy started to check None return type more strictly (it's required in type signature and explicit return None is required for Option return type). Fixes #168

Hello community,

I am using eli5 library for image explanations, I am referring this tutorial, but at the end I am facing estimator is not supported, Please help me to solve this error,

any help is highly appreciated,

Thanks!

while trying to import package eli5.lime, interpret gives the following error:

ImportError                               Traceback (most recent call last)
Cell In [10], line 1
----> 1 import eli5.lime

File c:\Users\magna\AppData\Local\Programs\Python\Python310\lib\site-packages\eli5\lime\__init__.py:2
      1 # -*- coding: utf-8 -*-
----> 2 from .lime import (
      3     TextExplainer,
      4     _train_local_classifier
      5 )

File c:\Users\magna\AppData\Local\Programs\Python\Python310\lib\site-packages\eli5\lime\lime.py:18
     16 import eli5
     17 from eli5.sklearn.utils import sklearn_version
---> 18 from eli5.lime.samplers import BaseSampler
     19 from eli5.lime.textutils import DEFAULT_TOKEN_PATTERN, CHAR_TOKEN_PATTERN
     20 from eli5.lime.samplers import MaskingTextSamplers

File c:\Users\magna\AppData\Local\Programs\Python\Python310\lib\site-packages\eli5\lime\samplers.py:9
      6 import six
      8 import numpy as np
----> 9 from scipy.stats import itemfreq
     10 from sklearn.base import BaseEstimator, clone
     11 from sklearn.neighbors import KernelDensity

ImportError: cannot import name 'itemfreq' from 'scipy.stats' (c:\Users\magna\AppData\Local\Programs\Python\Python310\lib\site-packages\scipy\stats\__init__.py)

I'm trying to pair RFECV with permutation importance. I would like to use the non-cv version of the permutation importance computation. I tried several approaches but I get always a NotFittedError for the estimator_func while trying to fit the rfecv. Am I missing something? Everything works fine when i pass a cv instance to the permutation function, even without fitting the estimator. Help is greatly appreciated

estimator_funct = estimator_funct.fit(extract_relevant_features, choosen_target)
estimator_funct.fit(extract_relevant_features, choosen_target)

pi = PermutationImportance(estimator_funct,  scoring='r2', n_iter=10, random_state=1)
rfecv = RFECV(
   estimator=pi,
   step=1,
   cv=cv_func,
   scoring=score,
   min_features_to_select=min_features_to_select,
)

rfecv.fit(extract_relevant_features,
         choosen_target,
         groups=extract_relevant_features.index) 

```python


The exception:
...
   rfecv.fit(extract_relevant_features,
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\feature_selection\_rfe.py", line 723, in fit
   scores = parallel(
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\feature_selection\_rfe.py", line 724, in <genexpr>
   func(rfe, self.estimator, X, y, train, test, scorer)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\feature_selection\_rfe.py", line 37, in _rfe_single_fit
   return rfe._fit(
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\feature_selection\_rfe.py", line 296, in _fit
   estimator.fit(X[:, features], y, **fit_params)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\eli5\sklearn\permutation_importance.py", line 204, in fit
   si = self._non_cv_scores_importances(X, y)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\eli5\sklearn\permutation_importance.py", line 232, in _non_cv_scores_importances
   base_score, importances = self._get_score_importances(score_func, X, y)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\eli5\sklearn\permutation_importance.py", line 236, in _get_score_importances
   return get_score_importances(score_func, X, y, n_iter=self.n_iter,
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\eli5\permutation_importance.py", line 86, in get_score_importances
   base_score = score_func(X, y)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\metrics\_scorer.py", line 219, in __call__
   return self._score(
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\metrics\_scorer.py", line 261, in _score
   y_pred = method_caller(estimator, "predict", X)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\metrics\_scorer.py", line 71, in _cached_call
   return getattr(estimator, method)(*args, **kwargs)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\ensemble\_forest.py", line 989, in predict
   check_is_fitted(self)
 File "C:\Users\T450\anaconda3\envs\Master\lib\site-packages\sklearn\utils\validation.py", line 1345, in check_is_fitted
   raise NotFittedError(msg % {"name": type(estimator).__name__})
sklearn.exceptions.NotFittedError: This RandomForestRegressor instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

Hi there, thanks for your contribution. I have some questions about eli5.show_weights.

How are weights.pos and weights.neg defined? How do you compute the hsl color for the cell in the html table? Thank you

I have trained a binary classification model with Keras on a custom dataset. Want to check what is in the image that made the model predict a certain class over another? It worked for only one image and I am no longer able to test it on the other images. Does it mean my model is corrupted? Or is it a bug of ELI5? These are the versions of my packages in the notebook:

tensorflow==1.15.2
scikit-learn==0.21.3
keras==2.3.1
eli5==0.10.1

Thanks in advance

Hi,

I wanted to generate the explain_prediction_df for each of my observations in a test set, but when I loop over it it crashes eventually.

for i in range(len(X_test)): eli5.explain_prediction_df(clf, X_test[i,:]).to_csv('f{i}.csv')

I am getting a userwarning: X has feature names, but clf was fitted without feature names.

Do you have any idea what is going on or if there is an easier way to go about this?