• I've been using this Bayesian kernel density classifier for a few years and I thought I should move it out from my poorly organized project to this one here.

The prior is $P(y=0)$. I primarily use it for spatial problems.

It is similar to the GMM Classifier with only 2 caveats I can think of.

• Hyperparameters are easier to decide on.
• Scaling is worse as I believe due to the KDE part scaling linearly with the sample size.

# noinspection PyPep8Naming
class BayesianKernelDensityClassifier(BaseEstimator, ClassifierMixin):
    """
    Bayesian Classifier that uses Kernel Density Estimations to generate the joint distribution
    Parameters:
        - bandwidth: float
        - kernel: for scikit learn KernelDensity
    """
    def __init__(self, bandwidth=0.2, kernel='gaussian'):
        self.classes_, self.models_, self.priors_logp_ = [None] * 3
        self.bandwidth = bandwidth
        self.kernel = kernel

    def fit(self, X, y):
        self.classes_ = np.sort(np.unique(y))
        training_sets = [X[y == yi] for yi in self.classes_]
        self.models_ = [KernelDensity(bandwidth=self.bandwidth, kernel=self.kernel).fit(x_subset)
                        for x_subset in training_sets]

        self.priors_logp_ = [np.log(x_subset.shape[0] / X.shape[0]) for x_subset in training_sets]
        return self

    def predict_proba(self, X):
        logp = np.array([model.score_samples(X) for model in self.models_]).T
        result = np.exp(logp + self.priors_logp_)
        return result / result.sum(1, keepdims=True)

    def predict(self, X):
        return self.classes_[np.argmax(self.predict_proba(X), 1)]

Please explain clearly what you'd like to see added.

• [x] convince us of the use-case, we're open to many suggestions but we prefer to solve problems with pipelines that are at least somewhat general When using the package in a Docker container without C installed, installation (can?) fail on CVXPY

• [x] ~~add a screenshot if applicable (ML stuff is hard to explain with words, pictures say 1000 words)~~

• [x] ~~make sure that the feature you want is not already supported by sklearn~~

Happy to work on this if you agree

Fixed the bug caused when using the grouped_estimator with a string column as grouping variable.

Solution: Try the checks without adjustments, if that fails: remove the grouping column from the array or dataframe.

• replace two parameters df, date_col by one date_serie
• insure no mutation with a copy
• check if index match and if index is unique
• optimise the iloc/loc/get_loc with a join and index
• add plotting function with unit test
• add summary function with unit test
• update notebook doc

Description:

I'm able to use the thresholder on sklearn's voting classifer, but not on the stacking classifier. It throws this error, which I believe is in error. StackingClassifier does have predict_proba. Maybe I'm missunderstanding the use case, but this seems to fit.

ValueError: The Thresholder meta model only works on classifcation models with .predict_proba.

Code for reproduction (using the sklearn sample data for StackingClassifier):

from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import LinearSVC
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import StackingClassifier

X, y = load_iris(return_X_y=True)
estimators = [
    ('rf', RandomForestClassifier(n_estimators=10, random_state=42)),
    ('svr', make_pipeline(StandardScaler(), LinearSVC(random_state=42)))]
clf = StackingClassifier(    estimators=estimators, final_estimator=LogisticRegression())
clf.fit(X, y)

a = Thresholder(clf, threshold=0.2)
a.fit(X, y)
a.predict(X)

Full trace:

ValueError                                Traceback (most recent call last)
<ipython-input-26-1b89dbfa16b8> in <module>
     16 
     17 a = Thresholder(clf, threshold=0.2)
---> 18 a.fit(X_train_std, np.ceil(y_train[targets[2]]))
     19 a.predict(X_train_std)

~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\sklego\meta\thresholder.py in fit(self, X, y, sample_weight)
     54         self.estimator_ = clone(self.model)
     55         if not isinstance(self.estimator_, ProbabilisticClassifier):
---> 56             raise ValueError(
     57                 "The Thresholder meta model only works on classifcation models with .predict_proba."
     58             )

ValueError: The Thresholder meta model only works on classifcation models with .predict_proba.

As far as outlier detection is concerned, this is the current flow:

import numpy as np
import pandas as pd
import plotnine as p9

from sklego.mixture import GMMOutlierDetector

X = np.random.normal(-10, 1, (2000, 2))
mod = GMMOutlierDetector(n_components=1, threshold=0.99).fit(X)

df = pd.DataFrame({"x1": X[:, 0], "x2": X[:, 1],
                   "loglik": mod.gmm.score_samples(X), 
                   "prediction": mod.predict(X).astype(str)})

(p9.ggplot() + p9.geom_point(data=df, mapping=p9.aes("x1", "x2", color="loglik")))

(p9.ggplot() + p9.geom_point(data=df, mapping=p9.aes("x1", "x2", color="prediction")))

I feel that by copying the log_step function many times and slightly changing the logging section I'm repeating a lot of code. Do you have any suggestion to avoid this?

This is WIP, some TODOs:

• rename log_step -> log_shape.
• tests, docs.
• log_dtypes

In EstimatorTransformer the underlying estimator is being fitted without the ability to pass along additional arguments to self.estimator_.fit.

This limits use cases for EstimatorTransformer. For example, if the underlying estimator is an XGBClassifier we would like to be able to pass eval_set to monitor validation performance and enable early stopping. This is currently not possible. Adding *args, **kwargs should fix this issue.

https://github.com/koaning/scikit-lego/blob/b4d087f0131ff164e6feebf238356ba6512b3635/sklego/meta/estimator_transformer.py#L31

I worked on this a while ago in PR #147, but I started from a fresh branch because I decided to limit the scope only to repeating basis functions (#20), excluding the spanning basis functions (#29).

Feedback adapted so far:

• Write a transformer when X contains only one column
• Write a wrapper which selects one column, when X has more than one column
• Added basic tests
• Added docstring

To Do:

• Write documentation

Could someone review the code? I'll work on the documentation in the meantime as well.

Tagging people involved in PR #147: @kayhoogland @koaning @MBrouns

Addresses changes in #192 and #232 I am currently working on a Time Series problem with vibration data where I needed a functionality like the one suggested in #232 so I decided to add it here.

I tried to explain the changes in functionality in the docstring:

Each validation fold doesn't overlap. The entire 'window' moves by 1 valid_duration until there is not enough data. If this would lead to more splits then specified with n_splits, the 'window' moves by the validation_duration times the fraction of possible splits and requested splits -- n_possible_splits = (total_length-train_duration-gap_duration)//valid_duration -- time_shift = valid_duratiopn n_possible_splits/n_slits so the CV spans the whole dataset. If train_duration is not passed but n_split, the training duration is increased to -- train_duration = total_length-(self.gap_duration + self.valid_duration * self.n_splits) such that the shifting the entire window by one validation duration spans the whole training set

The changes are also added to the docs notebook for visualization.

In the world of hype @MBrouns and myself came up with a very normal thought.

One way to acomplish this is by introduction of a new meta model: WontPredict.

This thread is meant as a place to discuss the implementation.

Hi!

The only feature selections that scikit-learn offers are quite naive. MRMR seems like a bit more advanced and reasonable approach to select informative and non-redundant features as described here.

Long story short:

1. Pick a feature that is most informative in some metric (e.g. F-statistic).
2. Pick the next feature that is very informative, but doesn't correlate with the previous feature too much (e.g. the average absolute Pearson correlation between the current feature and the feature selected in step 2).
3. Pick the next feature that is very informative, but doesn't correlate with the previous 2 features too much.
4. Pick the next feature that is very informative, but doesn't correlate with the previous 3 features too much.
5. (repeat until K features selected)

Here, K, the measure of information and correlation can be specified by the user.

Hi,

When using @log_step in debugging a Pandas Pipeline, the current function must accept a single argument of df:pd.Dataframe.

However if the user sends all the parameters as kwargs there is an error .

It would be useful if the @log_step will check the first kwargs and if it is a pd.Dataframe then it will convert it into an arg - possible implementation before running the def wrapper()as follows

    _kwargs = {**kwargs}
    first_arg= next(iter(_kwargs))
    if isinstance(_kwargs[first_arg],pd.DataFrame) and len(args)==0:
        args=args+(_kwargs.pop[first_arg],)

Thanks for this great set of extensions to sklearn.

The Tresholder() model is quite close to something I've been looking for for a while.

I'm looking to include threshold optimisation as part of a broader parameter search.

I can perhaps best describe the desired behaviour as follows

for each parameters in grid:
    fit model with parameters
    for each threshold in thresholds:
        evaluate model

However, if I pass a model that has not yet been fit to Thresholder(), then, even with refit=False, the same model is fit also for each threshold.

Is there an easy way around this? Thinking about this the best way to achieve this would be tinkering with the GridSearchCV code, but perhaps you have an idea and would also find this interesting?

Thanks!

Before working on a large PR, please check with @koaning or @MBrouns that they agree with the direction of the PR. This discussion should take place in a Github issue before working on the PR, unless it's a minor change like spelling in the docs.

Description

Consider you want to build a semi-auto Pipeline. So, the pipeline may looks like:

import pandas as pd
import numpy as np
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.preprocessing import OneHotEncoder, StandardScaler

transformer = Pipeline([
    ('features', FeatureUnion(n_jobs=1, transformer_list=[
        # Part 1
        ('boolean', Pipeline([
            ('selector', PandasTypeSelector(include='bool')),
        ])),  # booleans close
        
        ('numericals', Pipeline([
            ('selector', PandasTypeSelector(include='number')),
            ('scaler', StandardScaler()),
            ('add_pca', FeatureUnion([
                ('orig', IdentityTransformer()),
                ('pca', PCA(2))
            ]))
        ])),  # numericals close
        
        # Part 2
        ('categoricals', Pipeline([
            ('selector', PandasTypeSelector(include='category')),
            ('labeler', StringIndexer()),
            ('encoder', OneHotEncoder(handle_unknown='ignore')),
        ]))  # categoricals close
    
    ])),  # features close
])  # pipeline close

There may be boolean, numericals and categoricals variables , but may not exists erther . Current behaviour is raise Exception when a Selector return empty DataFrame , I think we can expose a parameter let user choose .

Fixes # (issue)

Type of change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x ] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• [ ] My code follows the style guidelines (flake8)
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation (also to the readme.md)
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added tests to check whether the new feature adheres to the sklearn convention
• [ ] New and existing unit tests pass locally with my changes

If you feel your PR is ready for a review, ping @koaning or @mbrouns.

This PR solves issue #543 and implements get_feature_names_out for all relevant transformers in sklego.preprocessing (i.e. transformers that do not contain the TrainOnlyTransformerMixin).

Functionality is implemented through adding the _ClassNamePrefixFeaturesOutMixin to the class and making sure self._n_features_out is defined in .fit. This is also generally how scikit-learn implements get_feature_names_out for many of its transformers (Example). Unit tests are added for all new functionality.

P.S. Don't pay attention to the commit history before October 10th. These changes have already been merged into koaning/scikit-lego/main, but is still displayed here as commit history. Will try to fix this. Suggestions to remove these redundant commits from the commit history of this CarloLepelaars/scikit-lego/ fork are welcome.

• [x] ~~Find alternative solution for using _ClassNamePrefixFeaturesOutMixin so it works with scikit-learn on Python 3.7. (Remove Python 3.7. support)~~
• [x] Add implementation of get_feature_names_out to contributing guidelines so people implement this for each new preprocessor that is not TrainOnly.
• [x] Remove Python 3.7. GitHub Actions pipelines and update Optional dependencies GitHub Actions pipeline to use Python 3.8.
• [x] Add general unit test that checks if get_feature_names_out can be called for all relevant preprocessors and EstimatorTransformer.