Library for machine learning stacking generalization.

Last update: Jul 19, 2022

Related tags

Deep Learning machine-learning

Overview

stacked_generalization

Implemented machine learning *stacking technic[1]* as handy library in Python. Feature weighted linear stacking is also available. (See https://github.com/fukatani/stacked_generalization/tree/master/stacked_generalization/example)

Including simple model cache system Joblibed claasifier and Joblibed Regressor.

Feature

1) Any scikit-learn model is availavle for Stage 0 and Stage 1 model.

And stacked model itself has the same interface as scikit-learn library.

You can replace model such as RandomForestClassifier to stacked model easily in your scripts. And multi stage stacking is also easy.

ex.

from stacked_generalization.lib.stacking import StackedClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.linear_model import LogisticRegression, RidgeClassifier
from sklearn import datasets, metrics
iris = datasets.load_iris()

# Stage 1 model
bclf = LogisticRegression(random_state=1)

# Stage 0 models
clfs = [RandomForestClassifier(n_estimators=40, criterion = 'gini', random_state=1),
        GradientBoostingClassifier(n_estimators=25, random_state=1),
        RidgeClassifier(random_state=1)]

# same interface as scikit-learn
sl = StackedClassifier(bclf, clfs)
sl.fit(iris.target, iris.data)
score = metrics.accuracy_score(iris.target, sl.predict(iris.data))
print("Accuracy: %f" % score)

More detail example is here. https://github.com/fukatani/stacked_generalization/blob/master/stacked_generalization/example/cross_validation_for_iris.py

https://github.com/fukatani/stacked_generalization/blob/master/stacked_generalization/example/simple_regression.py

2) Evaluation model by out-of-bugs score.

Stacking technic itself uses CV to stage0. So if you use CV for entire stacked model, *each stage 0 model are fitted n_folds squared times.* Sometimes its computational cost can be significent, therefore we implemented CV only for stage1[2].

For example, when we get 3 blends (stage0 prediction), 2 blends are used for stage 1 fitting. The remaining one blend is used for model test. Repitation this cycle for all 3 blends, and averaging scores, we can get oob (out-of-bugs) score *with only n_fold times stage0 fitting.*

ex.

sl = StackedClassifier(bclf, clfs, oob_score_flag=True)
sl.fit(iris.data, iris.target)
print("Accuracy: %f" % sl.oob_score_)

3) Caching stage1 blend_data and trained model. (optional)

If cache is exists, recalculation for stage 0 will be skipped. This function is useful for stage 1 tuning.

sl = StackedClassifier(bclf, clfs, save_stage0=True, save_dir='stack_temp')

Feature of Joblibed Classifier / Regressor

Joblibed Classifier / Regressor is simple cache system for scikit-learn machine learning model. You can use it easily by minimum code modification.

At first fitting and prediction, model calculation is performed normally. At the same time, model fitting result and prediction result are saved as .pkl and .csv respectively.

At second fitting and prediction, if cache is existence, model and prediction results will be loaded from cache and never recalculation.

e.g.

from sklearn import datasets
from sklearn.cross_validation import StratifiedKFold
from sklearn.ensemble import RandomForestClassifier
from stacked_generalization.lib.joblibed import JoblibedClassifier

# Load iris
iris = datasets.load_iris()

# Declaration of Joblibed model
rf = RandomForestClassifier(n_estimators=40)
clf = JoblibedClassifier(rf, "rf")

train_idx, test_idx = list(StratifiedKFold(iris.target, 3))[0]

xs_train = iris.data[train_idx]
y_train = iris.target[train_idx]
xs_test = iris.data[test_idx]
y_test = iris.target[test_idx]

# Need to indicate sample for discriminating cache existence.
clf.fit(xs_train, y_train, train_idx)
score = clf.score(xs_test, y_test, test_idx)

Software Requirement

Python (2.7 or 3.5 or later)
numpy
scikit-learn
pandas

Installation

pip install stacked_generalization

License

MIT License. (http://opensource.org/licenses/mit-license.php)

Copyright

Many part of the implementation of stacking is based on the following. Thanks! https://github.com/log0/vertebral/blob/master/stacked_generalization.py

Other

Any contributions (implement, documentation, test or idea...) are welcome.

References

[1] L. Breiman, "Stacked Regressions", Machine Learning, 24, 49-64 (1996). [2] J. Sill1 et al, "Feature Weighted Linear Stacking", https://arxiv.org/abs/0911.0460, 2009.

A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization

sam.pytorch A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization ( Foret+2020) Paper, Official implementa

102 Dec 28, 2022

The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

Domain Generalization for Medical Imaging Classification with Linear Dependency Regularization The code release of paper 'Domain Generalization for Me

56 Dec 28, 2022

Domain Generalization with MixStyle, ICLR'21.

MixStyle This repo contains the code of our ICLR'21 paper, "Domain Generalization with MixStyle". The OpenReview link is https://openreview.net/forum?

208 Dec 28, 2022

Benchmarks for semi-supervised domain generalization.

Semi-Supervised Domain Generalization This code is the official implementation of the following paper: Semi-Supervised Domain Generalization with Stoc

49 Dec 10, 2022

Sharpness-Aware Minimization for Efficiently Improving Generalization

Sharpness-Aware-Minimization-TensorFlow This repository provides a minimal implementation of sharpness-aware minimization (SAM) (Sharpness-Aware Minim

54 Dec 8, 2022

PyTorch evaluation code for Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.

Out-of-distribution Generalization Investigation on Vision Transformers This repository contains PyTorch evaluation code for Delving Deep into the Gen

72 Dec 13, 2022

Official implementation of paper Gradient Matching for Domain Generalization

Gradient Matching for Domain Generalisation This is the official PyTorch implementation of Gradient Matching for Domain Generalisation. In our paper,

94 Dec 23, 2022

Code used to generate the results appearing in "Train longer, generalize better: closing the generalization gap in large batch training of neural networks"

Train longer, generalize better - Big batch training This is a code repository used to generate the results appearing in "Train longer, generalize bet

145 Sep 16, 2022

CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)

CrossNorm (CN) and SelfNorm (SN) (Accepted at ICCV 2021) This is the official PyTorch implementation of our CNSN paper, in which we propose CrossNorm

100 Dec 28, 2022

Comments

something goes wrong when I use command line "pip install stacked_generalization"
here is the detail. pip install stacked_generalization Collecting stacked_generalization Using cached https://files.pythonhosted.org/packages/d0/99/e42d99355f00068aa2f8eef5f88c1610558f10217e2704067b5134629397/stacked_generalization-0.0.4.zip Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-mm6gy5sl/stacked-generalization/setup.py", line 28, in long_description=read_md('Readme.md'), File "/tmp/pip-build-mm6gy5sl/stacked-generalization/setup.py", line 13, in read_md = lambda f: pypandoc.convert(f, 'rst') File "/home/wxk/anaconda3/lib/python3.6/site-packages/pypandoc/init.py", line 66, in convert raise RuntimeError("Format missing, but need one (identified source as text as no " RuntimeError: Format missing, but need one (identified source as text as no file with that name was found).

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-mm6gy5sl/stacked-generalization/

Please tell me how to fix it if you know it.
opened by wEEang763162 2
fix import util

you can try it with pip3 install -e [email protected]:geoHeil/stacked_generalization.git@fixImportUtil#egg=stacked-generalization

I fixed the failing import of utils

opened by geoHeil 1
ModuleNotFoundError and ImportError due to deprecated sklearn sub-modules
Hi,

I was trying to rerun the example, after doing:

from stacked_generalization.lib.stacking import FWLSRegressor

I got the following errors, for stacking.py and util.py:

ModuleNotFoundError: No module named 'sklearn.cross_validation' ImportError: cannot import name 'joblib' from 'sklearn.externals'

According to https://stackoverflow.com/questions/30667525/importerror-no-module-named-sklearn-cross-validation and https://stackoverflow.com/questions/61893719/importerror-cannot-import-name-joblib-from-sklearn-externals, it seems to be related to the deprecation of sub-modules. It worked for me to replace all the:

from sklearn.cross_validation import StratifiedKFold, KFold from sklearn.externals import joblib

by

from sklearn.model_selection import StratifiedKFold, KFold import joblib

Thanks.
opened by ShaunFChen 0
Improve memory management.

Current version of stacked_genelarization continue to holt all stage0 model after fitting. But in case of huge stacking model (~100), stacked_genelarization needs huge memory.

By adding lazy evaluation, I want to improve memory management. i.e. hold stage0 model only when predicting, and release memory. (Save model to joblib cache and load only when needed.)

opened by fukatani 0

Library for machine learning stacking generalization.

Related tags

Overview

stacked_generalization

Feature

1) Any scikit-learn model is availavle for Stage 0 and Stage 1 model.

And stacked model itself has the same interface as scikit-learn library.

2) Evaluation model by out-of-bugs score.

3) Caching stage1 blend_data and trained model. (optional)

Feature of Joblibed Classifier / Regressor

Software Requirement

Installation

License

Copyright

Other

References

You might also like...

A PyTorch implementation of Sharpness-Aware Minimization for Efficiently Improving Generalization

The code release of paper 'Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization' NIPS 2020.

Domain Generalization with MixStyle, ICLR'21.

Benchmarks for semi-supervised domain generalization.

Sharpness-Aware Minimization for Efficiently Improving Generalization

PyTorch evaluation code for Delving Deep into the Generalization of Vision Transformers under Distribution Shifts.

Official implementation of paper Gradient Matching for Domain Generalization

Code used to generate the results appearing in "Train longer, generalize better: closing the generalization gap in large batch training of neural networks"

CrossNorm and SelfNorm for Generalization under Distribution Shifts (ICCV 2021)

Comments

something goes wrong when I use command line "pip install stacked_generalization"

Please tell me how to fix it if you know it.

fix import util

ModuleNotFoundError and ImportError due to deprecated sklearn sub-modules

Improve memory management.

Owner

StackRec: Efficient Training of Very Deep Sequential Recommender Models by Iterative Stacking

RGB-stacking 🛑 🟩 🔷 for robotic manipulation

BESS: Balanced Evolutionary Semi-Stacking for Disease Detection via Partially Labeled Imbalanced Tongue Data

[CVPR'21] FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space

A list of papers regarding generalization in (deep) reinforcement learning

Official repository for CVPR21 paper "Deep Stable Learning for Out-Of-Distribution Generalization".

Implementation for paper "Towards the Generalization of Contrastive Self-Supervised Learning"

Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.