DESlib is an easy-to-use ensemble learning library focused on the implementation of the state-of-the-art techniques for dynamic classifier and ensemble selection. The library is is based on scikit-learn, using the same method signatures: fit, predict, predict_proba and score. All dynamic selection techniques were implemented according to the definitions from [1].
Dynamic Selection:
Dynamic Selection (DS) refers to techniques in which the base classifiers are selected dynamically at test time, according to each new sample to be classified. Only the most competent, or an ensemble of the most competent classifiers is selected to predict the label of a specific test sample. The rationale for these techniques is that not every classifier in the pool is an expert in classifying all unknown samples, but rather each base classifier is an expert in a different local region of the feature space.
DS is one of the most promising MCS approaches (Multiple Classifier Systems) due to an increasing number of empirical studies reporting superior performance over static combination methods. Such techniques have achieved better classification performance especially when dealing with small-sized and imbalanced datasets.
DESlib is tested to work with Python 3.5, 3.6 and 3.7. The dependency requirements are:
scipy(>=1.4.0)
numpy(>=1.17.0)
scikit-learn(>=0.20.0)
These dependencies are automatically installed using the pip commands above.
Examples:
Here we show an example using the KNORA-E method with random forest as a pool of classifiers:
fromdeslib.des.knora_eimportKNORAE# Train a pool of 10 classifierspool_classifiers=RandomForestClassifier(n_estimators=10)
pool_classifiers.fit(X_train, y_train)
# Initialize the DES modelknorae=KNORAE(pool_classifiers)
# Preprocess the Dynamic Selection dataset (DSEL)knorae.fit(X_dsel, y_dsel)
# Predict new examples:knorae.predict(X_test)
The library accepts any list of classifiers (compatible with scikit-learn) as input, including a list containing different classifier models (heterogeneous ensembles). More examples on how to use the API can be found in the documentation and in the Examples directory.
Organization:
The library is divided into four modules:
deslib.des: Implementation of DES techniques (Dynamic Ensemble Selection).
deslib.dcs: Implementation of DCS techniques (Dynamic Classifier Selection).
deslib.static: Implementation of baseline ensemble methods.
deslib.util: A collection of aggregation functions and diversity measures for ensemble of classifiers.
Variations of each DES techniques are also provided by the library (e.g., different versions of the META-DES framework).
The following techniques are also available for all methods:
For DES techniques, the combination of the selected classifiers can be done as Dynamic Selection (majority voting), Dynamic Weighting (weighted majority voting) or a Hybrid (selection + weighting).
For all DS techniques, Dynamic Frienemy Pruning (DFP) [13] can be used.
For all DS techniques, Instance Hardness (IH) can be used to classify easy samples with a KNN and hard samples using the DS technique. More details on IH and Dynamic Selection can be found in [14].
As an optional requirement, the fast KNN implementation from FAISS can be used to speed-up the computation of the region of competence.
Citation
If you use DESLib in a scientific paper, please consider citing the following paper:
@article{JMLR:v21:18-144,
author = {Rafael M. O. Cruz and Luiz G. Hafemann and Robert Sabourin and George D. C. Cavalcanti},
title = {DESlib: A Dynamic ensemble selection library in Python},
journal = {Journal of Machine Learning Research},
year = {2020},
volume = {21},
number = {8},
pages = {1-5},
url = {http://jmlr.org/papers/v21/18-144.html}
}
: R. M. O. Cruz, R. Sabourin, and G. D. Cavalcanti, “Dynamic classifier selection: Recent advances and perspectives,” Information Fusion, vol. 41, pp. 195 – 216, 2018.
[2]
(1, 2) : A. S. Britto, R. Sabourin, L. E. S. de Oliveira, Dynamic selection of classifiers - A comprehensive review, Pattern Recognition 47 (11) (2014) 3665–3680.
[3]
(1, 2) : A. H. R. Ko, R. Sabourin, u. S. Britto, Jr., From dynamic classifier selection to dynamic ensemble selection, Pattern Recognition 41 (2008) 1735–1748.
[4]
(1, 2) : K. Woods, W. P. Kegelmeyer, Jr., K. Bowyer, Combination of multiple classifiers using local accuracy estimates, IEEE Transactions on Pattern Analysis Machine Intelligence 19 (1997) 405–410.
: G. Giacinto, F. Roli, Dynamic classifier selection based on multiple classifier behaviour, Pattern Recognition 34 (2001) 1879–1881.
[6]
(1, 2) : L. Didaci, G. Giacinto, F. Roli, G. L. Marcialis, A study on the performances of dynamic classifier selection based on local accuracy estimation, Pattern Recognition 38 (11) (2005) 2188–2191.
: R. M. O. Cruz, R. Sabourin, G. D. C. Cavalcanti, T. I. Ren, META-DES: A dynamic ensemble selection framework using meta-learning, Pattern Recognition 48 (5) (2015) 1925–1935.
: Cruz, R.M., Sabourin, R. and Cavalcanti, G.D., 2015, July. META-DES. H: a dynamic ensemble selection technique using meta-learning and a dynamic weighting approach. In Neural Networks (IJCNN), 2015 International Joint Conference on (pp. 1-8)
: P. R. Cavalin, R. Sabourin, C. Y. Suen, Dynamic selection approaches for multiple classifier systems, Neural Computing and Applications 22 (3-4) (2013) 673–688.
: T.Woloszynski, M. Kurzynski, A measure of competence based on randomized reference classifier for dynamic ensemble selection, in: International Conference on Pattern Recognition (ICPR), 2010, pp. 4194–4197.
[12]
(1, 2) : T. Woloszynski, M. Kurzynski, P. Podsiadlo, G. W. Stachowiak, A measure of competence based on random classification for dynamic ensemble selection, Information Fusion 13 (3) (2012) 207–213.
: Oliveira, D.V.R., Cavalcanti, G.D.C. and Sabourin, R., Online Pruning of Base Classifiers for Dynamic Ensemble Selection, Pattern Recognition, vol. 72, December 2017, pp 44-58.
: R. M. O. Cruz, R. Sabourin, G. D. C. Cavalcanti, META-DES.Oracle: Meta-learning and feature selection for dynamic ensemble selection, Information Fusion 38 (2017) 84–103.Nov 30;38:84-103.
[16]
(1, 2) : R. G. F. Soares, A. Santana, A. M. P. Canuto, M. C. P. de Souto, Using accuracy and diversity to select classifiers to build ensembles, Proceedings of the International Joint Conference on Neural Networks (2006) 1310–1316.
[17]
: L. I. Kuncheva, Combining Pattern Classifiers: Methods and Algorithms, Wiley-Interscience, 2004.
[18]
: Shipp, Catherine A., and Ludmila I. Kuncheva. "Relationships between combination methods and measures of diversity in combining classifiers." Information fusion 3.2 (2002): 135-148.
: M. Sabourin, A. Mitiche, D. Thomas, G. Nagy, Classifier combination for handprinted digit recognition, International Conference on Document Analysis and Recognition (1993) 163–166.
: L. I. Kuncheva, A theoretical study on six classifier fusion strategies, IEEE Transactions on Pattern Analysis and Machine Intelligence 24 (2) (2002) 281–286.
[21]
(1, 2) : B. Antosik, M. Kurzynski, New measures of classifier competence – heuristics and application to the design of multiple classifier systems., in: Computer recognition systems 4., 2011, pp. 197–206.
[22]
: Smith, Michael R., Tony Martinez, and Christophe Giraud-Carrier. "An instance level analysis of data complexity." Machine learning 95.2 (2014), pp 225-256.
: P. C. Smits, Multiple classifier systems for supervised remote sensing image classification based on dynamic classifier selection, IEEE Transactions on Geoscience and Remote Sensing 40 (4) (2002) 801–813.
: García, S., Zhang, Z.L., Altalhi, A., Alshomrani, S. and Herrera, F., "Dynamic ensemble selection for multi-class imbalanced datasets." Information Sciences 445 (2018): 22-37.
As you have mentioned in your examples, BaggingClassifier or RandomeForest classifier are considered as a pool of classifier itself.
I am wondering is it possible if I create a pool of classifiers including traditional ensemble methods like RF, Adaboost in combination of single classifiers like SVM, kNN?
It seems that other DS methods (i.e., OLA, MLA, DESP, and etc) are successfully fitting the X_dsel on the DS models, however DESClustering is raising the value error:
ValueError Traceback (most recent call last)
<ipython-input-10-6583b8d75519> in <module>
11 desclustering = DESClustering(pool_classifiers=pool_classifiers, random_state = rng, clustering = kmeans)
12
---> 13 desclustering.fit(X_dsel, y_dsel)
14 end = time.clock()
15 print(" DES-Clustering fitting time for 5 patients in DSEL = {}".format(end))
~/deslib-env/lib/python3.6/site-packages/deslib/des/des_clustering.py in fit(self, X, y)
132 self.J_ = int(np.ceil(self.n_classifiers_ * self.pct_diversity))
133
--> 134 self._check_parameters()
135
136 if self.clustering is None:
~/deslib-env/lib/python3.6/site-packages/deslib/des/des_clustering.py in _check_parameters(self)
374 if self.N_ <= 0 or self.J_ <= 0:
375 raise ValueError("The values of N_ and J_ should be higher than 0"
--> 376 "N_ = {}, J_= {} ".format(self.N_, self.J_))
377 if self.N_ < self.J_:
378 raise ValueError(
ValueError: The values of N_ and J_ should be higher than 0N_ = 0, J_= 1
Why this is happening when a Random Forest is given as a pool to DESClustering?
Is it because I have a pre-trained RF model and loading it as the pool? Is there any difference between loading a pre-trained classifier and training the pool with RF model (as it can be seen here
...\Anaconda3\lib\site-packages\sklearn\model_selection_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.
warnings.warn(CV_WARNING, FutureWarning)
...\Anaconda3\lib\site-packages\sklearn\calibration.py:455: RuntimeWarning: invalid value encountered in multiply
TEP_minus_T1P = P * (T * E - T1)
...\Anaconda3\lib\site-packages\sklearn\calibration.py:455: RuntimeWarning: invalid value encountered in multiply
TEP_minus_T1P = P * (T * E - T1)
...\PhD\Anaconda3\lib\site-packages\sklearn\calibration.py:455: RuntimeWarning: invalid value encountered in multiply
TEP_minus_T1P = P * (T * E - T1)
...\Anaconda3\lib\site-packages\sklearn\model_selection_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.
warnings.warn(CV_WARNING, FutureWarning)
...\Anaconda3\lib\site-packages\sklearn\svm\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
...\Anaconda3\lib\site-packages\sklearn\svm\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
...\Anaconda3\lib\site-packages\sklearn\svm\base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
"the number of iterations.", ConvergenceWarning)
...\Anaconda3\lib\site-packages\sklearn\svm\base.py:193: FutureWarning: The default value of gamma will change from 'auto' to 'scale' in version 0.22 to account better for unscaled features. Set gamma explicitly to 'auto' or 'scale' to avoid this warning.
"avoid this warning.", FutureWarning)
...\Anaconda3\lib\site-packages\deslib\des\probabilistic\base.py:191: RuntimeWarning: invalid value encountered in true_divide
competences = competences / sum_potential.reshape(-1, 1)
...\Anaconda3\lib\site-packages\deslib\des\probabilistic\base.py:222: RuntimeWarning: invalid value encountered in greater
selected_classifiers = (competences > selection_threshold)
I have been using the Deslib library to classify a binary problem and used various DES algorithms. When I applied the model on the test dataset, I ran into some problems.
According to my understanding, for a binary classification problem, 0.5 is the default threshold. If the "predict_proba" is higher than 0.5, then it should be classified as 1, otherwise 0.
However, for some instances when the predicted probabilites were higher than 0.5, I got a 0 (instead of 1) and when the predicted probabilites were less than 0.5, I got a 1 (instead of 0).
I have never seen this kind of phenomenon whilst using the scikit-learn library, therefore I would like to know if this is normal behaviour in DES?
Please note that I am using the latest version (0.3.5) of DESlib.
I used Faiss method for knn calculation. Creating DS models and saving them was performed, and then the model was fitted into DSEL dataset. However, once I tried to do prediction and scoring for the test set, I am facing an error:
score = knorae.score(X_test, y_test)
File "/home/esara/deslib-env/lib/python3.6/site-packages/sklearn/base.py", line 357, in score
return accuracy_score(y, self.predict(X), sample_weight=sample_weight)
File "/home/esara/deslib-env/lib/python3.6/site-packages/deslib/base.py", line 440, in predict
distances, neighbors = self._get_region_competence(X_DS)
File "/home/esara/deslib-env/lib/python3.6/site-packages/deslib/base.py", line 381, in _get_region_competence
return_distance=True)
File "/home/esara/deslib-env/lib/python3.6/site-packages/deslib/util/faiss_knn_wrapper.py", line 112, in kneighbors
dist, idx = self.index_.search(X, n_neighbors)
AttributeError: 'numpy.ndarray' object has no attribute 'search'
Do you have any idea about that? Your help and expert opinion is appreciated in advance.
The current FaissKNNClassifier class only perform brute force search. The class should also allow approximate methods in order to speed up inference time on large datasets.
More info about approximate search on faiss can be found in the following links: https://github.com/facebookresearch/faiss/wiki/Getting-started and https://github.com/facebookresearch/faiss/wiki/Faster-search
Hi,
Thank you for your great contribution to science.
I have a question and wondering if DESlib can be applied for GCNs or is there any other library similar to DESlib algorithms that they have extended DESlib for dynamic selection of GCNs?
This PR allows to use DESlib with multiples datasets. In simple terms, instead of using data with the next shape (samples x dimensions), we can use data with this shape (datasets x samples x dimensions). It's useful when we have different types of data for the same samples.
It's not compatible with all techniques, lacks some static techniques, but it is for a lot of them. Some checks in base.py have been disabled to make the feature works. Finally, some code has been let unfinished or blank as all normal technique features were not required in the original research it came from.
Hey, I'm working on a research paper focused on building a binary classification model in the biomedical domain.The dataset comprises approximately 800 data points. Let's say I want to feed an heterogeneous pool of classifiers to the dynamic selection methods. By following the instructions on the examples, I've found two different ways of splitting the dataset and fitting the base classifiers of the pool.
Split in train/test (e.g., 75/25) and then split the training in train/dsel (e.g., 50/50).
In this random forest example, the RF is fitted on the 75% training portion and the DS methods on the 50% DSEL portion.
In all the other examples, the 50% training portion is used to fit the classifier and the 50% DSEL portion is used to fit DS methods.
Furthermore, I wanted to point out this tip taken from the tutorial :
An important point here is that in case of small datasets or when the base classifier models in the pool are weak estimators such as Decision Stumps or Perceptrons, an overlap between the training data and DSEL may be beneficial for achieving better performance.
That seems my case, as my dataset is rather small compared to most datasets in the ML domain. Hence, I was thinking of fitting my base classifiers on the 75% part and then leveraging some overlap to get better performance (and this is really the case! In fact, overlapping leads to a median auc of 0.76 whereas non-overlapping gives 0.71).
What would be the best way of dealing with the problem ?
When I use XGBClassifier (from XGboost library) using any DES or DCS algorithm, I am getting a features_names mismatch error. I have pooled several other classifiers successfully. This error only arises when Xgboost is included in the pooled classifiers. Moreover I have successfully been able to use XGBoost in the Deslib Stacked Classifier algorithm. Please note as per previous advice, I have installed the latest version (0.3.5) of the library using the code:
Do any of the implementations of KNN within this repo extend to the multi-label case? That is to say, usable in the context of multi-object detection from neural network embeddings or similar.
I was particularly hopeful that KNNE might work? But it's not obvious from the relevant literature.
zoofs is a Python library for performing feature selection using a variety of nature-inspired wrapper algorithms. The algorithms range from swarm-intelligence to physics-based to Evolutionary. It's easy to use , flexible and powerful tool to reduce your feature size.
2.1k Jan 2, 2023
122 Dec 27, 2022
1.1k Dec 18, 2022
606 Dec 21, 2022
373 Dec 24, 2022
3.7k Jan 1, 2023
764 Dec 31, 2022
1 Jan 25, 2022
2 Nov 5, 2021
47 Nov 22, 2022
70 Nov 16, 2022
2.3k Jan 1, 2023
139 Dec 28, 2022
10 Oct 7, 2022
168 Dec 30, 2022
1.3k Jan 4, 2023