Implementation of hyperparameter optimization/tuning methods for machine learning & deep learning models

Li Yang

Last update: Dec 19, 2022

Related tags

Deep Learning machine-learning deep-learning random-forest optimization svm genetic-algorithm machine-learning-algorithms hyperparameter-optimization artificial-neural-networks grid-search tuning-parameters knn bayesian-optimization hyperparameter-tuning random-search particle-swarm-optimization hpo python-examples python-samples hyperband

Overview

Hyperparameter Optimization of Machine Learning Algorithms

This code provides a hyper-parameter optimization implementation for machine learning algorithms, as described in the paper:
L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization.

This paper and code will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.

Paper

On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
One-column version: arXiv
Two-column version: Elsevier

Quick Navigation

Section 3: Important hyper-parameters of common machine learning algorithms
Section 4: Hyper-parameter optimization techniques introduction
Section 5: How to choose optimization techniques for different machine learning models
Section 6: Common Python libraries/tools for hyper-parameter optimization
Section 7: Experimental results (sample code in "HPO_Regression.ipynb" and "HPO_Classification.ipynb")
Section 8: Open challenges and future research directions
Summary table for Sections 3-6: Table 2: A comprehensive overview of common ML models, their hyper-parameters, suitable optimization techniques, and available Python libraries
Summary table for Sections 8: Table 10: The open challenges and future directions of HPO research

Implementation

Sample code for hyper-parameter optimization implementation for machine learning algorithms is provided in this repository.

Sample code for Regression problems

HPO_Regression.ipynb
Dataset used: Boston-Housing

Sample code for Classification problems

HPO_Classification.ipynb
Dataset used: MNIST

Machine Learning & Deep Learning Algorithms

Random forest (RF)
Support vector machine (SVM)
K-nearest neighbor (KNN)
Artificial Neural Networks (ANN)

Hyperparameter Configuration Space

ML Model	Hyper-parameter	Type	Search Space
RF Classifier	n_estimators	Discrete	[10,100]
	max_depth	Discrete	[5,50]
	min_samples_split	Discrete	[2,11]
	min_samples_leaf	Discrete	[1,11]
	criterion	Categorical	'gini', 'entropy'
	max_features	Discrete	[1,64]
SVM Classifier	C	Continuous	[0.1,50]
	kernel	Categorical	'linear', 'poly', 'rbf', 'sigmoid'
KNN Classifier	n_neighbors	Discrete	[1,20]
ANN Classifier	optimizer	Categorical	'adam', 'rmsprop', 'sgd'
	activation	Categorical	'relu', 'tanh'
	batch_size	Discrete	[16,64]
	neurons	Discrete	[10,100]
	epochs	Discrete	[20,50]
	patience	Discrete	[3,20]
RF Regressor	n_estimators	Discrete	[10,100]
	max_depth	Discrete	[5,50]
	min_samples_split	Discrete	[2,11]
	min_samples_leaf	Discrete	[1,11]
	criterion	Categorical	'mse', 'mae'
	max_features	Discrete	[1,13]
SVM Regressor	C	Continuous	[0.1,50]
	kernel	Categorical	'linear', 'poly', 'rbf', 'sigmoid'
	epsilon	Continuous	[0.001,1]
KNN Regressor	n_neighbors	Discrete	[1,20]
ANN Regressor	optimizer	Categorical	'adam', 'rmsprop'
	activation	Categorical	'relu', 'tanh'
	loss	Categorical	'mse', 'mae'
	batch_size	Discrete	[16,64]
	neurons	Discrete	[10,100]
	epochs	Discrete	[20,50]
	patience	Discrete	[3,20]

HPO Algorithms

Grid search
Random search
Hyperband
Bayesian Optimization with Gaussian Processes (BO-GP)
Bayesian Optimization with Tree-structured Parzen Estimator (BO-TPE)
Particle swarm optimization (PSO)
Genetic algorithm (GA)

Requirements

Contact-Info

Please feel free to contact me for any questions or cooperation opportunities. I'd be happy to help.

Email: [email protected]
GitHub: LiYangHart and Western OC2 Lab
LinkedIn: Li Yang
Google Scholar: Li Yang and OC2 Lab

Citation

If you find this repository useful in your research, please cite this article as:

L. Yang and A. Shami, “On hyperparameter optimization of machine learning algorithms: Theory and practice,” Neurocomputing, vol. 415, pp. 295–316, 2020, doi: https://doi.org/10.1016/j.neucom.2020.07.061.

@article{YANG2020295,
title = "On hyperparameter optimization of machine learning algorithms: Theory and practice",
author = "Li Yang and Abdallah Shami",
volume = "415",
pages = "295 - 316",
journal = "Neurocomputing",
year = "2020",
issn = "0925-2312",
doi = "https://doi.org/10.1016/j.neucom.2020.07.061",
url = "http://www.sciencedirect.com/science/article/pii/S0925231220311693"
}

Comments

LightGBMError: Cannot change bin_construct_sample_cnt after constructed Dataset handle.

Environment info

Operating System: windows 10 Python version: python 3.8 LightGBM version: 3.3.2

code

This error occurs when I execute this code. seek help The code is in this file HPO_Classification.ipynb

params = {'class_weight': 'balanced',
'boosting_type': 'dart',
'num_leaves': 44,
'learning_rate': 0.05979047181117413,
'subsample_for_bin': 220000,
'min_child_samples': 390,
'reg_alpha': 0.3877551020408163,
'reg_lambda': 0.5102040816326531,
'colsample_bytree': 0.7333333333333333,
'subsample': 0.8535353535353536}

def random_objective(params, iteration, n_folds = N_FOLDS):
   """Random search objective function. Takes in hyperparameters
      and returns a list of results to be saved."""

   start = timer()
   
   # Perform n_folds cross validation
   cv_results = lgb.cv(params, train_set, num_boost_round = 10000, nfold = n_folds, 
                       early_stopping_rounds = 100, metrics = 'auc', seed = 50)
   end = timer()
   best_score = np.max(cv_results['auc-mean'])
   
   # Loss must be minimized
   loss = 1 - best_score
   
   # Boosting rounds that returned the highest cv score
   n_estimators = int(np.argmax(cv_results['auc-mean']) + 1)
   
   # Return list of results
   return [loss, params, iteration, n_estimators, end - start]

%%capture

random.seed(50)

for i in range(MAX_EVALS):
   
   # Randomly sample parameters for gbm
   params = {key: random.sample(value, 1)[0] for key, value in param_grid.items()}
   
   print(params)
   
   if params['boosting_type'] == 'goss':
       # Cannot subsample with goss
       params['subsample'] = 1.0
   else:
       # Subsample supported for gdbt and dart
       params['subsample'] = random.sample(subsample_dist, 1)[0]
       
       
   results_list = random_objective(params, i)
   
   # Add results to next row in dataframe
   random_results.loc[i, :] = results_list

Error message

---------------------------------------------------------------------------
LightGBMError                             Traceback (most recent call last)
<ipython-input-20-9bb1341efb38> in <module>
    17 
    18 
---> 19     results_list = random_objective(params, i)
    20 
    21     # Add results to next row in dataframe

<ipython-input-19-060262a0c5a6> in random_objective(params, iteration, n_folds)
     6 
     7     # Perform n_folds cross validation
----> 8     cv_results = lgb.cv(params, train_set, num_boost_round = 10000, nfold = n_folds, 
     9                         early_stopping_rounds = 100, metrics = 'auc', seed = 50)
    10     end = timer()

F:\anaconda\lib\site-packages\lightgbm\engine.py in cv(params, train_set, num_boost_round, folds, nfold, stratified, shuffle, metrics, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, fpreproc, verbose_eval, show_stdv, seed, callbacks, eval_train_metric, return_cvbooster)
   597         params['metric'] = metrics
   598 
--> 599     train_set._update_params(params) \
   600              ._set_predictor(predictor) \
   601              .set_feature_name(feature_name) \

F:\anaconda\lib\site-packages\lightgbm\basic.py in _update_params(self, params)
  1932                     self._free_handle()
  1933                 else:
-> 1934                     raise LightGBMError(_LIB.LGBM_GetLastError().decode('utf-8'))
  1935         return self
  1936 

LightGBMError: Cannot change bin_construct_sample_cnt after constructed Dataset handle.

opened by Z-zhh 2

Black-Box-Tuning - Black-Box Tuning for Language-Model-as-a-Service

Black-Box-Tuning Source code for paper "Black-Box Tuning for Language-Model-as-a

149 Jan 4, 2023

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.

H2O H2O is an in-memory platform for distributed, scalable machine learning. H2O uses familiar interfaces like R, Python, Scala, Java, JSON and the Fl

6.1k Jan 5, 2023

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

5 Sep 16, 2022

DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

What is DeepHyper? DeepHyper is a software package that uses learning, optimization, and parallel computing to automate the design and development of

214 Jan 8, 2023

Code and datasets for the paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction"

KnowPrompt Code and datasets for our paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction" Requireme

137 Dec 31, 2022

aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

Bayesian Methods for Hackers Using Python and PyMC The Bayesian method is the natural approach to inference, yet it is hidden from readers behind chap

25.1k Jan 2, 2023

A PyTorch-based open-source framework that provides methods for improving the weakly annotated data and allows researchers to efficiently develop and compare their own methods.

Knodle (Knowledge-supervised Deep Learning Framework) - a new framework for weak supervision with neural networks. It provides a modularization for se

93 Nov 6, 2022

Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Machine Learning From Scratch About Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The purpose

21.8k Jan 9, 2023

Deep Learning: Architectures & Methods Project: Deep Learning for Audio Super-Resolution

Deep Learning: Architectures & Methods Project: Deep Learning for Audio Super-Resolution Figure: Example visualization of the method and baseline as a

16 Dec 23, 2022