Hi all, I'm trying to benchmark auto-sklearn against the same data sets as TPOT on my local cluster (running Linux), but I can't get pyrfr to install when I pip install auto-sklearn.

Is there an earlier version of auto-sklearn that I can use to perform this benchmark?

Summary

Currently the show_models() function returns an str that has to be manually parsed with no way to access the models. I have changed it so that it returns a dictionary of models in ensemble and their information. This helps fix issue #1298 and the issues mentioned inside that thread.

What's changed

Using show_models() will return a dictionary where the key would be model_id. Each entry is a model dictionary which contains the following:

• model_id
• rank
• ensemble_weight
• data preprocessor
• balancing
• feature_preprocessor
• regressor or classifier (autosklearn wrapped model)
• sklearn model

Example

import sklearn.datasets
import sklearn.metrics

import autosklearn.regression
import matplotlib.pyplot as plt

X, y = sklearn.datasets.load_diabetes(return_X_y=True)

X_train, X_test, y_train, y_test = \
    sklearn.model_selection.train_test_split(X, y, random_state=1)

automl = autosklearn.regression.AutoSklearnRegressor(
    time_left_for_this_task=120,
    per_run_time_limit=30,
    tmp_folder='/tmp/autosklearn_regression_example_tmp',
)
automl.fit(X_train, y_train, dataset_name='diabetes')

ensemble_dict = automl.show_models()
print(ensemble_dict)

Output:

{
25: {'model_id': 25.0, 'rank': 1.0, 'ensemble_weight': 0.46, 'data_preprocessor': <autosklearn.pipeline.components.data_preprocessing.DataPreprocessorChoice object at 0x7ff2c06588d0>, 'feature_preprocessor': <autosklearn.pipeline.components.feature_preprocessing.FeaturePreprocessorChoice object at 0x7ff2c057bd50>, 'regressor': <autosklearn.pipeline.components.regression.RegressorChoice object at 0x7ff2c057ba90>, 'sklearn_model': SGDRegressor(alpha=0.0006517033225329654, epsilon=0.012150149892783745,
             eta0=0.016444224834275295, l1_ratio=1.7462342366289323e-09,
             loss='epsilon_insensitive', max_iter=16, penalty='elasticnet',
             power_t=0.21521743568582094, random_state=1,
             tol=0.002431731981071206, warm_start=True)}, 
6: {'model_id': 6.0, 'rank': 2.0, 'ensemble_weight': 0.32, 'data_preprocessor': <autosklearn.pipeline.components.data_preprocessing.DataPreprocessorChoice object at 0x7ff2c05b3f50>, 'feature_preprocessor': <autosklearn.pipeline.components.feature_preprocessing.FeaturePreprocessorChoice object at 0x7ff2c065c990>, 'regressor': <autosklearn.pipeline.components.regression.RegressorChoice object at 0x7ff2c057ba10>, 'sklearn_model': ARDRegression(alpha_1=0.0003701926442639788, alpha_2=2.2118001735899097e-07,
              copy_X=False, lambda_1=1.2037591637980971e-06,
              lambda_2=4.358378124977852e-09,
              threshold_lambda=1136.5286041327277, tol=0.021944240404849075)}, ....

when try the sample code:

import autosklearn.classification
import sklearn.model_selection
import sklearn.datasets
import sklearn.metrics
X, y = sklearn.datasets.load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = sklearn.model_selection.train_test_split(X, y, random_state=1)
automl = autosklearn.classification.AutoSklearnClassifier()
automl.fit(X_train, y_train)
y_hat = automl.predict(X_test)
print("Accuracy score", sklearn.metrics.accuracy_score(y_test, y_hat))

something wrong with the automl.fit line, and got "Segmentation fault", the python command console exit.

>automl.fit(X_train, y_train)
/home/work/.pyenv/versions/py3_env/lib/python3.6/site-packages/autosklearn/evaluation/train_evaluator.py:197: RuntimeWarning: Mean of empty slice
  Y_train_pred = np.nanmean(Y_train_pred_full, axis=0)
[WARNING] [2019-05-29 16:26:32,672:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-29 16:26:32,680:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
Segmentation fault
(py3_env) [work@*** ~]$ [WARNING] [2019-05-29 16:26:34,685:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!
[WARNING] [2019-05-29 16:26:36,689:EnsembleBuilder(1):d74860caaa557f473ce23908ff7ba369] No models better than random - using Dummy Score!

does anyone come across this problem?

I got the following error when installing pyrfr. What is installation directory here? My current directory?

(ml_env) ubuntu@ip-10-0-0-10:~/random_forest_run/build$ curl https://raw.githubusercontent.com/automl/auto-sklearn/master/requirements.txt | xargs -n 1 -L 1 pip install
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   230  100   230    0     0   2334      0 --:--:-- --:--:-- --:--:--  4893
Requirement already satisfied: unittest2 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: six>=1.4 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from unittest2)
Requirement already satisfied: argparse in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from unittest2)
Requirement already satisfied: traceback2 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from unittest2)
Requirement already satisfied: linecache2 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from traceback2->unittest2)
Requirement already satisfied: setuptools in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg
Requirement already satisfied: nose in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: six in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: Cython in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: numpy<1.12,>=1.9.0 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: scipy>=0.14.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: numpy>=1.8.2 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from scipy>=0.14.1)
Requirement already satisfied: scikit-learn==0.17.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: lockfile in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: joblib in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: psutil in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: pyyaml in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: ConfigArgParse in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: liac-arff in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: pandas in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: python-dateutil>=2 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: pytz>=2011k in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: numpy>=1.7.0 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from pandas)
Requirement already satisfied: six>=1.5 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from python-dateutil>=2->pandas)
Requirement already satisfied: xgboost==0.4a30 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: scipy in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from xgboost==0.4a30)
Requirement already satisfied: numpy in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from xgboost==0.4a30)
Requirement already satisfied: ConfigSpace<0.4,>=0.3.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: typing in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from ConfigSpace<0.4,>=0.3.1)
Requirement already satisfied: numpy in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from ConfigSpace<0.4,>=0.3.1)
Requirement already satisfied: argparse in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from ConfigSpace<0.4,>=0.3.1)
Requirement already satisfied: pyparsing in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from ConfigSpace<0.4,>=0.3.1)
Requirement already satisfied: pynisher>=0.4 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: docutils>=0.3 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from pynisher>=0.4)
Requirement already satisfied: psutil in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from pynisher>=0.4)
Requirement already satisfied: setuptools in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg (from pynisher>=0.4)

Collecting pyrfr
  Using cached pyrfr-0.4.0.tar.gz
Building wheels for collected packages: pyrfr
  Running setup.py bdist_wheel for pyrfr ... error
  Complete output from command /home/ubuntu/anaconda3/envs/ml_env/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-axcsozeu/pyrfr/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpmdz8gclmpip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/pyrfr
  copying pyrfr/__init__.py -> build/lib.linux-x86_64-3.6/pyrfr
  running build_ext
  building '_regression' extension
  swigging pyrfr/regression.i to pyrfr/regression_wrap.cpp
  swig -python -c++ -I${CMAKE_SOURCE_DIR}/include -I./include -o pyrfr/regression_wrap.cpp pyrfr/regression.i
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/pyrfr
  gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I${CMAKE_SOURCE_DIR}/include -I./include -I/home/ubuntu/anaconda3/envs/ml_env/include/python3.6m -c pyrfr/regression_wrap.cpp -o build/temp.linux-x86_64-3.6/pyrfr/regression_wrap.o -O2 -std=c++11
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  g++ -pthread -shared -L/home/ubuntu/anaconda3/envs/ml_env/lib -Wl,-rpath=/home/ubuntu/anaconda3/envs/ml_env/lib,--no-as-needed build/temp.linux-x86_64-3.6/pyrfr/regression_wrap.o -L/home/ubuntu/anaconda3/envs/ml_env/lib -lpython3.6m -o build/lib.linux-x86_64-3.6/_regression.cpython-36m-x86_64-linux-gnu.so
  building '_util' extension
  swigging pyrfr/util.i to pyrfr/util_wrap.cpp
  swig -python -c++ -I${CMAKE_SOURCE_DIR}/include -I./include -o pyrfr/util_wrap.cpp pyrfr/util.i
  gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I${CMAKE_SOURCE_DIR}/include -I./include -I/home/ubuntu/anaconda3/envs/ml_env/include/python3.6m -c pyrfr/util_wrap.cpp -o build/temp.linux-x86_64-3.6/pyrfr/util_wrap.o -O2 -std=c++11
  cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
  g++ -pthread -shared -L/home/ubuntu/anaconda3/envs/ml_env/lib -Wl,-rpath=/home/ubuntu/anaconda3/envs/ml_env/lib,--no-as-needed build/temp.linux-x86_64-3.6/pyrfr/util_wrap.o -L/home/ubuntu/anaconda3/envs/ml_env/lib -lpython3.6m -o build/lib.linux-x86_64-3.6/_util.cpython-36m-x86_64-linux-gnu.so
  installing to build/bdist.linux-x86_64/wheel
  running install
  running install_lib
  creating build/bdist.linux-x86_64
  creating build/bdist.linux-x86_64/wheel
  copying build/lib.linux-x86_64-3.6/_regression.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel
  creating build/bdist.linux-x86_64/wheel/pyrfr
  copying build/lib.linux-x86_64-3.6/pyrfr/__init__.py -> build/bdist.linux-x86_64/wheel/pyrfr
  copying build/lib.linux-x86_64-3.6/_util.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel
  running install_egg_info
  running egg_info
  writing pyrfr.egg-info/PKG-INFO
  writing dependency_links to pyrfr.egg-info/dependency_links.txt
  writing top-level names to pyrfr.egg-info/top_level.txt
  warning: manifest_maker: standard file '-c' not found
  
  reading manifest file 'pyrfr.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.pxd' under directory 'pyrfr'
  warning: no files found matching '*.pyx' under directory 'pyrfr'
  warning: manifest_maker: MANIFEST.in, line 11: unknown action 'CMakeList.txt'
  
  writing manifest file 'pyrfr.egg-info/SOURCES.txt'
  Copying pyrfr.egg-info to build/bdist.linux-x86_64/wheel/pyrfr-0.4.0-py3.6.egg-info
  running install_scripts
  Checking .pth file support in build/bdist.linux-x86_64/wheel/
  /home/ubuntu/anaconda3/envs/ml_env/bin/python -E -c pass
  TEST FAILED: build/bdist.linux-x86_64/wheel/ does NOT support .pth files
  error: bad install directory or PYTHONPATH
  creating build/bdist.linux-x86_64/wheel/pyrfr
  copying build/lib.linux-x86_64-3.6/pyrfr/__init__.py -> build/bdist.linux-x86_64/wheel/pyrfr
  copying build/lib.linux-x86_64-3.6/_util.cpython-36m-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel
  running install_egg_info
  running egg_info
  writing pyrfr.egg-info/PKG-INFO
  writing dependency_links to pyrfr.egg-info/dependency_links.txt
  writing top-level names to pyrfr.egg-info/top_level.txt
  warning: manifest_maker: standard file '-c' not found 

  reading manifest file 'pyrfr.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'


      You are attempting to install a package to a directory that is not
      on PYTHONPATH and which Python does not read ".pth" files from.  The
      installation directory you specified (via --install-dir, --prefix, or
      the distutils default setting) was:
  
      build/bdist.linux-x86_64/wheel/


  and your PYTHONPATH environment variable currently contains:
  
      ''
  
  Here are some of your options for correcting the problem:
  
  * You can choose a different installation directory, i.e., one that is
    on PYTHONPATH or supports .pth files
  
  * You can add the installation directory to the PYTHONPATH environment
    variable.  (It must then also be on PYTHONPATH whenever you run
    Python and want to use the package(s) you are installing.)
  
  * You can set up the installation directory to support ".pth" files by
    using one of the approaches described here:
  
    https://setuptools.readthedocs.io/en/latest/easy_install.html#custom-installation-locations
  
  
  Please make the appropriate changes for your system and try again.
  
  ----------------------------------------
  Failed building wheel for pyrfr
  Running setup.py clean for pyrfr

Failed to build pyrfr
Installing collected packages: pyrfr
  Running setup.py install for pyrfr ... done
  Could not find .egg-info directory in install record for pyrfr from https://pypi.python.org/packages/21/4c/58533c51ab301f61d3521dc4cd29ba8145eed8f11b84f70aba9fd28f6aca/pyrfr-0.4.0.tar.gz#md5=70ccd2527bd85c18b8a65c7498d5e0de
Successfully installed pyrfr-0.4.0
Requirement already satisfied: smac==0.3.0 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages
Requirement already satisfied: numpy>=1.7.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: scipy>=0.18.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: scikit-learn in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: pyrfr in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: ConfigSpace>=0.3.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: pynisher>=0.4.1 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: psutil in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: typing in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: setuptools in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg (from smac==0.3.0)
Requirement already satisfied: Cython in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: six in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from smac==0.3.0)
Requirement already satisfied: pyparsing in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from ConfigSpace>=0.3.1->smac==0.3.0)
Requirement already satisfied: argparse in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from ConfigSpace>=0.3.1->smac==0.3.0)
Requirement already satisfied: docutils>=0.3 in /home/ubuntu/anaconda3/envs/ml_env/lib/python3.6/site-packages (from pynisher>=0.4.1->smac==0.3.0)

New approach for #900 : Data preprocessing step of AutoML system will have two options:

• use feature_type split method (existing implementation)
• disable step by selecting no_preprocessing

Introduced new parameters:

• include_data_preprocessors
• exclude_data_preprocessors

If none of the parameter is set by user, by default, it will add 'no_preprocessing' to step exclude_data_preprocessors and will only use existing FeatureTypeSplit component.

Trying to install auto-sklearn and more specifically pyrfr in an anaconda environment failed due to a wrong version of GCC. This happened even though - and as it turns out especially because - I followed the instruction to conda install gcc swig given in the installation guide.

Building pyrfr failed for me with GCC 4.8.5 (which was installed through conda) and worked with the system GCC 7.1.1. The pyrfr readme hints on using GCC 5.4 or 6.2. Sadly I can't identify the minimum working version of GCC to build pyrfr right now but would recommend adding an according hint to the installation instructions on the website.

Dear auto-sklearn team, I have just learned about this project and am very excited to try to include it into my modelling flow!

It seems the command line option names for autosklearn are not the same as what AutoSklearnClassifier() constructor would accept. So I have kind of reverse engineered a few but I still can not figure out whether it is possible to specify task_type for example. task_type="binary.classification" is rejected by the AutoSklearnClassifier() constructor.

I understand this is a very young project and is actively worked on. Will be happy to supply you with the feedback from the fields as I am actively running modelling experiments on various datasets available at my company. Currently I am successfully using scikit learn with SGD Classifier for one. Is there a better way to reconnect with you in a forum or a chat somewhere to ask questions or give feedback?

Currently, the easiest way for a user to see the pipelines included in the ensemble is through estimator.show_models() which just returns a str which needs to be manually parsed and looked through. There could definitely be a nicer format to view any such pipeline and provide easy access.

Greetings,

I am trying to run autosklearn 0.6.0 on an HPC infrastructure. I am using the kr-vs-kp dataset (https://www.openml.org/d/3) and I have made all the values of the dataset numeric. The error I get is:

Unhandled exception in thread started by libgcc_s.so.1 must be installed for pthread_cancel to work Traceback (most recent call last): File "main_script.py", line 41, in <module> autosklearn_script.autosklearn_func(train_data,train_target,test_data,test_target,dataset_info_dict,task,time_threshold) File "/users/pr004/ixanthos/test_files/autosklearn_script.py", line 31, in autosklearn_func automl.fit(train_data, train_target, metric=autosklearn.metrics.roc_auc) File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/estimators.py", line 664, in fit dataset_name=dataset_name, File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/estimators.py", line 337, in fit self._automl[0].fit(**kwargs) File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 996, in fit load_models=load_models, File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 208, in fit only_return_configuration_space=only_return_configuration_space, File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 384, in _fit num_run = self._do_dummy_prediction(datamanager, num_run) File "/apps/applications/pytorch/1.3.1/lib/python3.6/site-packages/autosklearn/automl.py", line 313, in _do_dummy_prediction raise ValueError("Dummy prediction failed: %s " % str(additional_info)) ValueError: Dummy prediction failed: {'error': 'Result queue is empty', 'configuration_origin': 'DUMMY'}

Any ideas?

Thank you in advance.

I have a data set which is more than 100k records. When I try to fit into AutoSklearnRegressor, it always thrown an warning. It seems causing that I cannot get a expected output.

However, if number of records is small enough (says less than 20k), it can execute without any warming/ error. May you advise this situation? I am using 0.2 version

Sample code

import autosklearn.regression
import numpy as np

x = np.random.randint(2, size=(250000,100))
y = np.random.randint(2, size=(250000,1))


feature_types = (['numerical'] * 100)
automl = autosklearn.regression.AutoSklearnRegressor(
    time_left_for_this_task=120, per_run_time_limit=30
)
automl.fit(x, y, dataset_name='boston', feat_type=feature_types)

The exception is

WARNING] [2017-10-26 11:21:28,580:AutoMLSMBO(1)::boston] Could not find meta-data directory /home/anaconda3/lib/python3.5/site-packages/autosklearn/metalearning/files/r2_regression_dense

/home/anaconda3/lib/python3.5/site-packages/autosklearn/smbo.py:737: RuntimeWarning: invalid value encountered in true_divide
  (1. - dataset_minimum))
/home/anaconda3/lib/python3.5/site-packages/scipy/stats/_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in greater
  return (self.a < x) & (x < self.b)
/home/anaconda3/lib/python3.5/site-packages/scipy/stats/_distn_infrastructure.py:879: RuntimeWarning: invalid value encountered in less
  return (self.a < x) & (x < self.b)
/home/anaconda3/lib/python3.5/site-packages/scipy/stats/_distn_infrastructure.py:1735: RuntimeWarning: invalid value encountered in greater_equal
  cond2 = (x >= self.b) & cond0
/home/anaconda3/lib/python3.5/site-packages/scipy/stats/_distn_infrastructure.py:876: RuntimeWarning: invalid value encountered in greater_equal

In a text classification task the SGDClassifier needs just a few minutes to get to the same result as auto-sklearn that I let running for 20 hours. Lesser time budget for auto-sklearn resulted in absolute or relative failure in prediction.

I wonder if there is a strategy to try the fastest algorithms first and if time is up use their results at least?

Another question is about the recommended per_run_time_limit value. Is there a rule of thumb choosing it?

SGDClassifier Precision: 0.20 Test FR Recall: 0.53 F1: 0.29 Auto-sklearn Precision: 0.28 Recall: 0.31 F1: 0.29 classifier.fit(X_train, y_train, metric='f1_metric') AutoSklearnClassifier( time_left_for_this_task=72000, per_run_time_limit=19000, ml_memory_limit=10000)

Bumps actions/stale from 6 to 7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

How can one get back the values of categorical variables from a fitted model? Let's say that I have a model where one of the features is a categorical variable, is there a way to get back the values of that variable that were observed during training?

My use case is I have a multi-model system that I am building (multiple auto-sklearn models, not ensemble) and I want to implement some logic for deciding which model to use depending on if a certain categorical value was observed during training. In scikit-learn this could be easily accessed from the categories_ attribute of a OneHotEncoder, but given the complex nature of auto-sklearn classes and use of ensembles I'm not sure where to begin looking.

Alternatively, one could set an encoder to error on unknown values, and build logic around catching these errors. This doesn't work for auto-sklearn either because the "missing" category is always created, so models will always successfully predict on missing values without any sign that it was on an unknown categorical value.

Any help here would be appreciated.

Running version 0.14.7

Hi, the method AutoSklearnClassifier().show_models() displays the models found in the ensemble. I wonder if it is possible to know exactly which data or feature preprocessing steps have been done before training the model. The method show_models only gives only the object:

{'model_id': 2, 
'rank': 1, 
'cost': 0.04255319148936165, 
'ensemble_weight': 0.04, 
'data_preprocessor': <autosklearn.pipeline.components.data_preprocessing.DataPreprocessorChoice object at 0x7f704fb2dee0>, 
'balancing': Balancing(random_state=1), 
'feature_preprocessor': <autosklearn.pipeline.components.feature_preprocessing.FeaturePreprocessorChoice object at 0x7f70a7e4e7f0>,
'classifier': <autosklearn.pipeline.components.classification.ClassifierChoice object at 0x7f70a7e4e1c0>, 
'sklearn_classifier': RandomForestClassifier(max_features=5, n_estimators=512, n_jobs=1,
                       random_state=1, warm_start=True)}

and it is not clear to know which steps they are. Is it possible to get the preprocessing steps in such a case?

Many thanks!

Bumps actions/checkout from 3.1.0 to 3.2.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hello Auto-sklearn team,

Describe the bug

AutoSklearn2Classifier with y-variable of type multilabel-indicator erroring with component sgd. Demonstrated below with example data.

To Reproduce

Steps to reproduce the behavior:

1. Using the sample multilabel classification data from https://automl.github.io/auto-sklearn/master/examples/20_basic/example_multilabel_classification.html
2. autosklearn.classification.AutoSklearnClassifier is working fine with y-variable of type multilabel-indicator
3. Importing AutoSklearn2Classifier (from autosklearn.experimental.askl2 import AutoSklearn2Classifier)
4. AutoSklearn2Classifier works fine with y-variable of type binary (e.g. using just the first column of the y-matrix)
5. but AutoSklearn2Classifier with y-variable of type multilabel-indicator is producing the error below

Actual behavior, stacktrace or logfile

Text-based error: ValueError: The provided component 'sgd' for the key 'classifier' in the 'include' argument is not valid. The supported components for the step 'classifier' for this task are ['bernoulli_nb', 'decision_tree', 'extra_trees', 'gaussian_nb', 'k_nearest_neighbors', 'lda', 'liblinear_svc', 'mlp', 'multinomial_nb', 'passive_aggressive', 'qda', 'random_forest']

Environment and installation:

• Google Colab Jupyter Notebook
• Python 3.8.16
• Auto-sklearn version 0.15.0

Best, Anthony