AutoML Survey
An (in-progress) AutoML survey focusing on practical systems.
This project is a community effort in constructing and maintaining an up-to-date beginner-friendly introduction to AutoML, focusing on practical systems. AutoML is a big field, and continues to grow daily. Hence, we cannot hope to provide a comprehensive description of every interesting idea or approach available. Thus, we decided to focus on practical AutoML systems, and spread outwards from there into the methodologies and theoretical concepts that power these systems. Our intuition is that, even though there are a lot of interesting ideas still in research stage, the most mature and battle-tested concepts are those that have been succesfully applied to construct practical AutoML systems.
To this end, we are building a database of qualitative criteria for all AutoML systems we've heard of. We define an AutoML system as a software project that can be used by non-experts in machine learning to build effective ML pipelines on at least some common domains and tasks. It doesn't matter if its open-source and/or commercial, a library or an application with a GUI, or a cloud service. What matters is that it is intended to be used in practice, as opposed to, say, a reference implementation of a novel AutoML strategy in a Jupyter Notebook.
Features of an AutoML system
For each of them we are creating a system card that describes, in our opinion, the most relevant features of the system, both from the scientific and the engineering points of view. To describe an AutoML system, we use a YAML-based definition. Most of the features are self-explanatory.
💡 Check data/systems/_template.yml for a starting template.
Basic information
Characteristics about the basic information of the system as a software product.
name
(str
): Name of the system.description
(str
): A short (2-4 sentences) description of the sytem.website
(str
): The URL of the main website or documentation.open_source
(bool
): Whether the system is open-source.institutions
(list[str]
): List of businesses or academic institutions that directly support the development of the system, and/or hold intellectual property over it.repository
(str
): If it's open-source, link of a public source code repository, otherwisenull
.license
(str
): If it's open-source, a license key, otherwisenull
.references
(list[str]
): List of links to relevant papers, preferably DOIs or other universal handlers, but can also be links to arxiv.org or other repositories sorted by most relevant papers, not date.
User interfaces
Characteristics describing how the users interact with the system.
cli
(bool
): Whether the system has a command line interfacegui
(bool
): Whether the system has a graphic user interfacehttp
(bool
): Whether the system can used from an HTTP RESTful APIlibrary
(bool
): Whether the system can be linked as a code libraryprogramming_languages
(list[str]
): List of programming languages in which the system can be used, i.e., it is either natively coded in that language or there are maintained bindings (as opposed to using language X's standard way to call code from language Y).
Domains
Characteristics describing the domains in which the system can be applied, which roughly correspond to the types of input data that the system can handle.
domains
(list[str]
): Domains in which the system can be deployed. Valid values are:images
nlp
tabular
time_series
multi_domain
(bool
): Whether the system supports multiple domains for a single workflow, e.g., by allowing multiple inputs of different types simultaneously
Techniques
Characteristics describing the actual models and techniques used in the system, and the underlying ML libraries where those techniques are implemented.
techniques
(list[str]
): List of high-level techniques that are available in the systems, broadly classified according to model families. Valid values are:linear_models
trees
bayesian
kernel_machines
graphical_models
mlp
cnn
rnn
pretrained
ensembles
ad_hoc
(📝 indicates non-ML algorithms, e.g., tokenizers)
distillation
(bool
): Whether the system supports model distillationml_libraries
(list[str]
): List of ML libraries that support the system, i.e., where the techniques are actually implemented, if any. Valid values are lists of strings. Some examples are:scikit-learn
keras
pytorch
nltk
spacy
transformers
Tasks
Characteristics describing the types of tasks, or problems, in which the system can be applied, which roughly correspond to the types of outputs supported.
tasks
(list[str]
): List of high-level tasks the system can perform automatically. Valid values are:classification
structured_prediction
structured_generation
unstructured_generation
regression
clustering
imputation
segmentation
feature_preprocessing
feature_selection
data_augmentation
dimensionality_reduction
data_preprocessing
(📝 domain-agonostic data preprocessing such as normalization and scaling)domain_preprocessing
(📝 refers to domain-specific preprocessing, e.g., stemming)
multi_task
: Whether the system supports multiple tasks in a single workflow, e.g., by allowing multiple output heads from the same neural network
Search strategies
Characteristics describing the optimizaction/search strategies used for model search and/or hyperparameter tunning.
search_strategies
(list[str]
): List of high-level search strategies that are available in the system. Valid values are:random
evolutionary
gradient_descent
hill_climbing
bayesian
grid
hyperband
reinforcement_learning
constructive
monte_carlo
meta_learning
(list[str]
): If the system includes meta-learning, list of broadly classified techniques used. Valid values are:portfolio
warm_start
Search space
Characteristics describing the search space, the types of hyperparameters that can be optimized, and the types of ML pipelines that can be represented in this space.
search_space
: High-level characteristics of the hyperparameter search space.hierarchical
(bool
): If there are hyperparameters that only make sense conditioned to others.probabilistic
(bool
): If the hyperparameter space has an associated probabilistic model.differentiable
(bool
): If the hyperameter space can be used for gradient descent.automatic
(bool
): If the global structure of the hyperparameter space is inferred automatically from, e.g., type annotations or model's documentation, as opposed to explicitely defined by the developers or the user.hyperparameters
(list[str]
): Types of hyperparameters that can be optimized. Valid values are:continuous
discrete
categorical
conditional
pipelines
: Types of pipelines that can be discovered by the AutoML process. Each of the following keys is boolean.single
(bool
): A single estimator (or model in general)fixed
(bool
): A fixed pipeline with several, but predefined, stepslinear
(bool
): A variable-length pipeline where each step feeds on the immediately previous outputgraph
(bool
): An arbitrarily graph-shaped pipeline where each step can feed on any of the previous outputs
robust
(bool
): Whether the seach space contains potentially invalid pipelines that are only discovered when evaluated, e.g., allowing a dense-only estimator to precede a sparse transformer.
Software architecture
Other characteristics describing general features of the system as a software product.
extensible
(bool
): Whether the system is designed to be extensible, in the sense that a user can add a single new type of model, or search algorithm, etc., in an easy manner, not needing to modify any part of the system/accessible
(bool
): Whether the models obtained from the AutoML process can be freely inspected by the user up to the level of individual parameters (e.g., neural network weights).portable
(bool
): Whether the models obtained can be exported out of the AutoML system, either on a standard format, or, at least, in a format native of the underlying ML library,such that they can be deployed on another platform without depending on the AutoML system itself.computational_resources
: Computational resources that, if available, can be leveraged by the system.gpu
(bool
): Whether the system supports GPUs.tpu
(bool
): Whether the system supports TPUs.cluster
(bool
): Whether the system supports cluster-based parallelism.
How to contribute
If you are an author or a user of any practical AutoML system that roughly fits the previous criteria, we would love to have your contributions. You can add new systems, add information for existing ones, or fix anything that is incorrect.
To do this, either create a new or modify an existing file in data/systems. Once done, you can run make check
to ensure that the modifications are valid with respect to the schema defined in scripts/models.py. If you need to add new fields, or new values to any of the enumerations defined, feel free to modify the corresponding schema as well (and modify both data/systems/_template.yml and this README).
Once validated, you can open a pull request.
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.