OMLT: Optimization and Machine Learning Toolkit

Discussed in https://github.com/cog-imperial/OMLT/discussions/78

In some cases, it is possible to compute an upper bound on zhat that is strictly negative. In such cases, setting the upper bound on z to be equal to the upper bound on zhat is incorrect and produces an infeasible model. Even if zhat is strictly negative, z can still be 0.

I am confused about what format of input bounds should be passed with an onnx model to omlt. Trying a list or a dictionary:

lb = np.maximum(0, image - epsilon_infty)
ub = np.minimum(1, image + epsilon_infty)
input_bounds = [(float(l), float(u)) for l, u in zip(lb[0], ub[0])]

or

input_bounds = {}
for i in range(28*28):
    input_bounds[i] = (float(lb[0][i]), float(ub[0][i])) 

Attempting to create the omlt model with either:

write_onnx_model_with_bounds(f.name, None, input_bounds)
network_definition = load_onnx_neural_network_with_bounds(f.name)
formulation = NeuralNetworkFormulation(network_definition)
m = pyo.ConcreteModel()
m.nn = OmltBlock()
m.nn.build_formulation(formulation) 

Results in: ValueError: Variable 'bounds' keyword must be a tuple or function

This is the code I mentioned during our last meeting that changes how networks are defined. This is still a WIP but now I think I'm ready for feedback.

A couple of things to notice:

• I store layers in a graph so that we can visit them in topological order
• Layers inputs and outputs are multi indexed
• Sometimes we need to "reshape" outputs to fit another node input, for this reason I added a input_index_transformer (now I realise it should be input_index_mapper). This is useful with CNN because the output of the conv layer needs to be reshaped for the dense layers. The input index mapper makes sure we don't need to introduce extra variables.
• Since z and zhat are now multi indexed, I need to scope them with a block for each layer. This has the added benefit that now models are much easier to debug.

I didn't have time to check the tests so the best way to run an example is to run the _test.py script I included (and will delete for the final PR).

This code adds support for formulating max pooling layers into OMLT, using the formulation in Anderson et al. (2020). Changes are made in the io directory to parse ONNX max pool nodes; in neuralnet/layer.py to add a layer class for arbitrary pooling operations; in neuralnet/layers/full_space.py to add the Anderson formulation of max pooling; and in the relevant test programs.

Legal Acknowledgement
By contributing to this software project, I agree my contributions are submitted under the BSD license. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

I added comments and math formulations for all functions used to generate the gradient-boosted trees optimization model. Is there an easy way to test how these comments will look like in the docs before merging?

This commit makes the relu network in neural_network_formulations.ipynb smaller. It also uses a smaller convolutional network in mnist_example_convolutionsal.ipynb. Both of these should find solutions with cbc in a few seconds. If Cbc still hangs, it may be a separate issue.

This PR should hopefully get the notebook tests to pass. I updated all of the neural network formulations to use the final names we picked. I also had to make some adjustments to build_network.ipynb.

This PR uses the pyomo attempt_import to defer the import of dependent packages until they are used (or to check if the package exists). See src/omlt/dependencies.py. This should address issue #96.

The function attempt_import offers a lot of extra functionality - not sure whether we need to use all of it.

E.g., to check if keras (or onnx) is available, use: from omlt.dependencies import keras_available

E.g., to import keras (or onnx) use: from omlt.dependencies import keras

If you then try to use keras and it is not installed, Pyomo's attempt import will throw an exception.

Legal Acknowledgement
By contributing to this software project, I agree my contributions are submitted under the BSD license. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

This PR contains minor changes to configuration files to fix linting and notebook test failures that appeared due to dependency updates.

Legal Acknowledgement
By contributing to this software project, I agree my contributions are submitted under the BSD license. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Legal Acknowledgement
By contributing to this software project, I agree my contributions are submitted under the BSD license. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

This PR makes many changes because it updates all of the code formatting. The main changes are:

• All src and test files updated using isort and black linting operations.
• setup.cfg requires python >= 3.7
• A doc string for OMLT in the outer __init__.py file
• changes to tox.ini such that github actions SHOULD execute linting checks with flake8 and black.

Note that most linting is ignored. A separate issue will be opened to address flake8 stylistic requirements.

Hi, I tried making a simple notebook that optimizes the output of a single-input single-output sklearn gradient boosting model. The output-varieble seems to be ok, but the input variable (decision variable) is not. I've tested it with IPOPT and CBC. I am running this on Linux in Github Codespaces.

It's hard to boil down my code to an MRE, so I've attached the full notebook. It is not that big.

OMLT with sklearn.zip

I believe that there is a missing required dependency in the OMLT 1.0 release: onnx is listed as a "testing" dependency, but is required in order to import anything from omlt.io.

Steps to reproduce:

% pip install --user omlt
Collecting omlt
  Downloading omlt-1.0-py2.py3-none-any.whl (29 kB)
Requirement already satisfied: importlib-metadata in [...] (from omlt) (4.8.1)
Requirement already satisfied: numpy in [...] (from omlt) (1.21.2)
Requirement already satisfied: pyomo in [...] (from omlt) (6.4.3.dev0)
Requirement already satisfied: networkx in [...] (from omlt) (2.6.3)
Requirement already satisfied: typing-extensions>=3.6.4 in [...] (from importlib-metadata->omlt) (3.10.0.2)
Requirement already satisfied: zipp>=0.5 in [...] (from importlib-metadata->omlt) (3.4.0)
Requirement already satisfied: ply in [...] (from pyomo->omlt) (3.11)
Installing collected packages: omlt
Successfully installed omlt-1.0

% python
Python 3.7.12 (default, Nov 10 2021, 15:38:43)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from omlt.io.keras_reader import load_keras_sequential
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "[...]/site-packages/omlt/io/__init__.py", line 1, in <module>
    from omlt.io.onnx import load_onnx_neural_network, write_onnx_model_with_bounds, load_onnx_neural_network_with_bounds
  File "[...]/site-packages/omlt/io/onnx.py", line 4, in <module>
    import onnx
ModuleNotFoundError: No module named 'onnx'

Possible solutions:

• Don't import anything from omlt/io/__init__.py
• Leverage a delayed import mechanism in omlt/io/onnx.py (e.g., something like pyomo.common.dependencies.attempt_import())
• Add onnx as a required dependency

@fracek, we need documentation on write_input_bounds and load_input_bounds (both in https://github.com/cog-imperial/OMLT/blob/main/src/omlt/io/input_bounds.py).

I need to clarify. Does the following make sense and (if it doesn't) would you please suggest an alternative?

For write_input_bounds:

""" Write the specified input bounds to the given file. This input implicitly assumes that all inputs are defined (no indices missing) and all indices are bounded. """

` """

Parameters
----------
input_bounds_filename: file
input_bounds: dict or list
"""`

For load_input_bounds:

""" Read the input bounds from the given file. """

` """

Parameters
----------
input_bounds_filename: file
The file should be a list of tuples with a key (index of the input), lower bound (real number), and upper bound (real number).
"""`

This code adds an interface for scikit learn MLPRegressor() objects via the sklearn2onnx library and support for scikit offset scaling objects. All scikit scaling objects which do linear scaling are supported including StandardScaler, MaxAbsScaler, MinMaxScaler, and RobustScaler.

Some changes were required to the onnx_parser to handle the different conventions created by sklearn2onnx such as the biases being stored as (1, n) matrices instead of (n,) vectors.

Additionally, since sklearn_reader.py imports the omlt onnx reader it was renamed to onnx_reader.py instead of onnx.py as this leads to a circular import issue since the naming omlt.onnx.py conflicts with the global onnx library.

There is some design work to do with the activation functions and constraints to gain more code reuse and expand the list of supported activations.

This issue originally came from a PR comment by @jalving.

add_constraint=True is not used here

net_block and net are also not used in activation methods. we should discuss whether we need them. they might be there for consistency on the calling side.

Originally posted by @jalving in https://github.com/cog-imperial/OMLT/pull/24#discussion_r784112237

onnx_parser.py (and possibly keras_reader.py) does not contain checks for unsupported operation types:

=== from PR comment

will this use a linear activation if maybe_node.op_type is not in _ACTIVATION_OP_TYPES? this could lead to an incorrect neural network

Originally posted by @jalving in https://github.com/cog-imperial/OMLT/pull/24#r784097245