PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation

SUN Group @ UMN

Last update: Nov 16, 2022

Related tags

Deep Learning data-science machine-learning computer-vision deep-learning numerical-optimization mathematical-software

Overview

PyGRANSO

PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation

Please check https://ncvx.org/PyGRANSO for detailed instructions (introduction, installation, settings, examples...).

PyGRANSO is AGPL v3.0 licensed, but it also contains a small portion of GPL code. Please see the LICENSE_INFO folder for more information.

Brief Introduction

Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially deep learning) applications. To greatly lower the technical barrier, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation, GPU acceleration, tensor input, and support for new QP solvers. NCVX is built on freely available and widely used open-source frameworks, and as a highlight, can solve general constrained deep learning problems, the first of its kind.

Installation

NOTE: The installation process is tested on Ubuntu 20.04

Installing PyGRANSO is simple. Here is a step-by-step instruction:

Install Python >= 3.9
Get the most recent PyGRANSO package (including examples and requirements file):
```
 git clone https://github.com/sun-umn/PyGRANSO.git
 cd PyGRANSO
```
Install PyGRANSO solver from PyPI:
```
pip install pygranso==1.0.0.dev2
```

Install Dependencies from PyPI:

OS: Linux OR Windows; Compute Platform: CUDA:

 pip install -r requirements.txt -f https://download.pytorch.org/whl/cu111/torch_stable.html

OS: Linux; Compute Platform: CPU:

 pip install -r requirements_linux_cpu.txt -f https://download.pytorch.org/whl/cpu/torch_stable.html

OS: Mac OR Windows; Compute Platform: CPU:

 pip install -r requirements_cpu.txt

(CUDA) Run test to make sure the dependency installation is correct:
```
 python test_cuda.py
```
(CPU) Run test to make sure the dependency installation is correct:
```
 python test_cpu.py
```
Check the example folder in the source code or example section on the documentation website to get started.

Dependencies

Python-3.9.7

numpy-1.20.3

scipy-1.7.1

pytorch-1.9.0

osqp-0.6.2

Jupyter Notebook-6.4.5

Change Logs

Please check CHANGELOG.md in the main folder.

Notes on Documentation

PyGRANSO is a PyTorch-enabled port of GRANSO with auto-differentiation, and some of its documentation uses MATLAB conventions. In the PyGRANSO source code docstrings, please note that:

struct refers to pygransoStruct, which is a simple class that users need to use for specifying their problems and options.
Vector and matrix refer to PyTorch tensor with (n,1) and (m,n) dimension, respectively.

Acknowledgements

We would like to thank Frank E. Curtis and Michael L. Overton for their involvement in creating the BFGS-SQP algorithm that is implemented in the software package GRANSO. This work was supported by UMII Seed Grant Program and NSF CMMI 2038403.

Citation

If you publish work that uses or refers to PyGRANSO, please cite the following two papers, which respectively introduced PyGRANSO and GRANSO:

[1] Buyun Liang, Tim Mitchell, and Ju Sun, NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning, arXiv preprint arXiv:2111.13984 (2021). Available at https://arxiv.org/abs/2111.13984

[2] Frank E. Curtis, Tim Mitchell, and Michael L. Overton, A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles, Optimization Methods and Software, 32(1):148-181, 2017. Available at https://dx.doi.org/10.1080/10556788.2016.1208749

BibTex:

@article{liang2021ncvx,
    title={{NCVX}: {A} User-Friendly and Scalable Package for Nonconvex 
    Optimization in Machine Learning}, 
    author={Buyun Liang, Tim Mitchell, and Ju Sun},
    year={2021},
    eprint={2111.13984},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

@article{curtis2017bfgssqp,
    title={A {BFGS-SQP} method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles},
    author={Frank E. Curtis, Tim Mitchell, and Michael L. Overton},
    journal={Optimization Methods and Software},
    volume={32},
    number={1},
    pages={148--181},
    year={2017},
    publisher={Taylor \& Francis}
}

Contact

For questions or bug reports, please either:

raise issues in the PyGRANSO repository (https://github.com/sun-umn/PyGRANSO/) or
send an email to:
- Buyun Liang (liang664 an_at_symbol umn a_dot_symbol edu)
- Tim Mitchell (tim an_at_symbol timmitchell a_dot_symbol com)
- Ju Sun (jusun an_at_symbol umn a_dot_symbol edu)

Thanks to other contributors and bug reporters:

Chen Jiang: Tested perceptual attack example (ex6). Tested PyGRANSO on Win10. Debug updatePenaltyParameter function.

Comments

Eigenvalue opt example
Need to change:

[x] \lambda to \Lambda (lowercase to uppercase) (Typically, \Lambda uppercase denotes the spectrum, while \lambda lowercase denotes an eigenvalue.)

[x] "$\lambda(\cdot)$ is the eigenvalue of $\cdot$," to "$\Lambda(\cdot)$ is the spectrum of a square matrix,"

[x] "$\alpha$ is the maximum real part of eigenvalues of $M$." to "$\alpha(\cdot)$ is the spectral abscissa of a square matrix, i.e., the maximum real part of its eigenvalues."

[x] For imaginary part, \Im, in papers, I generally redefine this from the standard symbol to just "Im", e.g., "Im(A+BXC)", as I think this is usually much clearer. But this is a debatable suggestion.
opened by t-mitchell 13
Posting announcement for PyGRANSO

Machine Learning News: https://groups.google.com/g/ml-news

SIAM: https://www.siam.org/membership/activity-groups/detail/computational-science-and-engineering https://www.siam.org/membership/activity-groups/detail/data-science https://www.siam.org/membership/activity-groups/detail/imaging-science https://www.siam.org/membership/activity-groups/detail/optimization

opened by Buyun-Liang 11
Discussion: code and algorithms
General things:

[x] #13

[x] #20

[x] What is the deal with X_struct and the separation of variables as opposed to just an input vector?

[x] #14

Gradient stuff:

[x] Does the AD enabling have to happen in comb_fn that the user provides?

[x] #15

[x] Is adding support for the user to choose between explicit and AD gradients useful? Might be a bit of work to support this though!

Line search stuff

[x] What is the deal with disabling code 6 termination and how is the BFGS update then ensured to not lose positive definiteness?

[x] What is the deal with the backtracking line search? Same issue with positive definiteness and this may not work well for nonsmooth problems. The GRANSO line search is a key component of the BFGS-SQP algorithm.

[x] #16

Code TODO
opened by t-mitchell 8
update license and copyright
I went through the private folder and saw that we have four different types of files to update. I did some sample updates so you (a) can review to see if you agree and like my changes and (b) have templates to work from. The types are:

direct port from GRANSO (see bfgsDamping.py)

direct port from URTM / GRANSO (see isAnInteger.py)

port from GRANSO with changes (see linesearchWeakWolfe.py)

entirely new file and code (see getCiGradVec.py)

Note that I added a GPL header block linesearchWeakWolfe.py (and bfgsHessianInverse.py) as these individual routines were derived from HANSO which is GPL. But don't add this headers to any other files.

Overall, I tried to make things as compact as possible and only use one AGPL license block. I also eliminated some lines I thought were unnecessary or would be annoying to continually update for future releases. For example, maintaining version numbers in every file is tedious work, so I only put version numbers in the main user-facing routines, i.e., granso, gransoOptions, and gransoOptionsAdvanced.

p.s. Let me know what you think and if you have suggestions! I definitely went back and forth on how to clearly do all of this.
opened by t-mitchell 6
authors file
Regarding AUTHORS.md:

Do you think we need or want such a detailed file? I was thinking we just I wrote GRANSO and you ported it to Python and added PyTorch support and more example.

I wasn't sure how you determined the file groups, as they look different from what I see marked as direct port or not in the private folder.
opened by t-mitchell 4
docstring indentation

Probably we should remove a level of indentation in the docstring for each file, since as is, we have double the normal amount of indentation. Obviously, this is a super minor point and not critical.

opened by t-mitchell 4
Regarding dummy class in `pygransoStruct.py`

I'm going to replace in the dummy class with dict for the v1.0.0 release.

How should I define the dict for the opts so that the values in dict can have different data type?
Code TODO

opened by Buyun-Liang 3
"Please see the notes for the data structures used in PyGRANSO"?

What does "Please see the notes for the data structures used in PyGRANSO" refer to? Is it another document somewhere? Or is this self-referential, i.e., referring to the "Notes on Documentation" section in the readme that this phrase appears in?
question

opened by t-mitchell 2
Examples Updated

I just updated all examples in the ./example folder. Maybe we can find a time to carefully review the README.md tomorrow

Please let me know if there is any other thing to be updated in the source code before the initial public release.

opened by Buyun-Liang 2
TODO in JMLR manuscript
"zero dependency on proprietary packages": we should discuss this since GRANSO doesn't really have this because it could be run using OCTAVE

Role of the NCVX name in the PyGRANSO package. NCVX and PyGRANSO in the JMLR paper

Paper TODO
opened by Buyun-Liang 2
Discussion: naming and titles
[x] #9

[x] #10 optimization in machine learning" be for NCVX or PyGRANSO?

[x] Relatedly, is the title of the paper going to change?

[x] #12

Depending on the answers to these questions, we'll need to change the docstrings in all the files.
opened by t-mitchell 2
Minor bug in qpTerminationCondition.py#L226

https://github.com/sun-umn/PyGRANSO/blob/main/pygranso/private/qpTerminationCondition.py#L226

Use Hreg = torch.tensor(V) @ torch.diag(dvec) @ torch.tensor(V).T instead

opened by Buyun-Liang 0
TODO in future release
For avoiding unnecessary AD gradients, the GRANSO line search could be modified so that the AD gradient is only computed on demand, i.e., only compute the gradient when sufficient decrease holds

Adding the MPI parallel full BFGS Python code Tim has

Future Release
opened by Buyun-Liang 2

Releases(v1.2.0)

v1.2.0(Jul 27, 2022)
Version: 1.2.0 --- 2022-07-26

Description: major fixes and improvements on LBFGS.

Fixed

Reducing memory usage for LBFGS. Now PyGRANSO can solve problem with ~15k parameters by using 14 GB memory.

Update example: ortho RNN with max folding and orthonormal initialization.

Allow high precision for QP solver.

Allow part of optimization variables not showing up in objective (see SVM example).

Fixed Code 12: terminated with steering failure.

Fixed stationary failure: try different stationarity calculation, or set stationarity measure to be inf if encounter numerical issue

Added

Reorganize and add examples: perceptual/lp norm attack on ImageNet images. trace optimization with orthogonal constraints; unconstrained deep learning with LeNet5; logistic regression.

Source code(tar.gz)
Source code(zip)
v1.1.0(Feb 20, 2022)
Version: 1.1.0 --- 2022-02-20

Description: major fixes and improvements.

Fixed

Avoid gradient accumulating in deep learning problem;

Prevent memory leak problem when using torch tensor. See ex6 perceptual attack.

Changed

Update format of user-defined variables when using pygranso interface.

Packaging

Publish pygranso package on Pypi.

Added

Add examples: ex 10 dictionary learning with torch.nn module; ex 11 orthogonal recurrent neural networks.

Source code(tar.gz)
Source code(zip)
v1.0.0(Dec 31, 2021)

Description: initial public release of PyGRANSO.

Main features: auto-differentiation, GPU acceleration, tensor input, scalable QP solver, and zero dependency on proprietary packages. Multiple new examples added.
Source code(tar.gz)
Source code(zip)

Owner

SUN Group @ UMN

GitHub https://ncvx.org

Theano is a Python library that allows you to define, optimize, and evaluate mathematical expressions involving multi-dimensional arrays efficiently. It can use GPUs and perform efficient symbolic differentiation.

============================================================================================================ `MILA will stop developing Theano <https:

9.3k Feb 12, 2021

Automatic differentiation with weighted finite-state transducers.

GTN: Automatic Differentiation with WFSTs Quickstart | Installation | Documentation What is GTN? GTN is a framework for automatic differentiation with

100 Dec 29, 2022

JAX code for the paper "Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation"

Optimal Model Design for Reinforcement Learning This repository contains JAX code for the paper Control-Oriented Model-Based Reinforcement Learning wi

43 Sep 28, 2022

ShapeGlot: Learning Language for Shape Differentiation

ShapeGlot: Learning Language for Shape Differentiation Created by Panos Achlioptas, Judy Fan, Robert X.D. Hawkins, Noah D. Goodman, Leonidas J. Guibas

32 Dec 23, 2022

Automatic Differentiation Multipole Moment Molecular Forcefield

Automatic Differentiation Multipole Moment Molecular Forcefield Performance notes On a single gpu, using waterbox_31ang.pdb example from MPIDplugin wh

4 Jan 7, 2022

Aircraft design optimization made fast through modern automatic differentiation

Aircraft design optimization made fast through modern automatic differentiation. Plug-and-play analysis tools for aerodynamics, propulsion, structures, trajectory design, and much more.

394 Dec 23, 2022

🦕 NanoSaur is a little tracked robot ROS2 enabled, made for an NVIDIA Jetson Nano

?? nanosaur NanoSaur is a little tracked robot ROS2 enabled, made for an NVIDIA Jetson Nano Website: nanosaur.ai Do you need an help? Discord For tech

162 Dec 9, 2022

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.