OptNet: Differentiable Optimization as a Layer in Neural Networks

Overview

OptNet: Differentiable Optimization as a Layer in Neural Networks

This repository is by Brandon Amos and J. Zico Kolter and contains the PyTorch source code to reproduce the experiments in our ICML 2017 paper OptNet: Differentiable Optimization as a Layer in Neural Networks.

If you find this repository helpful in your publications, please consider citing our paper.

@InProceedings{amos2017optnet,
  title = {{O}pt{N}et: Differentiable Optimization as a Layer in Neural Networks},
  author = {Brandon Amos and J. Zico Kolter},
  booktitle = {Proceedings of the 34th International Conference on Machine Learning},
  pages = {136--145},
  year = {2017},
  volume = {70},
  series = {Proceedings of Machine Learning Research},
  publisher ={PMLR},
}

Informal Introduction

Mathematical optimization is a well-studied language of expressing solutions to many real-life problems that come up in machine learning and many other fields such as mechanics, economics, EE, operations research, control engineering, geophysics, and molecular modeling. As we build our machine learning systems to interact with real data from these fields, we often cannot (but sometimes can) simply ``learn away'' the optimization sub-problems by adding more layers in our network. Well-defined optimization problems may be added if you have a thorough understanding of your feature space, but oftentimes we don't have this understanding and resort to automatic feature learning for our tasks.

Until this repository, no modern deep learning library has provided a way of adding a learnable optimization layer (other than simply unrolling an optimization procedure, which is inefficient and inexact) into our model formulation that we can quickly try to see if it's a nice way of expressing our data.

See our paper OptNet: Differentiable Optimization as a Layer in Neural Networks and code at locuslab/optnet if you are interested in learning more about our initial exploration in this space of automatically learning quadratic program layers for signal denoising and sudoku.

Setup and Dependencies

  • Python/numpy/PyTorch
  • qpth: Our fast QP solver for PyTorch released in conjunction with this paper.
  • bamos/block: Our intelligent block matrix library for numpy, PyTorch, and beyond.
  • Optional: bamos/setGPU: A small library to set CUDA_VISIBLE_DEVICES on multi-GPU systems.

Denoising Experiments

denoising
├── create.py - Script to create the denoising dataset.
├── plot.py - Plot the results from any experiment.
├── main.py - Run the FC baseline and OptNet denoising experiments. (See arguments.)
├── main.tv.py - Run the TV baseline denoising experiment.
└── run-exps.sh - Run all experiments. (May need to uncomment some lines.)

Sudoku Experiments

  • The dataset we used in our experiments is available in sudoku/data.
sudoku
├── create.py - Script to create the dataset.
├── plot.py - Plot the results from any experiment.
├── main.py - Run the FC baseline and OptNet Sudoku experiments. (See arguments.)
└── models.py - Models used for Sudoku.

Classification Experiments

cls
├── train.py - Run the FC baseline and OptNet classification experiments. (See arguments.)
├── plot.py - Plot the results from any experiment.
└── models.py - Models used for classification.

Acknowledgments

The rapid development of this work would not have been possible without the immense amount of help from the PyTorch team, particularly Soumith Chintala and Adam Paszke.

Licensing

Unless otherwise stated, the source code is copyright Carnegie Mellon University and licensed under the Apache 2.0 License.

Comments
  • Sudoku experiment

    Sudoku experiment

    I tried running the sudoku example. My command line arguments are as follows:

    python main.py --boardSz 2 --batchSz 100 --testBatchSz 100 --nEpoch 50 --testPct 0.1 --save SAVE --work 'work' optnet
    

    I get errors arising from models.py:

    TypeError: super() takes at least 1 argument (0 given)
    > /home/robotec/Documents/NNs/optnet/sudoku/models.py(80)__init__()
         78     """ Solve a single SQP iteration of the scheduling problem"""
         79     def __init__(self, n, Qpenalty):
    ---> 80         super().__init__()
         81         nx = (n**2)**3
         82         self.Q = Variable(Qpenalty*torch.eye(nx).double().cuda())
    
    ipdb> 
    

    Did I miss something?

    opened by robotsorcerer 8
  • QP Solver with extra terms and more variables

    QP Solver with extra terms and more variables

    Hi @bamos ,

    Thank you for this promising work and clean code if you don't mind.

    I have some questions related to the solver.

    Is it possible to :

    1. Add extra terms in the objective function and variable constraints ? https://github.com/locuslab/qpth/blob/master/qpth/qp.py#L36

    A) extra terms in the objective function

    \hat z = argmin_z 1/2 z^T Q z + p^T z+......+....+......

    B) Add more variables constraints : for instance add y_, F_, K_, n_ to Q_, p_, G_, h_, A_, b_

    1. Is this constraint solver can deal with maximum solution rather than minimum ?

    \hat z = argmax_z 1/2 z^T Q z + p^T z+......+....+......

    Thank you a lot for your answer

    opened by Benjiou 1
  • several issues

    several issues

    Hi there, thx for sharing this wonderful works. I am trying to run the code, and encounter some situations:

    1. when I trying to run test and profile code, I fail to import adact. But I have no idea where the lib is?
    2. when running denoising example, I use the command as python main.py --nEpoch 50 optnet --learnD --Dpenalty 0.01 and I got the following error message: ('===>', 'Building model') Testing model: 0/1000 Traceback (most recent call last): File "main.py", line 229, in <module> main() File "main.py", line 136, in main test(args, 0, model, testF, testW, testX, testY) File "main.py", line 202, in test output = model(batch_data) File "/usr/local/lib/python2.7/dist-packages/torch/nn/modules/module.py", line 206, in __call__ result = self.forward(*input, **kwargs) File "/home/jdily/Desktop/project/optnet-master/denoising/models.py", line 155, in forward x = QPFunction()(p.double(), Q.double(), G.double(), h.double(), e, e).float() File "/usr/local/lib/python2.7/dist-packages/qpth/qp.py", line 65, in forward p, _ = expandParam(p_, nBatch, 2) File "/usr/local/lib/python2.7/dist-packages/qpth/util.py", line 50, in expandParam raise RuntimeError("Unexpected number of dimensions.") RuntimeError: Unexpected number of dimensions. I use python 2.7 instead of 3.x, but I do update some part of the code to support python 2.7... Do you know how can I solve these issues?

    Meanwhile, the link to the denoising data seems broken.

    thanks a lot!

    opened by jdily 1
  • how to run the cpu version

    how to run the cpu version

    /usr/local/lib/python3.6/site-packages/IPython/core/ultratb.py(1268)call() -> self.handler((etype, evalue, etb)) (Pdb)


    AssertionError Traceback (most recent call last) /Users/gaopeng/Desktop/tensorflow/project/optnet/denoising/main.py in () 229 230 if name=='main': --> 231 main() global main = <function main at 0x114ba7510>

    /Users/gaopeng/Desktop/tensorflow/project/optnet/denoising/main.py in main() 109 elif args.model == 'optnet': 110 if args.learnD: --> 111 model = models.OptNet_LearnD(nFeatures, args) model = undefined global models.OptNet_LearnD = <class 'models.OptNet_LearnD'> nFeatures = 100 args = Namespace(Dpenalty=0.01, batchSz=150, cuda=False, eps=0.0001, learnD=True, model='optnet', nEpoch=50, no_cuda=False, save='work/optnet.eps=0.0001.learnD.0.01', testBatchSz=100, testPct=0.1, tvInit=False) 112 else: 113 model = models.OptNet(nFeatures, args)

    /Users/gaopeng/Desktop/tensorflow/project/optnet/denoising/models.py in init(self=OptNet_LearnD ( ), nFeatures=100, args=Namespace(Dpenalty=0.01, batchSz=150, cuda=False....01', testBatchSz=100, testPct=0.1, tvInit=False)) 113 114 # self.fc1 = nn.Linear(nFeatures, nHidden) --> 115 self.M = Variable(torch.tril(torch.ones(nHidden, nHidden)).cuda()) self.M = undefined global Variable = <class 'torch.autograd.variable.Variable'> global torch.tril = global torch.ones = nHidden = 199 nHidden.cuda = undefined 116 117 Q = 1e-8*torch.eye(nHidden)

    /usr/local/lib/python3.6/site-packages/torch/utils.py in cuda(self= 1 0 0 ... 0 0 0 1... 1 1 [torch.FloatTensor of size 199x199] , device=-1, async=False) 63 else: 64 new_type = getattr(torch.cuda, self.class.name) ---> 65 return new_type(self.size()).copy(self, async) new_type = <class 'torch.cuda.FloatTensor'> self.size.copy = undefined self = 1 0 0 ... 0 0 0 1 1 0 ... 0 0 0 1 1 1 ... 0 0 0 ... ⋱ ...
    1 1 1 ... 1 0 0 1 1 1 ... 1 1 0 1 1 1 ... 1 1 1 [torch.FloatTensor of size 199x199]

        async = False
     66 
     67 
    

    /usr/local/lib/python3.6/site-packages/torch/cuda/init.py in new(cls=<class 'torch.cuda.FloatTensor'>, *args=(torch.Size([199, 199]),), **kwargs={}) 275 276 def new(cls, *args, **kwargs): --> 277 _lazy_init() global _lazy_init = <function _lazy_init at 0x110ab5378> 278 # We need this method only for lazy init, so we can remove it 279 del _CudaBase.new

    /usr/local/lib/python3.6/site-packages/torch/cuda/init.py in _lazy_init() 87 raise RuntimeError( 88 "Cannot re-initialize CUDA in forked subprocess. " + msg) ---> 89 _check_driver() global _check_driver = <function _check_driver at 0x110ab52f0> 90 assert torch._C._cuda_init() 91 assert torch._C._cuda_sparse_init()

    /usr/local/lib/python3.6/site-packages/torch/cuda/init.py in _check_driver() 54 def _check_driver(): 55 if not hasattr(torch._C, '_cuda_isDriverSufficient'): ---> 56 raise AssertionError("Torch not compiled with CUDA enabled") global AssertionError = undefined 57 if not torch._C._cuda_isDriverSufficient(): 58 if torch._C._cuda_getDriverVersion() == 0:

    AssertionError: Torch not compiled with CUDA enabled

    /usr/local/lib/python3.6/site-packages/IPython/core/ultratb.py(1269)call() -> try:

    opened by gaopeng-eugene 1
  • RuntimeError: invalid argument 3: dimensions of A and b must be equal

    RuntimeError: invalid argument 3: dimensions of A and b must be equal

    When I used the optnet to run all the three experiments, I got the following error:

    ` RuntimeError: invalid argument 3: dimensions of A and b must be equal at /pytorch/torch/lib/TH/generic/THTensorLapack.c:1036

    /usr/local/lib/python3.5/dist-packages/qpth/solvers/pdipm/batch.py(357)solve_kkt() 355 h = (invQ_rx.unsqueeze(1).bmm(G.transpose(1, 2)) + rs / d - rz).squeeze(1) 356 --> 357 w = -(h.btrisolve(*S_LU)) 358 359 g1 = -rx - w[:, neq:].unsqueeze(1).bmm(G) `

    Did I miss something or do something wrong with the qpth or other environment? I can run the fc and lenet network in the classification experiment. Since I don't have a CUDA, I uncomment all the “.cuda()” in the models.py

    opened by Blanchedingding 1
  • MNIST experiments creating qpth issues

    MNIST experiments creating qpth issues

    Hi,

    I was running the optnet code for MNIST classification with the default configurations for only 10 epochs. In the first couple of epochs I get the warning "qpth warning: Returning an inaccurate and potentially incorrect solutino" and in the subsequent iterations the loss becomes nan. Is there something obviously wrong with my configurations?

    opened by guptakartik 5
Owner
CMU Locus Lab
Zico Kolter's Research Group
CMU Locus Lab
An abstraction layer for mathematical optimization solvers.

MathOptInterface Documentation Build Status Social An abstraction layer for mathematical optimization solvers. Replaces MathProgBase. Citing MathOptIn

JuMP-dev 284 Jan 4, 2023
Differentiable Factor Graph Optimization for Learning Smoothers @ IROS 2021

Differentiable Factor Graph Optimization for Learning Smoothers Overview Status Setup Datasets Training Evaluation Acknowledgements Overview Code rele

Brent Yi 60 Nov 14, 2022
Pytorch implementation of "Forward Thinking: Building and Training Neural Networks One Layer at a Time"

forward-thinking-pytorch Pytorch implementation of Forward Thinking: Building and Training Neural Networks One Layer at a Time Requirements Python 2.7

Kim Heecheol 65 Oct 6, 2022
Official implementation of GraphMask as presented in our paper Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking.

GraphMask This repository contains an implementation of GraphMask, the interpretability technique for graph neural networks presented in our ICLR 2021

Michael Schlichtkrull 29 Sep 2, 2022
EdMIPS: Rethinking Differentiable Search for Mixed-Precision Neural Networks

EdMIPS is an efficient algorithm to search the optimal mixed-precision neural network directly without proxy task on ImageNet given computation budgets. It can be applied to many popular network architectures, including ResNet, GoogLeNet, and Inception-V3.

Zhaowei Cai 47 Dec 30, 2022
Neural Turing Machine (NTM) & Differentiable Neural Computer (DNC) with pytorch & visdom

Neural Turing Machine (NTM) & Differentiable Neural Computer (DNC) with pytorch & visdom Sample on-line plotting while training(avg loss)/testing(writ

Jingwei Zhang 269 Nov 15, 2022
Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

郭飞 3.7k Jan 3, 2023
library for nonlinear optimization, wrapping many algorithms for global and local, constrained or unconstrained, optimization

NLopt is a library for nonlinear local and global optimization, for functions with and without gradient information. It is designed as a simple, unifi

Steven G. Johnson 1.4k Dec 25, 2022
Racing line optimization algorithm in python that uses Particle Swarm Optimization.

Racing Line Optimization with PSO This repository contains a racing line optimization algorithm in python that uses Particle Swarm Optimization. Requi

Parsa Dahesh 6 Dec 14, 2022
A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

An Introduction to Deep Learning for the Physical Layer An usable PyTorch implementation of the noisy autoencoder infrastructure in the paper "An Intr

Gram.AI 120 Nov 21, 2022
Pytorch implementation of AngularGrad: A New Optimization Technique for Angular Convergence of Convolutional Neural Networks

AngularGrad Optimizer This repository contains the oficial implementation for AngularGrad: A New Optimization Technique for Angular Convergence of Con

mario 124 Sep 16, 2022
Differentiable architecture search for convolutional and recurrent networks

Differentiable Architecture Search Code accompanying the paper DARTS: Differentiable Architecture Search Hanxiao Liu, Karen Simonyan, Yiming Yang. arX

Hanxiao Liu 3.7k Jan 9, 2023
Complex-Valued Neural Networks (CVNN)Complex-Valued Neural Networks (CVNN)

Complex-Valued Neural Networks (CVNN) Done by @NEGU93 - J. Agustin Barrachina Using this library, the only difference with a Tensorflow code is that y

youceF 1 Nov 12, 2021
Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Parallel Tacotron2 Pytorch Implementation of Google's Parallel Tacotron 2: A Non-Autoregressive Neural TTS Model with Differentiable Duration Modeling

Keon Lee 170 Dec 27, 2022
Pytorch implementation of DeepMind's differentiable neural computer paper.

DNC pytorch This is a Pytorch implementation of DeepMind's Differentiable Neural Computer (DNC) architecture introduced in their recent Nature paper:

Yuanpu Xie 91 Nov 21, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 235 Dec 26, 2022
This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

DeepMind 892 Dec 28, 2022
Minimal PyTorch implementation of Generative Latent Optimization from the paper "Optimizing the Latent Space of Generative Networks"

Minimal PyTorch implementation of Generative Latent Optimization This is a reimplementation of the paper Piotr Bojanowski, Armand Joulin, David Lopez-

Thomas Neumann 117 Nov 27, 2022
ICCV2021 Oral SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks

Sign-Agnostic Convolutional Occupancy Networks Paper | Supplementary | Video | Teaser Video | Project Page This repository contains the implementation

null 63 Nov 18, 2022