Hi, I was wondering if cooper can use cuda to accelerate the computation. (I thought it could, but there is no place to claim something like probs.cuda() )

Enhancement

Remove existing aliases to objects in torch.optim. Update documentation accordingly.

Motivation

Having these aliases makes the interface opaque for the user: it is hard to tell that cooper.optim.SGD is in fact the same implementation as torch.optim.SGD.

Enhancement

__init__ files throughout Cooper are not fully consistent.

For instance, Cooper.__init__ imports UnconstrainedFormulation and LagrangianFormulation , but not AugmentedLagrangianFormulation. https://github.com/cooper-org/cooper/blob/a8b3400fecbe9e6e8e5e8c4d49c7959623a0154e/cooper/init.py#L18-L22

For consistency, we should import the AugmentedLagrangianFormulation as well.

Note: Creating an issue as there may be other instances of this throughout the library.

Bug

The StateLogger in cooper.utils.state_logger.py is not compatible with the latest version of Cooper's dev branch.

It assumes that a Formulation will have a ConstrainedMinimizationProblem as an attribute:

https://github.com/cooper-org/cooper/blob/3e0aa6fd80b2977936db8aae4affce78bb466379/cooper/utils/state_logger.py#L42-L48

This is not the case since merging #56.

Steps

Update the interface of StateLogger. store_metrics should receive a CMPState or the unpacked metrics.

Enhancement

Modify README to include a more comprehensive example of how to use Cooper for machine learning problems. Some features to cover include:

• Integration with torch.nn.Module
• Using CUDA acceleration
• Data loaders and mini-batching

We could to have two versions:

• One with "off-the-shelf Cooper": no explicit user-defined closure
• One with explicit closure definition, which enables the use of advanced features like extrapolation optimizers

Maybe only include the first one in the README, and keep the second version for the docs.

Enhancement

Create wrappers cooper.backward for formulation.custom_backward and cooper.compute_lagrangian for formulation.composite_objective.

This would change the way a user interacts with Cooper. However, it does not represent a change in Cooper's back-end.

Motivation

A cleaner pipeline for the user, where there is no need to access the methods of a formulation:

for step in range(num_steps):
    ...
    constrained_optimizer.zero_grad()
    lagrangian = cooper.compute_lagrangian(formulation, ...)
    cooper.backward(formulation, lagrangian)
    constrained_optimizer.step()

As opposed to the current pipeline:

for step in range(num_steps):
    ...
    constrained_optimizer.zero_grad()
    lagrangian = formulation.composite_objective(...)
    formulation.custom_backward(lagrangian)
    constrained_optimizer.step()

Enhancement

Make Cooper as easy as possible to integrate with a vanilla Pytorch pipeline.

In particular, allow for the user to populate a CMPState and provide it to Cooper without having to create a custom ConstrainedMinimizationProblem.

Note that we should not remove the CMP altogether. In particular, extrapolation and alternating updates would still require internal calls to cmp.closure and cmp.defect_fn.

This would require a major overhaul of the documentation, tutorials, and examples, showcasing that both the previous CMP approach and this new approach are admissible. Love for the documentation has also been requested in #53 and #29.

Motivation

Users are currently required to implement a custom ConstrainedMinimizationProblem with a closure method. This overhead can be detrimental to Cooper attracting ML researchers and practitioners who would otherwise compute the loss and constraint violations themselves.

For minimal integrations with Cooper which do not need "fancy features" like Augmented Lagrangian or Extrapolation, simply asking for the loss and constraint defects makes the user's life easier.

Alternatives

It may be possible to remove the CMP completely. Nonetheless, we would still require a closure for an internal call during extrapolation steps and a defect_fc for internal use by the AlternatingConstrainedOptimizer.

Enhancement

Modularize the constrained_optimizer.py script to have a BaseConstrainedOptimizerclass from which specific optimization approaches can stem.

For instance, a SimultaneousOptimizer, AlternatingOptimizer, ExtrapolationOptimizer, and in the future perhaps an ExtrapolationFromThePastOptimizer.

Motivation

Current implementation of ConstrainedOptimizer is very flexible: it allows for unconstrained optimization, constrained optimization with simultaneous and alternating updates and extragradient updates.

Because of this, the logic inside the ConstrainedOptimizer class has become overly complex. https://github.com/cooper-org/cooper/blob/f1d95543f48d1127da8111d285f45fc5936426c6/cooper/constrained_optimizer.py#L268-L270

A more modularized implementation would make the code easier to understand and less error prone. The added flexibility would enable implementing new optimization approaches, like extrapolation from the past [1].

A similar approach was taken with the Formulations, where a BaseLagrangianFormulation underpins the LagrangianFormulation and ProxyLagrangianFormulation.

References

• Gidel, G., Berard, H., Vignoud, G., Vincent, P., & Lacoste-Julien, S. (2019). A variational inequality perspective on generative adversarial networks. In ICLR.

Bug

The call to dual_scheduler.step() happens at the end of ConstrainedOptimizer.step(). This means that dual_scheduler steps happen at the end of every optimization step and not every optimization epoch.

https://github.com/cooper-org/cooper/blob/eae7c5a68563b83410913c8b24e99b70e32694f3/cooper/constrained_optimizer.py#L235-L240

This goes against the Pytorch convention on learning rate schedulers.

Expected behavior

Calls to dual_scheduler.step() should happen at the end of every epoch.

How to fix

We could: (i) Remove the call to dual_scheduler.step() from inside the ConstrainedOptimizer and let the user handle it. The instantiation of the dual scheduler could still be performed internally so that the partial_lr_scheduler logic is preserved (and for the user to not worry about it). (ii) Indicate whether a step is the last of its epoch and only then call dual_scheduler.step(). (iii) Create a class which wraps both the primal and dual schedulers (like how the ConstrainedOptimizer wraps primal and dual optimizers), whose step() method is manually called by the user at the end of an epoch. The class could internally handle the initialization of the dual scheduler. This approach is similar to (i)

Closes #39

Changes

Parameter primal_optimizer of ConstrainedOptimizer renamed to primal_optimizers. It accepts either a (i) torch.optim.Optimizer or a (ii) list of Optimizers. Behavior in case (i) is unchanged.

Lines previously performing constrained_optimizer.primal_optimizer.method() have been replaced by for primal_optimizer in constrained_optimizer.primal_optimizers; primal_optimizer.method(). For instance,

https://github.com/cooper-org/cooper/blob/4421115314e25386042cbd2c3731b03aa766f082/cooper/constrained_optimizer.py#L329-L330

Saving and loading of checkpoints was modified to allow for lists of primal optimizers.

⚠ Warning ⚠ This breaks backward compatibility when instantiating a ConstrainedOptimizer with keyword argument primal_optimizer.

Testing

Toy2D testing of constrained and unconstrained execution included in test_optimizer.py. Checkpointing test included in test_checkpoint.py

This functionality is not tested together with other optimization methods or formulations (Extragradient, Augmented Lagrangian, Proxy constraints).

Docs

Note added in constrained_optimizer.rst indicating that functionality exists. Small section added in optim.rst describing how to setup multiple primal optimizers and how Cooper handles them. Docstrings and overall documentation updated to be general enough for multiple optimizers.

Bug

The instantiation of the dual_scheduler inside ConstrainedOptimizer.load_from_state_dict method has a bug. The variable containing the dual scheduler class is called dual_scheduler_class as opposed to dual_scheduler. https://github.com/cooper-org/cooper/blob/f75dff39608f2d2895411c6e40e3675aba03c0f2/cooper/constrained_optimizer.py#L491

Steps

Replace current line to dual_scheduler = dual_scheduler_class(dual_optimizer)

Expected behavior

This should enable the loading of a dual_scheduler from a checkpoint.

Closes #54

Changes

On the multipliers module we created a new class called cooper.multipliers.MultiplierModel, which predicts the value of the Lagrange multipliers associated with the equality or inequality constraints of a cooper.problem.ConstrainedMinimizationProblem. Additionally, we created a new Lagrangian formulation --> cooper.formulation.LagrangianFormulation that works with the created MultiplierModel class.

Testing

We tested the class multipliers.MultiplierModel with the simplest model. A linear model with a single output. The new Lagrangian Formulation was also tested.

References

This idea comes from the paper by Narasimhan et al. and we took their repository as reference.

Enhancement

Implement the extrapolation from the past algorithm (Popov, 1980). A good and modern source is Gidel et al. (2019).

This is an algorithm for computing parameter updates similar to extragradient: it computes the direction for updating parameters based on a "lookahead step". It is less intensive computationally than extragradient and enjoys similar convergence results for some class problems (Gidel et al., 2019).

Motivation

Whereas extragradient requires two gradient computations per parameter update, extrapolation from the past stores and re-uses gradients from previous extrapolation steps for use during current extrapolation steps. This means less computational intensity in terms of gradient calculations, which may be helpful in some settings.

However, storing the previous gradients still means an overhead in terms of storage as opposed to gradient descent-ascent.

References

• G. Gidel, H. Berard, G. Vignoud, P. Vincent, S. Lacoste-Julien. A Variational Inequality Perspective on Generative Adversarial Networks. In ICLR, 2019.
• L. D. Popov. A modification of the arrow-hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR, 1980.

Enhancement

Implement a "Multiplier Model" in Cooper. This idea comes from the paper by Narasimhan et al.

Instead of tracking the Lagrange Multipliers of a constrained optimization problem explicitly, consider a model which takes as input features associated with each constraint and outputs their corresponding multiplier.

This requires a new kind of Multiplier in Cooper different from a DenseMultiplier. In turn, we might want to create and abstract class for multipliers outlining how a custom multiplier needs to be implemented.

This also requires a new formulation or the modification of an existing one. Note that the dual_parameters do not match the multipliers in this new functionality. Multipliers are the outputs of the Multiplier Model, whereas the dual_parameters would be the actual weights of said model.

Motivation

Consider problems with an extremely large (or even infinite) number of constraints. It becomes computationally intractable to evaluate and keep track of each constraint. Moreover, when solving the problem with an (explicit) Lagrangian formulation, there is the requirement to store and update an equally large number of multipliers.

With a Multiplier Model, Lagrange multipliers can be computed "on demand", given the constraints that were evaluated at each time step. As long as the Multiplier Model has less parameters than the total possible number of multipliers, there is a gain in terms of storage.

Also, note that the Multiplier Model's parameters are shared across constraints. Therefore, they can learn to have similar multipliers for constraints which tend to be similar to one another.

References

• H. Narasimhan, A. Cotter, Y. Zhou, S. Wang, and W. Guo. Approximate Heavily-Constrained Learning with Lagrange Multiplier Models. In NeurIPS, 2020b.

Enhancement

This PR #52 made significant changes to the structure of the code for constrained optimizers. However, this change has not been fully integrated into the docs.

The documentation of this library is very limited and it is hard to parse how it should be applied for different applications. For instance, in Many RL algorithms such as PPO and MPO, an entropy regularization terms or the KL constraints between policies employed to stabilise standard RL objectives. Is it possible to give a practical example of how one can use this library for a dual problem with lagrangian multipliers for such applications?

Many thanks.

Enhancement

Pytorch optimizers include a maximize flag (Pytorch Issue)*. When set to True, the sign of gradients is flipped inside optimizer.step() before computing parameter updates. This enables gradient ascent steps natively.

NOTE: This sign flip does not affect the parameter's .grad attribute.

Cooper currently populates the gradients of dual variables with their negative value, so that descent steps performed by the dual optimizer are in fact ascent steps towards optimizing the problem formulation.

https://github.com/cooper-org/cooper/blob/09df7597e9e42deed4c7a162a2bb345868c8bf23/cooper/constrained_optimizer.py#L401-L406

We should not do the sign flipping manually but rather force set maximize=True when instantiating the dual optimizer.

* This has been implemented on on Pytorch's master branch for every optimizer but LBFGS . On v1.12, Adam, SGD and AdaGrad support the flag, but not RMSProp. An assert could be included to ensure that the requested dual optimizer supports the flag.

⚠ This change would break compatibility with versions of Pytorch prior to 1.12.

Motivation

Manually flipping gradients immediately after calculating them (thus ensuring that this happens before calls to dual_optimizer.step()) is error prone. Moreover, keeping track of the fact that gradients have a sign flipped is inconvenient.

By implementing this change we would adopt the official Pytorch approach for performing ascent steps.

Alternatives

The current implementation is functional.

References

• https://github.com/pytorch/pytorch/issues/68052