CS 7301: Spring 2021 Course on Advanced Topics in Optimization in Machine Learning

Overview

Advanced Topics in Optimization for Machine Learning

CS 7301: Spring 2021 Course on Advanced Topics in Optimization for Machine Learning

Video Lectures

Video Lectures are on this youtube playlist: https://www.youtube.com/playlist?list=PLGod0_zT9w92_evaYrf3-rE67AmgPJoUU

Github Link to all Demos

https://github.com/rishabhk108/OptimizationDemos

Link to Google Spreadsheet for Paper Review and Project Topics

https://docs.google.com/spreadsheets/d/1UHHFlo_8QAvmXjWqoU02Calq86S-ewYl7Jczjhgr0wY/edit?usp=sharing

Deadline for finalizing on the papers to cover: February 26th

Deadine for finalizing on the project topic: March 5th

Topics Covered in this Course

  • Week 1
    • Logistics, Outline of this Course
    • Continuous Optimization in ML
    • Convex Sets and Basics of Convexity
  • Week 2: Gradient Descent and Family
    • Convex Functions, Properties, Minima, Subgradients
    • Gradient Descent and Line Search
  • Week 3: Gradient Descent Cont.
    • Accelerated Gradient Descent
    • Projected and Proximal Gradient Descent
  • Week 4
    • Projected GD and Conditional GD (Constrained Case)
    • Second Order Methods (Newton, Quasi-Newton, BFGS, LBFGS)
  • Week 5
    • Second Order Methods Completed
    • Barzelia Borwein and Conjugate GD
    • Coordinate Descent Family
  • Week 6
    • Stochastic Gradient and Family (SGD, SVRG)
    • SGD for Non-Convex Optimization. Modern variants of SGD particularly for deep learning (e.g. Adagrad, Adam, AdaDelta, RMSProp, Momentum etc.)
  • Week 7
    • Submodular Optimization: Basics, Definitions, Properties, and Examples.
  • Week 8
    • Submodular Information Measures: Conditional Gain, Submodular Mutual Information, Submodular Span, Submodular Multi-Set Mutual Information
  • Week 9
    • Submodular Minimization and Continuous Extensions of Submodular Functions. Submodular Minimization under constraints
  • Week 10
    • Submodular Maximization Variants, Submodular Set Cover, Approximate submodularity. Algorithms under different constraints and monotone/non-monotone settings. Also, distributed and streaming algorithms, DS Optimization, Submodular Optimization under Submodular Constraints
  • Week 11
    • Applications of Discrete Optimization: Data Subset Selection, Data Summarization, Feature Selection, Active Learning etc.
  • Rest of the Weeks
    • Paper Presentations/Project Presentations by the Students

Grading

  • 10% for Class Participation (Interaction, asking questions, answering questions)
  • 30% Assignments (2 Assignments, one on continuous optimization and one on discrete optimization)
  • 30% Paper Presentations (1-2 papers per student)
  • 30% for the Final Project
    • Take a new dataset/problem and study how existing optimization algorithms work on them
    • Take an existing problem and compare all optimization algorithms with your implementation from scratch
    • Design a ML optimization toolkit with algorithms implemented from scratch -- if one of you would like to extend my current python demos for optimization, that will be an awesome contribution and I might pick it up for my future classes and acknowledge you :)

Other Similar Courses

Resources/Books/Papers

You might also like...
MooGBT is a library for Multi-objective optimization in Gradient Boosted Trees.
MooGBT is a library for Multi-objective optimization in Gradient Boosted Trees.

MooGBT is a library for Multi-objective optimization in Gradient Boosted Trees. MooGBT optimizes for multiple objectives by defining constraints on sub-objective(s) along with a primary objective. The constraints are defined as upper bounds on sub-objective loss function. MooGBT uses a Augmented Lagrangian(AL) based constrained optimization framework with Gradient Boosted Trees, to optimize for multiple objectives.

 Bonsai: Gradient Boosted Trees + Bayesian Optimization
Bonsai: Gradient Boosted Trees + Bayesian Optimization

Bonsai is a wrapper for the XGBoost and Catboost model training pipelines that leverages Bayesian optimization for computationally efficient hyperparameter tuning.

Pyomo is an object-oriented algebraic modeling language in Python for structured optimization problems.

Pyomo is a Python-based open-source software package that supports a diverse set of optimization capabilities for formulating and analyzing optimization models. Pyomo can be used to define symbolic problems, create concrete problem instances, and solve these instances with standard solvers.

This is an implementation of the proximal policy optimization algorithm for the C++ API of Pytorch
This is an implementation of the proximal policy optimization algorithm for the C++ API of Pytorch

This is an implementation of the proximal policy optimization algorithm for the C++ API of Pytorch. It uses a simple TestEnvironment to test the algorithm

Bayesian optimization based on Gaussian processes (BO-GP) for CFD simulations.

BO-GP Bayesian optimization based on Gaussian processes (BO-GP) for CFD simulations. The BO-GP codes are developed using GPy and GPyOpt. The optimizer

Tools for mathematical optimization region

Tools for mathematical optimization region

Implementation of linesearch Optimization Algorithms in Python

Nonlinear Optimization Algorithms During my time as Scientific Assistant at the Karlsruhe Institute of Technology (Germany) I implemented various Opti

Microsoft contributing libraries, tools, recipes, sample codes and workshop contents for machine learning & deep learning.

Microsoft contributing libraries, tools, recipes, sample codes and workshop contents for machine learning & deep learning.

A data preprocessing package for time series data. Design for machine learning and deep learning.

A data preprocessing package for time series data. Design for machine learning and deep learning.

Owner
Rishabh Iyer
Currently Assistant Prof. at CSE @ UTD. 10+ years experience in Deep Learning, AI and ML. Ph.D. and PostDoc from UW and previously ML Researcher at Microsoft.
Rishabh Iyer
This handbook accompanies the course: Machine Learning with Hung-Yi Lee

This handbook accompanies the course: Machine Learning with Hung-Yi Lee

RenChu Wang 472 Dec 31, 2022
A Python step-by-step primer for Machine Learning and Optimization

early-ML Presentation General Machine Learning tutorials A Python step-by-step primer for Machine Learning and Optimization This github repository gat

Dimitri Bettebghor 8 Dec 1, 2022
A Python Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming.

Master status: Development status: Package information: TPOT stands for Tree-based Pipeline Optimization Tool. Consider TPOT your Data Science Assista

Epistasis Lab at UPenn 8.9k Jan 9, 2023
Python Extreme Learning Machine (ELM) is a machine learning technique used for classification/regression tasks.

Python Extreme Learning Machine (ELM) Python Extreme Learning Machine (ELM) is a machine learning technique used for classification/regression tasks.

Augusto Almeida 84 Nov 25, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques

Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

Vowpal Wabbit 8.1k Dec 30, 2022
CD) in machine learning projectsImplementing continuous integration & delivery (CI/CD) in machine learning projects

CML with cloud compute This repository contains a sample project using CML with Terraform (via the cml-runner function) to launch an AWS EC2 instance

Iterative 19 Oct 3, 2022
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective.

DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. 10x Larger Models 10x Faster Trainin

Microsoft 8.4k Dec 30, 2022
2021 Machine Learning Security Evasion Competition

2021 Machine Learning Security Evasion Competition This repository contains code samples for the 2021 Machine Learning Security Evasion Competition. P

Fabrício Ceschin 8 May 1, 2022
Bayesian optimization in JAX

Bayesian optimization in JAX

Predictive Intelligence Lab 26 May 11, 2022
jaxfg - Factor graph-based nonlinear optimization library for JAX.

Factor graphs + nonlinear optimization in JAX

Brent Yi 134 Dec 21, 2022