# Optimization Methods and Options for Tensor Toolbox

>> Tensor Toolbox >> Optimization Methods and Options

Within the optimization-based functions in Tensor Toolbox like `cp_opt`, we use a variety MATLAB optimization methods from different sources and with different interfaces. For some consisency, these methods are wrapped and given standardized parameters so that it is relatively easy to swap optimization methods. Here we describe the methods, the standardized parameters, installation instructions (if needed), and the mapping between the standardized parameters and the original ones.

- The optimization method is usually specified via the name-value pair with name `'method'`.
- The standardized parameters listed below are additional name-value pairs that can be passed directly to the calling routine.
- The standardized parameters marked with asterisks are not intended to be modified by the user but instead set by the calling routine, such as `cp_opt`.

For more information on the details of these methods, see Developer Information for Optimization Methods in Tensor Toolbox.

## Contents

- `lbfgsb`: Limited-Memory Quasi-Newton with Bound Constraints
- `compact`: Limited-Memory Compact Representation
- `lbfgs`: Limited-Memory Quasi-Newton
- `fminunc`: Optimizaton Toolbox Unconstrained Optimization
- `fmincon`: Optimizaton Toolbox Constrained Optimization
- `adam`: Stochastic Gradient Descent with Momentum

## `lbfgsb`: Limited-Memory Quasi-Newton with Bound Constraints

Source. Included with Tensor Toolbox, adopted without any modification from L-BFGS-B by Stephen Becker.

Standardized Parameters.

Name | Description | Default
`lower` | Lower bounds, can be vector or scalar | `-Inf`
`upper` | Upper bounds, can be vector or scalar | `+Inf`
`maxiters` | Max outer iterations | 1000
`printitn` | Printing frequency by iteration (0=no output) | 1
`m` | Limited-memory parameter | 5
`subiters` | Controls maximum calls to function-gradient evalations | 10
`ftol` | Stopping condition based on relative function change | 1e-10
`gtol` | Stopping condition based on gradient norm | 1e-5
`mdesc` (*) | Method description printed out before run | `'L-BFGS-B Optimization'`
`xdesc` (*) | Variable description | auto-generated

Installation Instructions. Nothing should be required as this code is contained in `libraries/lbfgs`. However, if needed, please see the GitHub site for full details on references, installation, etc. Here we provide cursory instructions for installation:

1. Download the zip file <https://github.com/stephenbeckr/L-BFGS-B-C/archive/master.zip>
2. Unzip and goto the `Matlab/` subdirectoy with MATLAB
3. Type `compile_mex`
4. Add this directory to your saved path!

Mapping of Standardized parameters. The wrapper for this method is `tt_opt_lbfgsb` in the Tensor Toolbox. Notes regarding mappings to the parameters of Becker's L-BFGS-B code:

- `maxIts` maps to `maxiters` and the default is increased from 100 to 1000
- `printEvery` maps to `printitn`
- `maxTotalIts` is set to `maxiters*subiters` and this effectively changes the default from 5000 to 10000
- `factr` is set to `ftol` / eps and this effectively changes the default from 1e7 to 4.5e5
- `pgtol` is set to `gtol`

## `compact`: Limited-Memory Compact Representation

Source. Included with Tensor Toolbox, adopted without any modification from CR Compact quasi-Newton by Johannes Brust with a line-search from Philip Gill

Standardized Parameters.

Name | Description | Default
`maxiters` | Max outer iterations | 1000
`printitn` | Printing frequency by iteration (0=no output) | 1
`m` | Limited-memory parameter | 5
`subiters` | Controls maximum calls to function-gradient evaluations | 10
`gtol` | Stopping condition based on gradient norm | 1e-5
`mdesc` (*) | Method description printed out before run | `'CR solver (limited-memory)'`
`xdesc` (*) | Variable description | auto-generated

Installation Instructions. Nothing should be required as this code is contained in `libraries/compact`. However, if needed, please see the GitHub site for full details. Here we provide cursory instructions for installation:

1. Download the zip file <https://github.com/johannesbrust/CR/archive/master.zip>
2. Unzip and add the `MATLAB/` subdirectoy to your saved path!

Mapping of Standardized parameters. The wrapper for this method is `tt_opt_compact` in the Tensor Toolbox. Notes regarding mappings to the parameters of Brust's code:

- `maxIt` maps to `maxiters`
- `l` maps to `m`
- `maxItLS` maps to `50*subiters`, note that any value for `subiters` is increased by 50
- `tol` maps to `gtol`

## `lbfgs`: Limited-Memory Quasi-Newton

Source. `lbfgs` method in Poblano Toolbox, v1.2.

Standardized Parameters.

Name | Description | Default
`maxiters` | Max outer iterations | 1000
`printitn` | Printing frequency by iteration (0=no output) | 1
`m` | Limited-memory parameter | 5
`subiters` | Controls maximum calls to function-gradient evalations | 10
`ftol` | Stopping condition based on relative function change | 1e-10
`gtol` | Stopping condition based on gradient norm | 1e-5
`mdesc` (*) | Method description printed out before run | `'Poblano L-BFGS Optimization'`
`xdesc` (*) | Variable description | auto-generated

Installation Instructions. Download and install Poblano Toolbox, v1.2. Please see that web page for full details on references, installation, etc. Here we provide cursory instructions for installation:

1. Download the zip file <https://github.com/sandialabs/poblano_toolbox/archive/v1.2.zip>
2. Unzip and goto the `poblano_toolbox-1.2/` subdirectoy within MATLAB
3. Type `install_poblano` to save this directory to your path

Mapping of Standardized parameters. The wrapper for this method is `tt_opt_lbfgs` in the Tensor Toolbox. Notes regarding mappings to the parameters of Poblano's L-BFGS code:

- `MaxIters` maps to `maxiters`
- `Display` maps to `printitn`
- `MaxFuncEvals` is set to `maxiters*subiters`
- `RelFuncTol` is set to `ftol`
- `StopTol` is set to `gtol`

## `fminunc`: Optimizaton Toolbox Unconstrained Optimization

Source. MATLAB Optimization Toolbox.

Standardized Parameters.

Name | Description | Default
`maxiters` | Max outer iterations | 1000
`printitn` | Display (0=no output) | 1
`subiters` | Controls maximum calls to function-gradient evalations | 10
`gtol` | Stopping condition based on gradient norm | 1e-5
`mdesc` (*) | Method description printed out before run | `'Unconstrained Optimization (via Optimization Toolbox)'`
`xdesc` (*) | Variable description | auto-generated

Installation Instructions. Install MATLAB Optimization Toolbox.

Mapping of Standardized parameters.

- `MaxFunctionEvaluations` is set to `maxiters*subiters`
- `MaxIterations` is set to `maxiters`
- `OptimalityTolerance` is set to `gtol`
- `Display` is set of `'off'` if `printitn` = 0, `'iter'` if `printitn` = 1, and `'final'` if `printitn` > 1.

## `fmincon`: Optimizaton Toolbox Constrained Optimization

Source. MATLAB Optimization Toolbox.

Standardized Parameters. Same as for `fminunc` plus the following:

Name | Description | Default
`lower` | Lower bounds, can be vector or scalar | `-Inf`
`upper` | Upper bounds, can be vector or scalar | `+Inf`
`mdesc` (*) | Method description printed out before run | `'Constrained Optimization (via Optimization Toolbox)'`

Installation Instructions. Install MATLAB Optimization Toolbox.

Mapping of Standardized parameters.

- The lower bound (7th input to `fmincon`) is set to `[]` if `lower` = `Inf`. If it's a scalar, then the lower bound is set to a vector of appropriate length with the scalar in every position. Otherwise, it's set to the provided vector.
- The upper bound (8th input) is analogous.

## `adam`: Stochastic Gradient Descent with Momentum

Source. This is our own implementation of Adam. A failed epoch is one where the function value does not decrease. After a failed epoch, the method either reduces the learning rate (by `decay`) or exits (once the number of failed epochs exceeds `maxfails`).

Standardized Parameters.

Name | Description | Default
`lower` | Lower bounds, can be vector or scalar | `-Inf`
`subiters` | Number of iterations per epoch | 100
`maxiters` | Maximum number of epochs | 100
`rate` | Initial learning rate | 1e-2
`maxfails` | Maximum number of failed epochs | 1
`decay` | Decay of learning rate after failed epoch | 0.1
`backup` | Revert to end of previous epoch after failure | true
`ftol` | Quit if function value goes below this value | `-Inf`
`beta1` | Adam parameter | 0.9
`beta2` | Adam parameter | 0.999
`epsilon` | Adam parameter | 1e-8
`printitn` | Printing frequency by epoch (0=no output) | 1
`state` (*) | State of random number generator | current state
`mdesc` (*) | Method description printed out before run | `'Adam Stochastic Optimization'`
`xdesc` (*) | Variable description | auto-generated
`fdesc` (*) | Description of (approximate) function computation | none
`gdesc` (*) | Description of stochastic gradient computation | none
`fexact` (*) | Boolean if function is computed exactly | true