[
  {
    "path": ".readthedocs.yaml",
    "content": "version: 2\n\n# Tell RTD which build image to use and which Python to install\nbuild:\n  os: ubuntu-22.04\n  tools:\n    python: \"3.8\"\n\n# Build from the docs/ directory with Sphinx\nsphinx:\n  configuration: docs/source/conf.py\n\n# Explicitly set the version of Python and its requirements\npython:\n  install:\n    - method: pip\n      path: .\n      extra_requirements:\n        - docs"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2021 Reuben Feinman\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "MANIFEST.in",
    "content": "# Include essential files\ninclude README.md\ninclude LICENSE\ninclude pyproject.toml\n\n# Include source package\nrecursive-include torchmin *.py\n\n# Include tests\nrecursive-include tests *.py\n\n# Exclude unwanted directories\nprune docs\nprune examples\nprune tmp\nglobal-exclude *.pyc\nglobal-exclude *.pyo\nglobal-exclude __pycache__\nglobal-exclude .DS_Store"
  },
  {
    "path": "README.md",
    "content": "# PyTorch Minimize\n\nFor the most up-to-date information on pytorch-minimize, see the docs site: [pytorch-minimize.readthedocs.io](https://pytorch-minimize.readthedocs.io/)\n\nPytorch-minimize represents a collection of utilities for minimizing multivariate functions in PyTorch. \nIt is inspired heavily by SciPy's `optimize` module and MATLAB's [Optimization Toolbox](https://www.mathworks.com/products/optimization.html). \nUnlike SciPy and MATLAB, which use numerical approximations of function derivatives, pytorch-minimize uses _real_ first- and second-order derivatives, computed seamlessly behind the scenes with autograd.\nBoth CPU and CUDA are supported.\n\n__Author__: Reuben Feinman\n\n__At a glance:__\n\n```python\nimport torch\nfrom torchmin import minimize\n\ndef rosen(x):\n    return torch.sum(100*(x[..., 1:] - x[..., :-1]**2)**2 \n                     + (1 - x[..., :-1])**2)\n\n# initial point\nx0 = torch.tensor([1., 8.])\n\n# Select from the following methods:\n#  ['bfgs', 'l-bfgs', 'cg', 'newton-cg', 'newton-exact', \n#   'trust-ncg', 'trust-krylov', 'trust-exact', 'dogleg']\n\n# BFGS\nresult = minimize(rosen, x0, method='bfgs')\n\n# Newton Conjugate Gradient\nresult = minimize(rosen, x0, method='newton-cg')\n\n# Newton Exact\nresult = minimize(rosen, x0, method='newton-exact')\n```\n\n__Solvers:__ BFGS, L-BFGS, Conjugate Gradient (CG), Newton Conjugate Gradient (NCG), Newton Exact, Dogleg, Trust-Region Exact, Trust-Region NCG, Trust-Region GLTR (Krylov)\n\n__Examples:__ See the [Rosenbrock minimization notebook](https://github.com/rfeinman/pytorch-minimize/blob/master/examples/rosen_minimize.ipynb) for a demonstration of function minimization with a handful of different algorithms.\n\n__Install with pip:__\n\n    pip install pytorch-minimize\n\n__Install from source (bleeding edge):__\n\n    pip install git+https://github.com/rfeinman/pytorch-minimize.git\n\n## Motivation\nAlthough PyTorch offers many routines for stochastic optimization, utilities for deterministic optimization are scarce; only L-BFGS is included in the `optim` package, and it's modified for mini-batch training.\n\nMATLAB and SciPy are industry standards for deterministic optimization. \nThese libraries have a comprehensive set of routines; however, automatic differentiation is not supported.* \nTherefore, the user must provide explicit 1st- and 2nd-order gradients (if they are known) or use finite-difference approximations.\n\nThe motivation for pytorch-minimize is to offer a set of tools for deterministic optimization with automatic gradients and GPU acceleration.\n\n__\n\n*MATLAB offers minimal autograd support via the Deep Learning Toolbox, but the integration is not seamless: data must be converted to \"dlarray\" structures, and only a [subset of functions](https://www.mathworks.com/help/deeplearning/ug/list-of-functions-with-dlarray-support.html) are supported.\nFurthermore, derivatives must still be constructed and provided as function handles. \nPytorch-minimize uses autograd to compute derivatives behind the scenes, so all you provide is an objective function.\n\n## Library\n\nThe pytorch-minimize library includes solvers for general-purpose function minimization (unconstrained & constrained), as well as for nonlinear least squares problems.\n\n### 1. Unconstrained Minimizers\n\nThe following solvers are available for _unconstrained_ minimization:\n\n- __BFGS/L-BFGS.__ BFGS is a cannonical quasi-Newton method for unconstrained optimization. I've implemented both the standard BFGS and the \"limited memory\" L-BFGS. For smaller scale problems where memory is not a concern, BFGS should be significantly faster than L-BFGS (especially on CUDA) since it avoids Python for loops and instead uses pure torch.\n\n- __Conjugate Gradient (CG).__ The conjugate gradient algorithm is a generalization of linear conjugate gradient to nonlinear optimization problems. Pytorch-minimize includes an implementation of the Polak-Ribiére CG algorithm described in Nocedal & Wright (2006) chapter 5.2.\n   \n- __Newton Conjugate Gradient (NCG).__ The Newton-Raphson method is a staple of unconstrained optimization. Although computing full Hessian matrices with PyTorch's reverse-mode automatic differentiation can be costly, computing Hessian-vector products is cheap, and it also saves a lot of memory. The Conjugate Gradient (CG) variant of Newton's method is an effective solution for unconstrained minimization with Hessian-vector products. I've implemented a lightweight NewtonCG minimizer that uses HVP for the linear inverse sub-problems.\n\n- __Newton Exact.__ In some cases, we may prefer a more precise variant of the Newton-Raphson method at the cost of additional complexity. I've also implemented an \"exact\" variant of Newton's method that computes the full Hessian matrix and uses Cholesky factorization for linear inverse sub-problems. When Cholesky fails--i.e. the Hessian is not positive definite--the solver resorts to one of two options as specified by the user: 1) steepest descent direction (default), or 2) solve the inverse hessian with LU factorization.\n\n- __Trust-Region Newton Conjugate Gradient.__ Description coming soon.\n\n- __Trust-Region Newton Generalized Lanczos (Krylov).__ Description coming soon.\n\n- __Trust-Region Exact.__ Description coming soon.\n\n- __Dogleg.__ Description coming soon.\n\nTo access the unconstrained minimizer interface, use the following import statement:\n\n    from torchmin import minimize\n\nUse the argument `method` to specify which of the afformentioned solvers should be applied.\n\n### 2. Constrained Minimizers\n\nThe following solvers are available for _constrained_ minimization:\n\n- __Trust-Region Constrained Algorithm.__ Pytorch-minimize includes a single constrained minimization routine based on SciPy's 'trust-constr' method. The algorithm accepts generalized nonlinear constraints and variable boundries via the \"constr\" and \"bounds\" arguments. For equality constrained problems, it is an implementation of the Byrd-Omojokun Trust-Region SQP method. When inequality constraints are imposed, the trust-region interior point method is used. NOTE: The current trust-region constrained minimizer is not a custom implementation, but rather a wrapper for SciPy's `optimize.minimize` routine. It uses autograd behind the scenes to build jacobian & hessian callables before invoking scipy. Inputs and objectivs should use torch tensors like other pytorch-minimize routines. CUDA is supported but not recommended; data will be moved back-and-forth between GPU/CPU. \n   \nTo access the constrained minimizer interface, use the following import statement:\n\n    from torchmin import minimize_constr\n\n### 3. Nonlinear Least Squares\n\nThe library also includes specialized solvers for nonlinear least squares problems. \nThese solvers revolve around the Gauss-Newton method, a modification of Newton's method tailored to the lstsq setting. \nThe least squares interface can be imported as follows:\n\n    from torchmin import least_squares\n\nThe least_squares function is heavily motivated by scipy's `optimize.least_squares`. \nMuch of the scipy code was borrowed directly (all rights reserved) and ported from numpy to torch. \nRather than have the user provide a jacobian function, in the new interface, jacobian-vector products are computed behind the scenes with autograd. \nAt the moment, only the Trust Region Reflective (\"trf\") method is implemented, and bounds are not yet supported.\n\n## Examples\n\nThe [Rosenbrock minimization tutorial](https://github.com/rfeinman/pytorch-minimize/blob/master/examples/rosen_minimize.ipynb) demonstrates how to use pytorch-minimize to find the minimum of a scalar-valued function of multiple variables using various optimization strategies.\n\nIn addition, the [SciPy benchmark](https://github.com/rfeinman/pytorch-minimize/blob/master/examples/scipy_benchmark.py) provides a comparison of pytorch-minimize solvers to their analogous solvers from the `scipy.optimize` library. \nFor those transitioning from scipy, this script will help get a feel for the design of the current library. \nUnlike scipy, jacobian and hessian functions need not be provided to pytorch-minimize solvers, and numerical approximations are never used.\n\nFor constrained optimization, the [adversarial examples tutorial](https://github.com/rfeinman/pytorch-minimize/blob/master/examples/constrained_optimization_adversarial_examples.ipynb) demonstrates how to use the trust-region constrained routine to generate an optimal adversarial perturbation given a constraint on the perturbation norm.\n\n## Optimizer API\n\nAs an alternative to the functional API, pytorch-minimize also includes an \"optimizer\" API based on the `torch.optim.Optimizer` class. \nTo access the optimizer class, import as follows:\n\n    from torchmin import Minimizer\n\n## Citing this work\n\nIf you use pytorch-minimize for academic research, you may cite the library as follows:\n\n```\n@misc{Feinman2021,\n  author = {Feinman, Reuben},\n  title = {Pytorch-minimize: a library for numerical optimization with autograd},\n  publisher = {GitHub},\n  year = {2021},\n  url = {https://github.com/rfeinman/pytorch-minimize},\n}\n```\n"
  },
  {
    "path": "docs/Makefile",
    "content": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the environment for the first two.\nSPHINXOPTS    ?=\nSPHINXBUILD   ?= sphinx-build\nSOURCEDIR     = source\nBUILDDIR      = build\n\n# Put it first so that \"make\" without argument is like \"make help\".\nhelp:\n\t@$(SPHINXBUILD) -M help \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n\n.PHONY: help Makefile\n\n# Catch-all target: route all unknown targets to Sphinx using the new\n# \"make mode\" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).\n%: Makefile\n\t@$(SPHINXBUILD) -M $@ \"$(SOURCEDIR)\" \"$(BUILDDIR)\" $(SPHINXOPTS) $(O)\n"
  },
  {
    "path": "docs/make.bat",
    "content": "@ECHO OFF\r\n\r\npushd %~dp0\r\n\r\nREM Command file for Sphinx documentation\r\n\r\nif \"%SPHINXBUILD%\" == \"\" (\r\n\tset SPHINXBUILD=sphinx-build\r\n)\r\nset SOURCEDIR=source\r\nset BUILDDIR=build\r\n\r\nif \"%1\" == \"\" goto help\r\n\r\n%SPHINXBUILD% >NUL 2>NUL\r\nif errorlevel 9009 (\r\n\techo.\r\n\techo.The 'sphinx-build' command was not found. Make sure you have Sphinx\r\n\techo.installed, then set the SPHINXBUILD environment variable to point\r\n\techo.to the full path of the 'sphinx-build' executable. Alternatively you\r\n\techo.may add the Sphinx directory to PATH.\r\n\techo.\r\n\techo.If you don't have Sphinx installed, grab it from\r\n\techo.http://sphinx-doc.org/\r\n\texit /b 1\r\n)\r\n\r\n%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\ngoto end\r\n\r\n:help\r\n%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%\r\n\r\n:end\r\npopd\r\n"
  },
  {
    "path": "docs/source/_static/custom.css",
    "content": ".wy-table-responsive table td {\n    white-space: normal;\n}"
  },
  {
    "path": "docs/source/api/index.rst",
    "content": "=================\nAPI Documentation\n=================\n\n.. currentmodule:: torchmin\n\n\nFunctional API\n==============\n\nThe functional API provides an interface similar to those of SciPy's :mod:`optimize` module and MATLAB's ``fminunc``/``fmincon`` routines. Parameters are provided as a single torch Tensor, and an :class:`OptimizeResult` instance is returned that includes the optimized parameter value as well as other useful information (e.g. final function value, parameter gradient, etc.).\n\nThere are 3 core utilities in the functional API, designed for 3 unique\nnumerical optimization problems.\n\n\n**Unconstrained minimization**\n\n.. autosummary::\n    :toctree: generated\n\n    minimize\n\nThe :func:`minimize` function is a general utility for *unconstrained* minimization. It implements a number of different routines based on Newton and Quasi-Newton methods for numerical optimization. The following methods are supported, accessed via the `method` argument:\n\n.. toctree::\n\n    minimize-bfgs\n    minimize-lbfgs\n    minimize-cg\n    minimize-newton-cg\n    minimize-newton-exact\n    minimize-dogleg\n    minimize-trust-ncg\n    minimize-trust-exact\n    minimize-trust-krylov\n\n\n**Constrained minimization**\n\n.. autosummary::\n    :toctree: generated\n\n    minimize_constr\n\nThe :func:`minimize_constr` function is a general utility for *constrained* minimization. Algorithms for constrained minimization use Newton and Quasi-Newton methods on the KKT conditions of the constrained optimization problem. The following methods are currently supported:\n\n.. toctree::\n\n    minimize-constr-lbfgsb\n    minimize-constr-frankwolfe\n    minimize-constr-trust-constr\n\n\n.. note::\n    Method ``'trust-constr'`` is currently a wrapper for SciPy's *trust-constr* minimization method. CUDA tensors are supported, but CUDA will only be used for function and gradient evaluation, with the remaining solver computations performed on CPU (with numpy arrays).\n\n\n**Nonlinear least-squares**\n\n.. autosummary::\n    :toctree: generated\n\n    least_squares\n\nThe :func:`least_squares` function is a specialized utility for nonlinear least-squares minimization problems. Algorithms for least-squares revolve around the Gauss-Newton method, a modification of Newton's method tailored to residual sum-of-squares (RSS) optimization. The following methods are currently supported:\n\n- Trust-region reflective\n- Dogleg - COMING SOON\n- Gauss-Newton line search - COMING SOON\n\n\n\nOptimizer API\n==============\n\nThe optimizer API provides an alternative interface based on PyTorch's :mod:`optim` module. This interface follows the schematic of PyTorch optimizers and will be familiar to those migrating from torch.\n\n.. autosummary::\n    :toctree: generated\n\n    Minimizer\n    Minimizer.step\n\nThe :class:`Minimizer` class inherits from :class:`torch.optim.Optimizer` and constructs an object that holds the state of the provided variables. Unlike the functional API, which expects parameters to be a single Tensor, parameters can be passed to :class:`Minimizer` as iterables of Tensors. The class serves as a wrapper for :func:`torchmin.minimize()` and can use any of its methods (selected via the `method` argument) to perform unconstrained minimization.\n\n.. autosummary::\n    :toctree: generated\n\n    ScipyMinimizer\n    ScipyMinimizer.step\n\nAlthough the :class:`Minimizer` class will be sufficient for most problems where torch optimizers would be used, it does not support constraints. Another optimizer is provided, :class:`ScipyMinimizer`, which supports parameter bounds and linear/nonlinear constraint functions. This optimizer is a wrapper for :func:`scipy.optimize.minimize`. When using bound constraints, `bounds` are passed as iterables with same length as `params`, i.e. one bound specification per parameter Tensor.\n"
  },
  {
    "path": "docs/source/api/minimize-bfgs.rst",
    "content": "minimize(method='bfgs')\n----------------------------------------\n\n.. autofunction:: torchmin.bfgs._minimize_bfgs"
  },
  {
    "path": "docs/source/api/minimize-cg.rst",
    "content": "minimize(method='cg')\n----------------------------------------\n\n.. autofunction:: torchmin.cg._minimize_cg"
  },
  {
    "path": "docs/source/api/minimize-constr-frankwolfe.rst",
    "content": "minimize_constr(method='frank-wolfe')\n----------------------------------------\n\n.. autofunction:: torchmin.constrained.frankwolfe._minimize_frankwolfe"
  },
  {
    "path": "docs/source/api/minimize-constr-lbfgsb.rst",
    "content": "minimize_constr(method='l-bfgs-b')\n----------------------------------------\n\n.. autofunction:: torchmin.constrained.lbfgsb._minimize_lbfgsb"
  },
  {
    "path": "docs/source/api/minimize-constr-trust-constr.rst",
    "content": "minimize_constr(method='trust-constr')\n----------------------------------------\n\n.. autofunction:: torchmin.constrained.trust_constr._minimize_trust_constr"
  },
  {
    "path": "docs/source/api/minimize-dogleg.rst",
    "content": "minimize(method='dogleg')\n----------------------------------------\n\n.. autofunction:: torchmin.trustregion._minimize_dogleg"
  },
  {
    "path": "docs/source/api/minimize-lbfgs.rst",
    "content": "minimize(method='l-bfgs')\n----------------------------------------\n\n.. autofunction:: torchmin.bfgs._minimize_lbfgs"
  },
  {
    "path": "docs/source/api/minimize-newton-cg.rst",
    "content": "minimize(method='newton-cg')\n----------------------------------------\n\n.. autofunction:: torchmin.newton._minimize_newton_cg"
  },
  {
    "path": "docs/source/api/minimize-newton-exact.rst",
    "content": "minimize(method='newton-exact')\n----------------------------------------\n\n.. autofunction:: torchmin.newton._minimize_newton_exact"
  },
  {
    "path": "docs/source/api/minimize-trust-exact.rst",
    "content": "minimize(method='trust-exact')\n----------------------------------------\n\n.. autofunction:: torchmin.trustregion._minimize_trust_exact"
  },
  {
    "path": "docs/source/api/minimize-trust-krylov.rst",
    "content": "minimize(method='trust-krylov')\n----------------------------------------\n\n.. autofunction:: torchmin.trustregion._minimize_trust_krylov"
  },
  {
    "path": "docs/source/api/minimize-trust-ncg.rst",
    "content": "minimize(method='trust-ncg')\n----------------------------------------\n\n.. autofunction:: torchmin.trustregion._minimize_trust_ncg"
  },
  {
    "path": "docs/source/conf.py",
    "content": "# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common options. For a full\n# list see the documentation:\n# https://www.sphinx-doc.org/en/master/usage/configuration.html\n\n# -- Path setup --------------------------------------------------------------\n\n# If extensions (or modules to document with autodoc) are in another directory,\n# add these directories to sys.path here. If the directory is relative to the\n# documentation root, use os.path.abspath to make it absolute, like shown here.\n#\nimport os\nimport sys\nsys.path.insert(0, os.path.abspath('../../'))\n\nimport torchmin\n\n\n# -- Project information -----------------------------------------------------\n\nproject = 'pytorch-minimize'\ncopyright = '2021, Reuben Feinman'\nauthor = 'Reuben Feinman'\n\n# The full version, including alpha/beta/rc tags\nrelease = torchmin.__version__\n\n\n# -- General configuration ---------------------------------------------------\n\n# Add any Sphinx extension module names here, as strings. They can be\n# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom\n# ones.\nimport sphinx_rtd_theme\n\nextensions = [\n    'sphinx.ext.autodoc',\n    'sphinx.ext.autosummary',\n    'sphinx.ext.doctest',\n    'sphinx.ext.intersphinx',\n    'sphinx.ext.todo',\n    'sphinx.ext.coverage',\n    'sphinx.ext.mathjax',\n    'sphinx.ext.napoleon',\n    'sphinx.ext.viewcode',\n    'sphinx.ext.autosectionlabel',\n    'sphinx_rtd_theme'\n]\n\n# autosectionlabel throws warnings if section names are duplicated.\n# The following tells autosectionlabel to not throw a warning for\n# duplicated section names that are in different documents.\nautosectionlabel_prefix_document = True\n\n\n# Add any paths that contain templates here, relative to this directory.\ntemplates_path = ['_templates']\n\n# List of patterns, relative to source directory, that match files and\n# directories to ignore when looking for source files.\n# This pattern also affects html_static_path and html_extra_path.\nexclude_patterns = []\n\n# ==== Customizations ====\n\n# Disable displaying type annotations, these can be very verbose\nautodoc_typehints = 'none'\n\n# build the templated autosummary files\nautosummary_generate = True\n#numpydoc_show_class_members = False\n\n# Enable overriding of function signatures in the first line of the docstring.\n#autodoc_docstring_signature = True\n\n\n\n# -- Options for HTML output -------------------------------------------------\n\n# The theme to use for HTML and HTML Help pages.  See the documentation for\n# a list of builtin themes.\n#\n#html_theme = 'alabaster'\nhtml_theme = 'sphinx_rtd_theme'  # addition\n\n# Add any paths that contain custom static files (such as style sheets) here,\n# relative to this directory. They are copied after the builtin static files,\n# so a file named \"default.css\" will overwrite the builtin \"default.css\".\nhtml_static_path = ['_static']\n\n\n# ==== Customizations ====\n\n# Called automatically by Sphinx, making this `conf.py` an \"extension\".\ndef setup(app):\n    # at the moment, we use custom.css to specify a maximum with for tables,\n    # such as those generated by autosummary.\n    app.add_css_file('custom.css')\n"
  },
  {
    "path": "docs/source/examples/index.rst",
    "content": "Examples\n=========\n\nThe examples site is in active development. Check back soon for more complete examples of how to use pytorch-minimize.\n\nUnconstrained minimization\n---------------------------\n\n.. code-block:: python\n\n    from torchmin import minimize\n    from torchmin.benchmarks import rosen\n\n    # initial point\n    x0 = torch.randn(100, device='cpu')\n\n    # BFGS\n    result = minimize(rosen, x0, method='bfgs')\n\n    # Newton Conjugate Gradient\n    result = minimize(rosen, x0, method='newton-cg')\n\nConstrained minimization\n---------------------------\n\nFor constrained optimization, the `adversarial examples tutorial <https://github.com/rfeinman/pytorch-minimize/blob/master/examples/constrained_optimization_adversarial_examples.ipynb>`_ demonstrates how to use trust-region constrained optimization to generate an optimal adversarial perturbation given a constraint on the perturbation norm.\n\nNonlinear least-squares\n---------------------------\n\nComing soon.\n\n\nScipy benchmark\n---------------------------\n\nThe `SciPy benchmark <https://github.com/rfeinman/pytorch-minimize/blob/master/examples/scipy_benchmark.py>`_ provides a comparison of pytorch-minimize solvers to their analogous solvers from the :mod:`scipy.optimize` module.\nFor those transitioning from scipy, this script will help get a feel for the design of the current library.\nUnlike scipy, jacobian and hessian functions need not be provided to pytorch-minimize solvers, and numerical approximations are never used.\n\n\nMinimizer (optimizer API)\n---------------------------\n\nAnother way to use the optimization tools from pytorch-minimize is via :class:`torchmin.Minimizer`, a pytorch Optimizer class. For a demo on how to use the Minimizer class, see the `MNIST classifier <https://github.com/rfeinman/pytorch-minimize/blob/master/examples/train_mnist_Minimizer.py>`_ tutorial."
  },
  {
    "path": "docs/source/index.rst",
    "content": "Pytorch-minimize\n================\n\nPytorch-minimize is a library for numerical optimization with automatic differentiation and GPU acceleration. It implements a number of canonical techniques for deterministic (or \"full-batch\") optimization not offered in the :mod:`torch.optim` module. The library is inspired heavily by SciPy's :mod:`optimize` module and MATLAB's `Optimization Toolbox <https://www.mathworks.com/products/optimization.html>`_. Unlike SciPy and MATLAB, which use numerical approximations of derivatives that are slow and often inaccurate, pytorch-minimize uses *real* first- and second-order derivatives, computed seamlessly behind the scenes with autograd. Both CPU and CUDA are supported.\n\n:Author: Reuben Feinman\n:Version: |release|\n\nPytorch-minimize is currently in Beta; expect the API to change before a first official release. Some of the source code was taken directly from SciPy and ported to PyTorch. As such, here is their copyright notice:\n\n    Copyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers. All rights reserved.\n\n\nTable of Contents\n=================\n\n.. toctree::\n    :maxdepth: 2\n\n    install\n\n.. toctree::\n    :maxdepth: 2\n\n    user_guide/index\n\n.. toctree::\n    :maxdepth: 2\n\n    api/index\n\n.. toctree::\n    :maxdepth: 2\n\n    examples/index\n"
  },
  {
    "path": "docs/source/install.rst",
    "content": "Install\n===========\n\nTo install pytorch-minimize, users may either 1) install the official PyPI release via pip, or 2) install a *bleeding edge* distribution from source.\n\n\n**Install via pip (official PyPI release)**::\n\n    pip install pytorch-minimize\n\n**Install from source (bleeding edge)**::\n\n    pip install git+https://github.com/rfeinman/pytorch-minimize.git\n\n\n**PyTorch requirement**\n\nThis library uses latest features from the actively-developed :mod:`torch.linalg` module. For maximum performance, users should install pytorch>=1.9, as it includes some new items not available in prior releases (e.g. `torch.linalg.cholesky_ex <https://pytorch.org/docs/stable/generated/torch.linalg.cholesky_ex.html>`_). Pytorch-minimize will automatically use these features when available.\n"
  },
  {
    "path": "docs/source/user_guide/index.rst",
    "content": "===========\nUser Guide\n===========\n\n.. currentmodule:: torchmin\n\nUsing the :func:`minimize` function\n------------------------------------\n\nComing soon."
  },
  {
    "path": "examples/constrained_optimization_adversarial_examples.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 1,\n   \"id\": \"dried-niagara\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"%matplotlib inline\\n\",\n    \"import matplotlib.pylab as plt\\n\",\n    \"import torch\\n\",\n    \"import torch.nn as nn\\n\",\n    \"import torch.nn.functional as F\\n\",\n    \"import torch.optim as optim\\n\",\n    \"from torch.utils.data import DataLoader\\n\",\n    \"from torchvision import transforms, datasets\\n\",\n    \"\\n\",\n    \"from torchmin import minimize_constr\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"id\": \"whole-fifty\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"device = torch.device('cuda:0')\\n\",\n    \"\\n\",\n    \"root = '/path/to/data'  # fill in torchvision dataset path\\n\",\n    \"train_data = datasets.MNIST(root, train=True, transform=transforms.ToTensor())\\n\",\n    \"train_loader = DataLoader(train_data, batch_size=128, shuffle=True)\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"closed-interview\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Train CNN classifier\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 3,\n   \"id\": \"following-knowing\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def CNN():\\n\",\n    \"    return nn.Sequential(\\n\",\n    \"        nn.Conv2d(1, 10, kernel_size=5),\\n\",\n    \"        nn.SiLU(),\\n\",\n    \"        nn.AvgPool2d(2),\\n\",\n    \"        nn.Conv2d(10, 20, kernel_size=5),\\n\",\n    \"        nn.SiLU(),\\n\",\n    \"        nn.AvgPool2d(2),\\n\",\n    \"        nn.Dropout(0.2),\\n\",\n    \"        nn.Flatten(1),\\n\",\n    \"        nn.Linear(320, 50),\\n\",\n    \"        nn.Dropout(0.2),\\n\",\n    \"        nn.Linear(50, 10),\\n\",\n    \"        nn.LogSoftmax(1)\\n\",\n    \"    )\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"id\": \"accessory-killer\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"epoch  1 - loss: 0.4923\\n\",\n      \"epoch  2 - loss: 0.1428\\n\",\n      \"epoch  3 - loss: 0.1048\\n\",\n      \"epoch  4 - loss: 0.0883\\n\",\n      \"epoch  5 - loss: 0.0754\\n\",\n      \"epoch  6 - loss: 0.0672\\n\",\n      \"epoch  7 - loss: 0.0626\\n\",\n      \"epoch  8 - loss: 0.0578\\n\",\n      \"epoch  9 - loss: 0.0524\\n\",\n      \"epoch 10 - loss: 0.0509\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"torch.manual_seed(382)\\n\",\n    \"net = CNN().to(device)\\n\",\n    \"optimizer = optim.Adam(net.parameters())\\n\",\n    \"for epoch in range(10):\\n\",\n    \"    epoch_loss = 0\\n\",\n    \"    for (x, y) in train_loader:\\n\",\n    \"        x = x.to(device, non_blocking=True)\\n\",\n    \"        y = y.to(device, non_blocking=True)\\n\",\n    \"        logits = net(x)\\n\",\n    \"        loss = F.nll_loss(logits, y)\\n\",\n    \"        optimizer.zero_grad(set_to_none=True)\\n\",\n    \"        loss.backward()\\n\",\n    \"        optimizer.step()\\n\",\n    \"        epoch_loss += loss.item() * x.size(0)\\n\",\n    \"    print('epoch %2d - loss: %0.4f' % (epoch+1, epoch_loss / len(train_data)))\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"therapeutic-elimination\",\n   \"metadata\": {},\n   \"source\": [\n    \"# set up adversarial example environment\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"id\": \"developing-afghanistan\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"# evaluation mode settings\\n\",\n    \"net = net.requires_grad_(False).eval()\\n\",\n    \"\\n\",\n    \"# move net to CPU\\n\",\n    \"# Note: using CUDA-based inputs and objectives is allowed\\n\",\n    \"# but inefficient with trust-constr, as the data will be\\n\",\n    \"# moved back-and-forth from CPU\\n\",\n    \"net = net.cpu()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"id\": \"mighty-realtor\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"def nll_objective(x, y):\\n\",\n    \"    assert x.numel() == 28**2\\n\",\n    \"    assert y.numel() == 1\\n\",\n    \"    x = x.view(1, 1, 28, 28)\\n\",\n    \"    return F.nll_loss(net(x), y.view(1))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"id\": \"better-nerve\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"# select a random image from the dataset\\n\",\n    \"torch.manual_seed(338)\\n\",\n    \"x, y = next(iter(train_loader))\\n\",\n    \"img = x[0]\\n\",\n    \"label = y[0]\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 9,\n   \"id\": \"presidential-astrology\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"tensor(1.4663e-05)\"\n      ]\n     },\n     \"execution_count\": 9,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"nll_objective(img, label)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"id\": \"independent-slovenia\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"# minimization objective for adversarial examples\\n\",\n    \"#   goal is to maximize NLL of perturbed image (image + perturbation)\\n\",\n    \"fn = lambda eps: - nll_objective(img + eps, label)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 18,\n   \"id\": \"bacterial-champagne\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"# plotting utility\\n\",\n    \"\\n\",\n    \"def plot_distortion(img, eps, y):\\n\",\n    \"    assert img.numel() == 28**2\\n\",\n    \"    assert eps.numel() == 28**2\\n\",\n    \"    img = img.view(28, 28)\\n\",\n    \"    img_ = img + eps.view(28, 28)\\n\",\n    \"    fig, axes = plt.subplots(1,2,figsize=(4,2))\\n\",\n    \"    for i, x in enumerate((img, img_)):\\n\",\n    \"        axes[i].imshow(x.cpu(), cmap=plt.cm.binary)\\n\",\n    \"        axes[i].set_xticks([])\\n\",\n    \"        axes[i].set_yticks([])\\n\",\n    \"        axes[i].set_title('nll: %0.4f' % nll_objective(x, y))\\n\",\n    \"    plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"id\": \"ambient-thread\",\n   \"metadata\": {},\n   \"source\": [\n    \"# craft adversarial example\\n\",\n    \"\\n\",\n    \"We will use our constrained optimizer to find the optimal unit-norm purturbation $\\\\epsilon$ \\n\",\n    \"\\n\",\n    \"\\\\begin{equation}\\n\",\n    \"\\\\max_{\\\\epsilon} NLL(x + \\\\epsilon) \\\\quad \\\\text{s.t.} \\\\quad ||\\\\epsilon|| = 1\\n\",\n    \"\\\\end{equation}\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 13,\n   \"id\": \"surprised-symposium\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"torch.manual_seed(227)\\n\",\n    \"eps0 = torch.randn_like(img)\\n\",\n    \"eps0 /= eps0.norm()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"id\": \"missing-bargain\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"-2.2291887944447808e-05\"\n      ]\n     },\n     \"execution_count\": 14,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"fn(eps0).item()\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"id\": \"miniature-fight\",\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"`xtol` termination condition is satisfied.\\n\",\n      \"Number of iterations: 32, function evaluations: 50, CG iterations: 52, optimality: 1.02e-04, constraint violation: 0.00e+00, execution time: 0.57 s.\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"res = minimize_constr(\\n\",\n    \"    fn, eps0, \\n\",\n    \"    max_iter=100,\\n\",\n    \"    constr=dict(\\n\",\n    \"        fun=lambda x: x.square().sum(), \\n\",\n    \"        lb=1, ub=1\\n\",\n    \"    ),\\n\",\n    \"    disp=1\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"id\": \"wanted-journal\",\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"tensor(1.)\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"eps = res.x\\n\",\n    \"print(eps.norm())\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 19,\n   \"id\": \"spanish-wright\",\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAPEAAACHCAYAAADHsL/VAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAOOUlEQVR4nO2de4xV1RXGvyXKGwVkBDsRBiFSA5QmQJsoKg8fRSEytdSmQCltItgYYgAjoaFgbXnUxCb9g2hMFCehtYrQClWIMRBBQyoJCExiQQggwQoDhfASVHb/uHdO916ds889973nfr9kkrVY++yz7xzWnPXdvc8+YowBISRcrqn0AAghhcEkJiRwmMSEBA6TmJDAYRITEjhMYkICpyqTWETGisgxyz8sIvdWckyk+PA6F4eqTOJCkAwrReRU9ucPIiKe9hNE5BMRuSgiW0RkQK59iUhD9piL2T7uVX3/VESOiMgFEfmbiPQuzaeuPdJcZxHpKCJrs38kjIiMVfGeIvKqiJzI/iy1Yv1F5Lz6MSIyPxtfpGKXROSqiPQp4cd3aHdJDOAxAFMAjADwHQCTAMxuq2H2F70OwGIAvQHsBPDXFH39BcAuADcC+DWAtSJSl+17KIAXAcwA0BfARQCrCv94JEvO1znLdgDTAfy7jdgfAXQF0ADgewBmiMgsADDGHDXGdG/9ATAcwFUAb2bjy1R8JYCtxpiWwj9ijhhjKvID4DCABQD2ADiLTPJ0zsbGAjim2t6bY78fAnjM8n8JYEdM28cAfGj53QBcAvDtpL4A3AbgMoAeVnwbgDlZexmAP1uxQQCu2O1r4acarrM67hiAserfWgCMtvxFALbFHL8EwJaYmAA4CGBmOX/Hlb4T/xjADwAMROav6c+TDhCRMSJyxtNkKICPLf/j7L8ltjXGXEDmIgxtK676GgrgkDHmnCdu930QmSS+zTP29kqlr3MuiLKHxbT7GYBXY2J3IVN1vVnAOFJT6ST+kzHmuDHmNIANAL6bdIAxZrsxpqenSXdk/uK3chZA9xi9pNu2tu+RQ19pj9XxWqLS1zmJTQAWikgPERkM4BfIlNcOItKapGtj+pkJYK0x5nweY8ibSiexrU8uInNhCuU8gOst/3oA50223klo29r+XEzc7ivtsTpeS1T6OicxFxkZdQDA35H5ruNYG+1mAnizrSQVkS4ApiL+Ll0yKp3EpaAZmS87WhmR/bfEtiLSDRnt2txWXPXVDOBWEenhidt93wqgE4D9KT4LiSfNdfZijDltjJlmjOlnjBmKTF78026TQ5L+EMBpAFvzGUMhtMckbgIwT0TqReRbAOYDWB3Tdj2AYSLyiIh0BvAbAHuMMZ8k9WWM2Q9gN4AlItJZRBqR0XutemgNgMkiclf2j8NvAaxTGprkT5rrDBHplL3GANAxe80kGxskIjeKSAcRmYjMF56/U100AjgDYEvMKWYCaMqzEiiI4JI4mxQ+zfEiMrprL4B9AP6R/bfW45tFZBoAGGNOAngEwO8B/AfA9wH8JNe+sm1HZY9dAeBH2T5hjGkGMAeZZD6BjBb+VV4fugYp5nXO8i9kSuZ6AJuzduuagJHZfs4BWA5gWvb62cQmqYjUAxiPzB+WsiMV+MNBCCkiwd2JCSEuTGJCAodJTEjgMIkJCRwmMSGBc22axn369DENDQ0lGgrJh8OHD6OlpSWfpYZt0qtXL1NfX1+s7oIlv9WbfgqZCWpubm4xxtS1FUuVxA0NDdi5c2feAyHFZ9SoUUXtr76+HuvWrcupbTmmJ0uRTJU6byG/ryFDhhyJi7GcJiRwUt2JSW2T5k6Spq3vrqf7KdWd2ddvuaqBfO/UvBMTEjhMYkICh+U0KQm6BLVLxTTlaZq2hZT7+jy2X6ySvlRfBPJOTEjgMIkJCRwmMSGBQ01MysI11xTnfuHTlTqWpq1PE2vy1fRJmjhfrc07MSGBwyQmJHCYxIQEDjUxyRnf3K+O+zRmkvaz+03Sub6233zzTU7naMvPdbwdOnSIjQHpvgvgsktCahQmMSGBw3Ka5I2vZNZlpl1WJpXTV69ejWxdEieV9HHn1P2mmY7S+EroUi0p9cE7MSGBwyQmJHCYxIQEDjUx8ZLmEUJbg2rdeO21//uvlqRrv/7668i2dWwShehRn6+1te3rWLl2InHGU/IzEEJKCpOYkMBp1+X0jh07IvvTTz91YnbJBgCzZs0q+vl1nzNmzHD8cePGFf2c5USXknYJbZfPOqbR18KeVtJTTLq83rNnT2QfO3bMiX311VeOv2TJkshOmlLySQObKVOmOP6DDz7o+L4thYtVevNOTEjgMIkJCRwmMSGBUxWa+PLly5H99NNPOzGtZdOwd+/eyP7ss8+8bUsxFbB69WrHf/vttx3/tddei+yxY8cW/fzFJs20jObSpUuR/fzzzzuxgwcPOr6tg5O0q33syZMnnZg+1tbpSVNXtg72aeKNGzc6/vvvv+/4y5Yti+zRo0d7z5kvvBMTEjhMYkICh0lMSOBURBNv2LDB8desWRPZr7/+ermHUzZOnDjh+J9//nmFRpI7ue7WAbga9L333nNi9jV/5513nJhvBw59Dt9yTh3zPW6YtONGro9Oam2tdbl9zZN0eJrHNZ3jcm5JCKlKmMSEBE7ZyunNmzdH9syZM53YmTNninKOAQMGOL6vZOrYsaPjv/LKK7Ft9dTQc889F9n29FgSd955p+Pfd999OR9bjehyddu2bZG9aNEiJ3b27NnI1mWlr2TWJfLNN98c21b3q4995plnYs+5fft2x29qaops37JQHRs+fLjj29NKpXrCiXdiQgKHSUxI4DCJCQmcsmli+6v3QjTwnDlzIruurs6JPfXUU47fvXv3vM9jo7XMyy+/HNnHjx/PuR/9XUCfPn0KG1iJiNvNI0nLnjp1KrJtDazRjynq7y6mTp0a2b169XJi06dPd/zOnTvHjk9fN9+OHJq33norsltaWpyYbzfOhx56yPFvuOGGyC5k2aoP3okJCRwmMSGBwyQmJHAqsuxSa6IRI0ZEtta5+rG1gQMHRnanTp1KMDrgwIEDjv/GG284vk8H2zrcfgwNKM0WQKUmzYu6bfQc7ZAhQyK7d+/eTmz+/PmO369fv9h+9Lys/YhjmmWN+tFUWwMD7pLY6667zol17do1sh9//HEn1tjYGHvOYmlgDe/EhAQOk5iQwClbOT1y5MjI1iXyE088Ua5h5MTs2bMdf+vWrTkfu3LlysjWpVZ75/bbb4/sBQsWOLFHH300spPeI2yXzHpZq25r95U0xWSzePFix7d3zQSALl26RLaejpo3b15k690u07wjWcP3ExNSozCJCQkcJjEhgVM2TWzrJduuFuxpo0KWhU6cOLEIowkDPUUyaNCgyB48eLATs/Wq1q5aN9q+fouDnmKy40kvDj99+nRkX7hwwYnZGhgAunXrFtl6isl+c4eeAtOfxY4nLfWkJiakRmESExI4TGJCAqcq3gBRDdg7MO7evbtyA2kn+Lbc0drQt/wwSRPb88haa2st+9FHH0W2XnbZs2dPx7e3b9JjsHVvmqWeSZrY7iuNPuadmJDAYRITEjg1W07rqYBdu3bl1c+wYcMc337CpdbwvZjbfnJNP8XmK5HTLNHU5ao9TQS4U0x9+/Z1Yrr0tkvoW265xYnZ4/ftHgK4vwctG9IsE/XBOzEhgcMkJiRwmMSEBE7NamK97G7VqlU5H2trJL3rx0033VTYwKqAvHddtPSgfsOGvaxR78iiteGXX34ZOxbt2+exd74E/l+Xv/vuu5GtNbDW5T169IjspUuXOjHf9x6l2r3DB+/EhAQOk5iQwGESExI4NauJC9l5cvLkyZFt7+LY3knSe/b8qda9tl7V2lVj96PnXbXWtueN9Tn1Fjy2DrZ3yQSAc+fOOb79Bkt7900g/g0ZbY037jggeclmrvBOTEjgMIkJCZyaKaf15uAffPBBzsfaOzUCwPLly4syptDxTZ/oUtGewtFPBWl8O0TqqSG79NYvCrefWgLckvnixYtO7I477nB8e8dT38vBk57Iso/Nd1llErwTExI4TGJCAodJTEjg1Iwm1rpWv1nARr+cfNq0aY5vL8mrJZKmSHy7bNgx/Siir9+kNyjYx+o3iehppCtXrkS2XpL5wAMPOL59jfV47WkurYmTdty04QvVCCEAmMSEBE+7Kqf379/v+C+88EJk26VUEnpz+0mTJhU2sMCIKwGTSkXb9/2+fdMw2tdtjx496vhr166NbL3pv29Ter0jy4QJExzftxOJXUInTRvZ4+cUEyGkTZjEhAQOk5iQwGlXmviee+5x/C+++CKvflasWFGM4bQLfC/x9r0Yzadzk/rxLWucPn2647e0tMSOXetp+ymnhQsXOjG9M6aPUmnbfPUz78SEBA6TmJDAYRITEjjBaeJ9+/ZFtr3DBgCcOnUq537sOWQAuP/++yNb7+RAMiQtE8x1GWHS43uHDh2K7CeffNKJ+V4Ar/t99tlnHf/uu++ObH2NfS8LT9Lw+VIsbc07MSGBwyQmJHCCK6dfeumlyD5y5Eje/dTV1Tl+Q0ND3n21Z3wlsl6OaJeZhZScGzdujGwtkfQ5fcsjdcncv3//yNalrN483vdZ8i2DS7WRPO/EhAQOk5iQwGESExI4Va+J169f7/hNTU159fPwww87fq09XpiGuOV/SY8Q2nE9ZRPXDgC2bNni+Js2bYpsrXN9fY0bN86JjRkzxvFt3Zs0bVSsXSrTLKXM+0V2eR1FCKkamMSEBA6TmJDAqXpNXCzmzp3r+PpNAqRtfJrOp+HSxLR+tpdPptlNcurUqbHnBJLfPOE7TyngbpeEEABMYkKCp+rL6cbGRq9PykeaKaakY23Gjx/v9X3nzDWWllIskeSyS0JImzCJCQkcJjEhgSNpdISInASQ//N/pBQMMMbUJTfLDV7jqiX2OqdKYkJI9cFympDAYRITEjhMYkICh0lMSOAwiQkJHCYxIYHDJCYkcJjEhAQOk5iQwPkvxBBGQYvNKc4AAAAASUVORK5CYII=\\n\",\n      \"text/plain\": [\n       \"<Figure size 288x144 with 2 Axes>\"\n      ]\n     },\n     \"metadata\": {},\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"plot_distortion(img.detach(), eps, label)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"id\": \"varying-commission\",\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.8\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 5\n}"
  },
  {
    "path": "examples/rosen_minimize.ipynb",
    "content": "{\n \"cells\": [\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 2,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": [\n    \"%matplotlib inline\\n\",\n    \"import matplotlib.pyplot as plt\\n\",\n    \"from matplotlib.colors import LogNorm\\n\",\n    \"import torch\\n\",\n    \"\\n\",\n    \"from torchmin import minimize\\n\",\n    \"from torchmin.benchmarks import rosen\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# Rosenbrock func\\n\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 4,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"image/png\": \"iVBORw0KGgoAAAANSUhEUgAAAYkAAAFECAYAAADSq8LXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAByz0lEQVR4nO29e9A12V3X+/n13vt5nvd9Z96ZzEyuk0BCiClHziGWMcHD0RMgwUkEIhRqRovDJeUYi1haSmkojmJJnQN6DiiYSBzJ1BBLE5HDZaIDIaCcARVIiAEnhMAQAnlnJpkkc3tvz2Xv/p0/ulfv1avX6tu+9d57fd/q9+nutXqt1b2712/97qKqRERERERE+JBsegAREREREcNFJBIREREREUFEIhEREREREUQkEhERERERQUQiERERERERRCQSERERERFBrJ1IiMi9IvK4iDxknbtFRD4gIr+b/31W4No7ReTjIvKwiLxtfaOOiIiI2E9sgpO4D7jTOfc24BdU9WXAL+THJYjICHgH8HrgDuAuEbljtUONiIiI2G+snUio6oPAE87pNwI/mu//KPDnPZe+CnhYVT+hqqfAe/PrIiIiIiJWhPGmB5Djuar6GICqPiYiz/HUuR34lHV8CXh1m8YP5EjPJRcAqRaK5xymqq9+Zcd7CNLyXGAc+aGaHd919rn8evWV2e3Z/Yjdj9seqJTP2ceaWPsjYJQyGacAnM0SmCaQ5k1p1oGYwam12V3Yzv/Wvre8DssKIhD6rQL11HPO3S+es8zLSs9VgESRcdbahYNTrp4eoNN8PZeCpNaztPfzDsR6tsU5ewD2szURF9xnVvlttFLmrx9qL/CjqKeyVnYC/Wrgtw6MITiO+bln0ic+p6rP9o61JW6T5+kpp52vu8yT71dVV8rSGyKSAN8DXAQ+pKo/2nCJF0MhEm3g+2SD04GI3A3cDXAkF/iyoz8HIkiS2JXAHJvJM5H8MMn2S5OqU1+kcl1x7F5v6lr1VGTOyyXJ/Bxk50VQ019xbF2fMD9Osmt1NCcqmmR1AHRUrq9Jfm6Ulaf5fjqa10/HkOZviFr76QHMDmF2lD3+6YWU5FmnPOeWywB89qkbmD11SHIt63x0LCRnkJzlQ52CTEFm+XFKMflBPgmmzCceBUm1dFye6DzHNtoSDecNUx/R9B2bZ106Zv7sE4rfOU0yoqrmWY4hnWQbwOyckp6fcXDrMQB/8oV/yK/+wRcy/fwRAKOrCePrQpLPQckpjM6y5wnZs02mILPsppNZ9pyT/FhmWdn8WWf7pn723K3ymWaEJNVSfTPZZr+LzhcEaVo+VlOen0itc5C1ax9bfaFpeVL31nXqWNerpvO2AvXVOf65a//6D1gQp5zyavmqztf9vP74bU11RORe4GuAx1X1S6zzdwI/CIyAH1HV7yOTtNxOJrm51HlAOYZi3fQZEXk+QP73cU+dS8CLrOMXAo+GGlTVe1T1lar6ygM5XOpgvUjbzkQLwl0l2ueXDNH55h2DZjOlpILOhFmabaNxio40e7sMx2FvlI/VmZxL9UyXIuX6AU4J2rXX3J+/bXOspfF7uEAPQVH3GVh1C6IioCNlMp4xGc84nk1IpwmkAmn2rKH62wR/qx7oTWRXhVTX930tC5J039rhPhydbo2+9uXAf1PVvw389b63MhQicT/wzfn+NwM/7anzQeBlIvISETkA3pRft1rYK5dV9+OBqM5FAi3Qqb5nNV5MNAHCYFa8RZ180zRhptk2Hs0gmTdWrKjtybIycGtzzzXdhkt4aNGHr7+u7YXa9x17RHelZ1EQC0VGytHBGUcHZzxzekg6S8rclW8r3UD5d3TFUW3Q+B55xFbB+usKIjqgYKWSSOcNuE1EPmRtd7vtBnS6IX3tJeDJvM6s772sXdwkIu8BXkP2QC4B3w18H/BjIvJm4A+Bv5DXfQEZ6/QGVZ2KyFuB95OxVPeq6kfXPf69g1Ka3JLTTNwUERERgEgXzsDG51T1lT2uC+lrfxD45yLyp4EH+wwINkAkVPWuQFFFiKeqjwJvsI4fAB5Y0dD6waxeQgrwrm0tox0PzKqy01pLy39dxbNYimmdCWfTTKExGc+QkVqrZEVFSjp/tdU9AXGP6U9zSZM9+OKcc95tozTuBjRyCyGuwuGObC5BHCJbEZcVx1ockygXDjKlw7Wzg0zM5IiVvL/JELHM1f2AOIU6CBjOoCtuEpF7gPep6vs6dulCVfUa8OY+A7GxTYrr/UZKxj85sCdSL5zvqqgfKA9NNmIRmIoYI9dJTGfZ6un8ZApJWczkE73YYxb7HtzJtbjHXB7voXY2UckaK5fZ9+4iKJpqKPfqIgJtq33sexaFkjsTN904OQHgkcs3obNMHwFzkZN9L15xUsPvGiI6IdQS2rT+2r1EP07iaVWtiJhaoJO+tiuGopPYPPqsUlawsmmtU/DUqVxX047voy9PPFqcs/UP6Zj5JGVxFJLCbJYwmyWMRElGaaaXSLSivPYpnr3KZt++92bmbZRk/A7sFX6jfqRFe7Xjde/XqVsaQzLfSCAZp1w8OObiwTGnZyNkJmFdhKtLovweNP3OFTjvjPddbPF+dtWltUbf77TNdctSjguZFWPXLeckRORrO/a4Un1t5CQiOiE5myuvIyIifLBlqZ3QyEn4dLqq+q5V6msjkahDqnO/hh2ELULKVlru8p7wsS2iUCAVZrlOIhElGaeko6xCaaWMo4swf7V6bIuJbB2EIghaFn95bzBwHyE0/NQlPwiREsdRKq9r26OTKImfRsponHKQZMYo0+koN301g2iwVvLdawNnsVMYiKlsyR+rPRp1EiGd7ir1tXtHJDRN+/6A64VPB9FFse2Z830wk08xgRv5vyvb9kxMhXgjd4ZLZ1mHqQrJaK6TCCl2zfjEmShNWUjH4L+ReZ2SfqLN9TW6hFB5XRvu/flMad3nQZIT1NxH4jT3apxNR9m7YD9zV4fgOXaJgXTRG3SZZ31inC3RUWi6woGuiJPYBPaOSCwNK7REagtJKTymW41HsT5q38zotO/oh4VMJ1FRdloOdQAzFcajGacFJ5GNreAMPN030jTfRG8phV1Ft5dQtIRPD+F14HPbDpSXhu0jiMncEkwSzf0jMg/rdCbIbO5EB47+wR1KG86iSzmUOZEhEIChWznZkRm6oa9100oRiUREJyTTeXiOiHpUCFXE/iByEhFrQR9uxaz0XFFVqjCyl8blYpdrsM9Xyi2LGmDuCZyvdqezhMl4BqP5ctdwE+bYXU0Llg7CldNbYiRzuSIl6xnfhLwsPwn3vNFHBHUQDeIl1wTWWDgBMMp8JC6fZaFkdJY4XJu1mSY84qXScHzchRsTyUVfjmHoq/x1YYd0mVsgnI+w0T1MR80E4jVvxCvTtstK+6Z+Kmi+nU1HjEcpkmRbRiAobeoc14mDakU9eMoqk3q9Utlb3nTslDUqrZ17L+ktEgpz4WSUcuPkhKunB1w9PYCZ5AEOmUd/rVNcE9BbFPX9L0PXuE8rM3HdBQiIJJ03+pvArhSRkzDKK0eGqJqaH65V/UYUUS/z4zyq6yphrIJq4axINVBmCIdPJyFKFt1zZjiJEZPkBHGtm8yqOgt4W9ZR2DuuYttRoxj1gwmjbiarEufg0WG0ir9k9+PjIDzjc9v3OszhL7P9IyDzkbj58DqffPKWrDx3pLM9q93AfnVEo6tH9lrEY6VIr9qf+wgonlUDbNAqFdUlSF9OIoqbIrYfUSfRHo3K+IjdxYaNWpaJ+LmvEqnOBXqel0ZUi9VwEE2xoezyVcxKNWInd3VrdBKzaYIYr2vI/CUsPwmgFLvJ7Be34vbn4QoqsOpUYia1Xai6K37P+TbX2rAMwIp6FfNXIQutDozGKedGZ0WIEze3RtuwG0tD0V9DBy25gcZIsQPxc1gYO0Qkok5iVQixvC5yH4NlI/Qx1uaIsMqziV9L16HzpESlNs095JvOBNUsr8RobHQSc71ERVQTmDh9vhPudYVjWx3a6DSaaLWVz6Lav9NO0/04OgkTr0lGyuFkyvXZhOnZiOlZ5kjnht+o84nIEvo4v3FAbxESLa1E19DlPde0/fezW4g6iYgthMOdbLW4yeVImjiUNhyMhc5OfBG7CaGvn0TUSUTUoMHctSKa6ipa0qoiu6qoDjTq4zqcFa3oXHGdzhJmqTDOxU0mbLgmlmjMVezax5bYyJS7k6/iWQlb1wd9FNzb891ug6ipUWntXhvgQIC55VcubjocT3nm9CjLRgeZdZOdxzpkuRQiRpWAfS7nUXNtCBXl+HJEUbsDS5a6A4hEogtaeTU36BAWbX/JqIQOD8G2ZsIjzkgtpUIqnM5Gma8EzMOGO3L5inUT83Lfqr8wcvIQh5LfRH6tSxBrb6+GeNhxmrzX+K51ymwiUiIySS5uSjKCeuHglKdPjwrv9UwfMe+gEBcWg6DVJL8xp75FCETPaMiDQCQSW4oBhNJYFYogeMYctJid2hEtSXPFsqWbUIsQFPTAfmMcPYapm+YJiG44yhLnyEjRRNHcLLBQWjv6BrGGLFCeaF0xDlWiUeSbMA1Y1/gIRq0znNVepQ2fDsO5n5LawKfHsHNtJJCMsytuOjzm0csX5/kjlLny2r5pt3uLYLvcQatQGjUhwnfaa3xVRGaH5pmouI7oBJluegTbg9Z+GRG7BWHd+SRWiv3iJNrAmOD1cYZZRmjxupW/Gxk2pEMoyuuLu8DmFNSVi5t9E4QuFWZpwigXoSQjZTaay0hKq2jmXEUp7IajU2iMg+RyG+Yc1fNNHETX83XtqannETsB2TJtpIxy0dzNB9f5xNmthbipYEsCOomuntKNaGrLXXmHuJRlrdAXMYndpDltjN0UsWq08qHog2Ky0crsVon66iq2nbKSiMMSichMmE0TkrzCaDxjmkyKcNgmIqw3r4Tnr+v34NI+V4mtUo7r1NrPwm7TJ2pydCp15UGlvKOTyMRwyihX8h8mU06no3nUVycUR9c4Tf7wKjU/7BIQw3UIPdOXDhL7TSQ65WdoqDsEfUfTGNwgf+YaoDpLev5qppMw+opSWA57tZtCOk1I85l0NEoLCydgHuzPJgKu3N7HFbSd6DVAKFrCDr/RCr56PiJizguWpVemszk8yOR4V2cHmX9ESXFdM/m7zx78z8k8C7esabU9hAl/mdZT67gfI27aEew3kYjoDJk6yuuIIGKo8D3GpheMS0T83IeMUHY6KF7CIrNcJTR4+drGYH82V0DDot0tNNZNRj6dCuksKUJLTMYzrjuZ6ko6CEcEUzANts7CGr8pD4qcivrGQqjdTF0SMYl9fj5m+9i1Wio4JcrXqnOtbQ6sicJIOTc5A+Cpk3N5UL/5jZWsk1yuIYDaFKd1dcGrZ6hYSIWe6V46SnsQiUTETsI7+Wu+K2UC4oqboJglJVU095WAjEgUIcMBTaSIBGsuc01iRebtmwm4ra+DGwW2iVhUwno0KaR9/foIhk+HUYib8uMEklHKxcNjAC6fHZJOk0InUegmahTXpXvz/IYR64YrP22NmJluJ7GI89wC7Ytqc8wiLM6g4/BCYcPTEZUJykc0jK8EwLnJNPeVMI1XV+EVj2T7r6M2cRXVISsm93yrGE+h83WKat/1IUW2ubYgEkoyTrnl8BoAv3Pt2fN4TWbsahOD8j1V9qGXiKtrLokK1iHrH4J+pA366yQGad00GBW8iLxcRD5ibc+IyN9y6rxGRJ626vyDDQ13b5HMNj2CiIgtgEj3baAYDCehqh8HXgEgIiPgEeAnPVV/SVW/ZkWDGPSPVQsz9oCxkkHIKqZYo4WsnUqVHGMjj06CVJjm4qZRkkWDnZl0prl1k6uTcC2c7KGUxE+m3L0nMxxXNOVe64OPUyBwzhVH5eMvuqjhOjTPQleI3kbKaJxy0+Q6AMenE3QmJLbHtaOHKDUfui9PzCa7vUbOocSpbMkK3ocNjb0Nl78tGAyRcPBVwO+p6h9seiALOdctEz0IWEUkVfO9+HwiClNXp9w1gXVFIiiFr4TBaJQWfhJFsD8TpiOdE4qsg2wr3a1VLr57sStrgFC0RCUmk+9an3jMul6d+6nsm3vJw4MbnBkfCeNzYnJJhJ41ZSLim/xrCUJFVNVjUh0KERlSLorByGgWx1CJxJuA9wTK/pSI/AbwKPAdqvpRXyURuRu4G+CIC8vxhnaxCOfh81no3D+dJr8mFPGbTNtuuZbDhFdSaJpjK68EwCxNmIxnHFvpTF3rpmwn8NeMx9IxqJQOvZxDSZHdAnXRX10LpUodn17Frm/v50H9AEiUo8mUJ08vAFnCJpPXurg5n04C/3FxjSlftrXRpufhleS6WHKbwvZKJDwYHL0TkQPg64B/7yn+MPCFqvqlwD8HfirUjqreo6qvVNVXHsjhSsa6j0hi7Kb22J15ImKPMTgiAbwe+LCqfsYtUNVnVPVKvv8AMBGR29Y9wN4wSd8Lc0VF8m0tfbeAL99AMBuatbnHxlcinSWc5ZwEiRZb4SvgrrY9m1Je5Qf1Ba5FkVVWZ91VKXfEXrX6CXO9Vb/OsqloL8k2GSkXDk546vSIp0weCTvTn0d/UPFwp6WeIYQ1vH/Fe66ardzdb2GnkCvRdkRxPUQicRcBUZOIPE8ke5oi8iqy8X++aweapmi6Zq+fpo+hxXCaiEkljeWicJtQf/pSU1YoRVPJRU4CaWYKezCakYzSLO91PkGq2azJ2Ju8p7gpSpNvJTeDXc+tT1lXUNEbeNp326z05+ub8j2496b5vRtimYxTbjm6zjMnRzxzkuWRkFTKCmZ3K3XmnHMU1H3QJUx4q/es6d1eM6FYy/c/ACKRW4P+koi8U0Re07edQREJETkPvA74CevcW0TkLfnhNwIP5TqJHwLepLqTS5H1oePTi+KmMOoSF0U0YEhK5yVARTpvbSAi94rI4yLykHP+ThH5uIg8LCJvM8MArgBHwKW+9zIoxbWqXgNudc6909p/O/D2dY/LizRtl8e2i3VUH0V4nckqzImAW6xWf2bfqtQUAda1bhLPcZorrs+mI8bnjknG2eotCxuu1opcKqInwaPYtu/FM7iQx3VnE9g6biZU37nGcA7FfRT3midgypX448mMWw+u8ntn+WtvHOnS8m9RZ91UGVqF27B+zFAE2Kbn03ct1uW6toRi3VKArjDixNXgPrI58N1Fd5nLwDvIFtiXgA+KyP1k7gL/n4g8F/gB4K/06XBQRGJoUE2RRUL+akrnkMFNHtzm+wiJfZpgPkSfZVXN5CGqaEkQP991s9iJUSLkE90sj+E0yonEWaJeE9GidVeub7qziEDFuskakxtavLUVmCOass/7rJsqsZo8YqhC3GRyaOS6GMmJxMF4xrnRKSen2adofCRC1k0VaC7yqSn3YtGV+6pESLoYAdAFr18aVqRjUNUHReTFzulXAQ+r6ieyruW9wBtV9bfy8ieB3tY7kUgYtOUM1oW+5rU2keka5A+qk79bbIflsLu0VrvGBFaMCex0hKowznMmFCE6rInT9pMQh0BUOAGXs2hzvgsnUXfOnK8TLXmIXEmXYhT4wNHkjGemRwUhFRPczya4bjM+3YQpt4lLC4SJzwJEZGgS4E1wHv2IxG0i8iHr+B5VvafFdbcDn7KOLwGvFpFvAP4scDMLSGAikYjoBJlRjTgbAVgcjEEb4hSxc1Da6xgcnJGZ+XcN8OfrTFX1J7D0u30RicSqsOYQH5IuefJuSI1qe2PPr5n/NdwE5MH+ZqPMDBayVfRIUeNxbcRNjojGXn2XHqUlVsqLKw51S/G4to+let61tLLvxdVJzBMuUZi+Alw4OOWJkwvM8mCIGMsmK1S463G90hwVQ5DWDI0T6Yr+Oom+Af4uAS+yjl9I5my8FAxIvrIlWKNtd615YccxhMI1NMXysc0wQ3Ur7ZgtN4HVWRY2/Gg85Wg8zRTYiZbFMnWbuWWz+cQ9WJOxZ8JuQiiURsgU1/hGlPQRnnEX47F8RBhlpq/JOOWmw2OePjkinQrpVEr+EdQ8d/d5l8o9cZvqfudGM9e2uSRoeGeXjSETk34msDeJyD0i8rUde/sg8DIReUnujPwm4P5l3UrkJJaNZcd6WlRRHYBPPxEKD146NZqXFaHCXZ2EkvtKZCc095W44fAEyPInzKzQFJkyV0oTtOR/zbHXWslSHBeyepw6ZrjWvTaGT/dwDL6cEZV9m6A5nFCJ00i0UOI/++gKj16+iKa5TsLykSjG6jzbWhGWh4BXbm8Vc+sqOJAtNovtKW5q5CRE5D3Aa8j0F5eA71bVd4nIW4H3k80U94bCFfVBJBIRnRB1EhERLdBvjdiYdEhV7wqcfwB4oFevDYhEYotRm3goX14XmeXEWnbXtWnSoTqCyJIOwteGRz9hVqzpTJjOEpL8xGicMh1pZaUtLudgmsqPbX+JksmszWFg6SfEKbfb8yHECTjlvpDhdUmUKpZcVuTXG8fHHJ9NYGYNPrUvro6/fDMBbqHRRNWpb4tuavpbqzhpm7EiTmIT2D8i0Vah3Cdq7CqV1W3aXmL/heipRlfhEo1SqHBrspM0Cxue5jPpeJRyMtJKOlM3vwTWcbaTN6/O/C0NiusAsbCvtxFSXJfO2dfl5SX/CVfEZOrmeomjPKf1STrmdDoqIuYWPhKeZ2r+ekVOjr5iaWhDFFZJOPq03VZM5ba9LB8LqS6ythk7dCtrwLavopag7PaJmgRKsvPCOsecm2XB/mZpth1Opkgyj+GEHezPJhCOXL/coaOodo+pWeH72nfrN7Xv4zLq7sG612SccuHglAsHp3z2+EamZyNkJiUfCVcp3RadCcSevdNrw3oV1yvF/nESEQsh6iQ6QGgU70XsJtpY1HkQxU0RC8InTnJFKhZEFa3ToCnzlZgKtb4Rauk3KvaflI/NCrhIZwrpNOE0T2d6MJplXtcmnaloYeFk7sXVUZTET1oybvLDmqBLMZ18txbQPYRQ3GaNjsL1sLbTlSbjlGcdZelKnzg5Nw8PTvbMxPK4Lp4l5WM7XEq9zkKd37n+3oI6h9B1Q13JbxRS/U63GFHctCy/h4F8LEGfB/dcSG7rTP5ue6kV8ynoQ+FskvtLnJ6NOT0bczCeMhqnxeTpCxVuH1dENz5dgCvqwalLub2mUOHeNj19u6KokrgrD4VeFjtl5q/PPrzCsw+vcDkPD258SirPziEWIVR+97r67m/vU3x3FHOtFDv2jdYgipsiFkexmu9pY9caAUJT+BjUXGPnlcjOKzrLfCUAEpTRKC28jou8EsYCyCzEnBV6CXWinHyuXUhx7emzuNwlLqYNnz4Dc2/Z1cay6ebJNQCOz8aZl3XBSUhlgvbt18ZoWsNcGK2cahB4f1ogipsith8yU3TR3Nz7gqiT2Fv0dKYbJCKR2BRUMzl0vnoWQFPp71ndEGvJoNHbONCm641tm8BW4jg5q1wjbjKRTlOEg8mU4+QgqzDKdS0hHYRHjCRY/RvOwem/ooto0jUEdBMhb2tXrOWawBZllp+EjFKOJlOuz7J7Pz0bZ+HBXc7A1UlYx76wKPObaEeVOomSFuEaUqM3ydtI3ZdlR7FDgvxIJNoi5DfRlP/BV7euftv2nDDg3nZMuHAa6jLXJVQ+X1d8XcdF2KIQpRSkTmaZrwRQBPubi5t0nvca5gTDmXRL/hI2VXAH7Zzvpbh2RVBQJRTW+KrKaiw/EM0IIZm46cbDYx4/uRGA6TTJTV/nxLiii2iDQL3WugXbRWAJqXYb2+hDKELXDCx8h9Kbk2j0uN4EIpGIKKEp50QUN1lwxUlNxxH7gaiT2HF04QxWPY4ljyEU1A/K5zNvabyhOczQfAH+7ONSkLo8sqnxKj6djTgYzYp0ptPEUlaT9SszalfuSFmJbkROxTjdR9dA/BrFSXZTjuirYvWU/9UEK8xJNrrxZMYth9d54uQcQG7+Oh+8pHNuoriXpmP3VtLq+boov0tHFy5haN7ay+p6h9ZRkUisC2sI89E1p0Qj12CLn+q+Ny3rCFxb/iz8tRSRTs+mI248OCkioc7yTHW2dZNPJ1EeXPmc/agUh1C4RCOko7Dbs4mS25/nuoo1k70/0kK0NhqlPO/oGT51+ebs0cwyH4lg/ghPdxXYdRvmxa5EoTEGVGUsPSbmgYmLloJNLzKXiB1SrwwUmrJITBhRrXFwapL79iwr9V+eWDQxikgqk5l9jWvrLylZELvcFFZEGY9mjEezTF7v5pew/Asqfg2uLsB3ex7OoHTcRCACbQb9OOx2hTxGE9aW3ePhQWb+ev1szPXC/FXKTnT2M3UJb77vvhed/BoWmZMD71zte9qq3cW+k6Eh5JNTtxH9JCJ2AeIRQ0X4oYnOfUUi9gtRJ7Gl6CPyqW0vhWQ7ZszasOJtUSPXVmsla8Q9cwsdQVRJc53EbJagKhxMsnSm15MUTUZgJSFSkfl35pP3WzqJgrOw1UmWKM2ro/DdnsN9qLXvFUW5HIRVPhc3ZRyE0b9cODjlqbPznJxOsvKZifoqxVh9SYeC1k5LkNS0DRM+CKRL5jZWIeoSljvPbBj7RSQcaJoiq5zkl6l8Dpi8VvNFNCjeQ+Waz6p2LKdSP+X5o85voiJTz3USRr49m444mY05GGVEQkZKOpoTMdcvQpxJ2fXILmiAo8i2TWCbRDFeAuEQpeCxfc5cZ71WJl4TwMXDYz59fLHwGZFZJm6qiJhqxtuolFas39HaL9VpeiDl8lpR0jLn7RUrm3XZRCbUz+7QiP0mEm2guZxUZAAcwyryVYSUuHaZqaKQjubnS9ZNznWFHsK0oxS+AOk0y3l94eAUyJIQzSw/CXWtnRwiUSEi7j14jlt9tA116pznihhN1vjV8o0wSvrnHF3h0tWbCp+RLGZT+VmV8kU4x43+E13PL4oBOcbpkHQakUisBiLySeAyMAOmqvpKp1yAHwTeAFwDvkVVP7zuce4zkllOKCKakbCa3M8Rg0fkJFaLr1DVzwXKXg+8LN9eDfxw/nd4WLb+oydKAQFd2Xyq0OQYp3Wshl2P0mpXnWPjKwGgacLp2Zib83DZRbA/E5sikdzaybAK4Mr9S0Py3Fope53LYbhwJW92n3h0EJ7+y391Lm7KLZtMutIXHj3Jx554buYfATBroYNoI3pqu6IPRIAdVMC+bTeJnScR2gkMQIbSCW8E3q0ZfgW4WUSev7TWu8grN/VRhcImN8mp6VZe177PxNVbvxAz5fVyc0+siLAAB5Np2WRUtBDhFD4HtqmgLf/Hrx8whKM0x0pgc4ZbIQiB/tzx2NFsjcKaJPOROJpMOZpMmSSzIvKrMX8ldQbgeZ5uGPayYYDn+S8Ar47DxbLCd/dBl77XpIOwoczfjWgCu3wo8HMiosC/VNV7nPLbgU9Zx5fyc4+5DYnI3cDdAEdcWM1odwyuLLzida2EnfWslbtt6WTazZSzOVeTZhZOs9y57nA8JRmnzEwsJxHEk/Pa1lFUHOwsvYNPR2FPKUHGwuUoPITH7Lt5IyqKa0snMRqn3Hh4DMCjxzdzkgf1K56N+9ytv+IdaD0qnImnPGLF6MdIRBPYFvhyVX1URJ4DfEBEfltVH7TKW6hX85MZgbkH4Kbk1vhZLAkxfWl7yEgLYhARsa0YFJFQ1Ufzv4+LyE8CrwJsInEJeJF1/ELg0fWNsAXWFPvJm5q0Llx4Q9TYUIiOUJIhUc3SmNqFtk7CvdZwE1BEhD2eZr4Ch5NpYeEEc5GNOCtz1y/B5hxUyhxCaQUu5Rvwrhh84iVbJ0GVU6iIuErjz/aNZdNtR1mSoc8c38jMRH6FIoy6vfL3ipKsvz6/hlKcLPfWmpZIdRIZj1hnbfqLIelJOmKXFNeD0UmIyAURudHsA18NPORUux/43yXDl5GxZxVRU0QO7wfuEQW1+BaLCT8hKDMv913eTNDAIo7TLDODPZ2NmCSzQnktI83zS1BKb+qV4TpbRWfRc/PpHHz9lLbEql/oV7RQWr/o/JO86PyTPFnktJYisJ+r42mlX7Lqtf39Wv3uWzwxDwYChfK6yzZQDImTeC7wk5mVK2Pg36rqz4rIWwBU9Z3AA2Tmrw+TmcB+64bGurfY6rAcDkdR4TCkYcJtuN41eY3ipv3FLnESgyESqvoJ4Es9599p7Svw7Wsb1EDMWINoSjzUhB4KUfsyWyRSEi/Z3EN+vhS+OgVNE07OstdvfD5lMp5xXJjAZhxEvmCYh9wOiHsKUZNzP96wHL6J3nNN+cZrRE2O6AujcM+9rFMSjiZnPGfyDABXTw8yTmKWX24sv2yuwMMl1CmiO2HR64fu9zEU89kBTxtdMRgisTWo0zl09Yg2KUwBknz+MgHhko5t5SjyF/ROg2qNrW455BMvWfveMB2pGWMmbpqadKaapTM1oStmySizaDIpPz3y/0oocYsQuDqJCofg++lq2i+dw19WKrd0Esk45cLBKZ87uwGA49PJ3PQV5t7WIR2Eu+8cB0NyLDhXdg4RXhrD/L0uUpeayburOKuu/oBFY1vLbXuwQ7fSANtlf5M23vYYllWn9kPCWpXashXPNSH/C/uypGY168rTPXoJs1LWXHk9myaczMYcjTPl9Wic6SbU5zfh6gkMPLoD33GtjXrDtY16iVJocC0U1qNxyq3nrvGp68/iU9efxdl0lImgbG7Lp7+xn6EHFZ2CR5eRna//nYt3InS9fc0y39lNwr2XZYcp97wvrbYVINf1/rqIfE3fNvaHSEQsBQutMHcchnswMF7WER5smlCsGD2d6RohIveKyOMi8pBz/k4R+biIPCwib7OK/h7wY4vcSxQ3GawieF6oH2jf17rSqdpy++A4XAH+/K8tXpK8umvWaQL8mQQ7JjTFydmYi0fHjEYZBTobacFBAGgimTjOEChXJwHl9KXO8Cqhwt2K5nRIz+B+xG7/peB+WnATR5MzAF547kl+6+nnATA7G809z6Ewf62E5bC7s1f5bbmFUnl98VLQZ9LfZVPa1X2v9wFvB94970pGwDuA15G5CXxQRO4HXgD8FnC0SIeRSCwLKyYyXr8It9zuv+14jKy4KYYTVGThwU9PnXJ7AoRMXj2TLHUncDIdM07SLDwHcJJM0FEyvx/jM2Hfnq2TMETAHZBFREr91xEHz/UVnYNbXxxCYUV9PZqc8eKjz/Orj784u/U8O1/lmYT0DJ6H3EYcVUEXhW6XMOEtyhfGlnEdSnvOwMFtIvIh6/geN+qEqj4oIi92rnsV8HBu/IOIvJcshNENwAXgDuC6iDygPULlRiKxTixqLbWANVNBZAITZbU+uOEsgJKfRBE8UKUkS/cSB3WOjZ8AWc7rVIXDcUYkro6VqVmRYwiEZhwFZMpe+x5kzk2YbsUeSE5Aaj9ch1OoxGpy+zOEgTn3oJZ1llFYAzw9O8e1IslQIKe188zCz859yM5tuLqKEEpEaYFJeFHx41CskZaNfp/559zI1y3hC1f0alV9K4CIfEvedq9fK+okIjoh6iTa49Zz1zY9hIhNoIc+YsEAfz6SNF+rqd6nqv+h7+1ETqILFuEENIU1Jy4KhdqoKy+4ASPKsfQApTVfzQLQDRVeWDaZtoxOwkpnejYbFYre0XjGbDTKPK8BZrlewphVJtm5ip+Evfqn42LOJ8pydRJ2g1YAQnJOp+B8LHHTreeu8fvXbuM09wlhZkV+Zb7yd0OD13IDTlnp2VrnvLfZsHDfSPC/RSyLhsqJ9Jsm+gb4W2m4oshJhDBUOWidKeKCQ24zgYSiirpiEQKTn31eZpLHcRpxPJ1wOJpyOJpmCmwr1HYh2rF0A675qs+ctWQ14k78nnOh6319zMNv6DyceR5SREYph5MpX3D+Cb7g/BM8fnwD02nCdJoU5q9zZX712fh8TuqefdNvthBC1w/BjDyEAYxtzZzEB4GXichLROQAeBNZCKOlIHISi8LEqw/lyl6VdVIfRbml03C5iCauo+g2sZTSwQnEc2xPcians5XO9Hg65lm5eOZgMuVkPGGW5IPNFddF0LxUKit9+6/LRRSqiYA+JqRzcBXXIUU2dg4JIMnzR7z06HEAfu2zX1hYcmWWXfOLK8TVHbQLW1fREm7d0nGfRfwqJuA2bW4gN0RvrIiTEJH3AK8hU3JfAr5bVd8lIm8F3k/2hd+rqh/tNQIPIpGI6IStjt20ZlxOj7gxOd70MCI2gJ7WTTeJyD3A+1T1fd52Ve8KnH+ALLbd0hGJxK7BTTe6CCdTE5qjYgJrcQpmJa/WqteuX4oIC3k608zCCeBwPOOKiQgL86iqJpaTUCQiMrcqWHoUqtxEK+smS+zk85MoHRvvanND1njHk1mRZOhyepTFa5qZdKVStm6ynk/xHCmX+XQQFRTX91zlu9cNVZS0DXDFmu0Rkw5FLBdNvhOtUBFzWJprV2YTynfdIF4qlNj2RJiHDAdgJkzzsOGQ+RaMrEx1hZLYKIaTLJeFLT5SKJvAaploNELKREQ9RKPsPKdlvwmjQyHL2f2cc1f4navPBeDkdJwRB5jnjvDoasDR4dio1Q146lYm/cD1HTGoXNgDhvYTLzdyEpvA/hGJnhZKqinS1TppgFFkC9+GnjoSTaQyoRXTRo0cveRVXOS8zo7TlFISoouHx0zGM86SrEI6SiCRwjjMTNqldKZQVUoH6J13iDUKbVdHUSjSCz8JSg50hwdTvuj85/jIUy8EYDodFYEbk5yDqnijO8/Lu08NEemIwU72PayVepr/D9EyapCcRJQuR3SCDO/DGiyeOjm36SFEbArSYxso9p5IaJqi22Q14cKS7duoOL155dgLiCSsuuKIPEImsSVTTytmkcyEdJpw/WzC9bMJY8lMSJOxkoxzWX9isSO5Oaz6Vvi2PsHaGs0PnbpePYRT12TOY6RIkoUFN57Wf+Tcp4GMUJh0pTKz4jSZ3009z8bhvLJz2uht3fQ7Zceei11uxfc5BN6zbcK6vnOl+X1bsgnsSrF/4qZlYl1BAdeBVL3xm0qJghLmoTEgKApxxU+uc11BQKwgdzoTzqajvCvh3OSMK+MsM88sGZWd10xqUMuM1BYpGf2EPTxXie2edMVNPp2EHYbDjtVEojBKC3HTzYfXuXR6C1dPD7L7mSYlnYSd07piHtxEADzlpYROPuwS97dCMZku8zntkOJ67zmJiG7Ya3GTq69vCA1+crq/a7DB6jzWgT6ipgGvNff3LQ6hyTmuLfpwGTq3VdJUumWn8/W3AKdTBPgT5xz5eU/9EgfhKF9t8YmxPrLDdJDOOYnrZxOOxmdF6HAZpYXyGpinMrUV15ai2vfR+biKirLa3vdxFqbMcDLkXMVISXJLrMPJlNvPP83vXH0u0/x+sMJwuCE3SqI5e7B1im3Kim/3/EJe1m2TUdVd72alW8YY+mCDYuSefhKDROQkNg03vWOofAkoZ6Yz5wKTVmNb9hgDdfCIeQqdhL0vpUx1x9MJR6MpB5NsE+ODkG9zcU82WHV1FFT1DD79RF25acMWNRkdxDwsR7YZfcTRZMrRZMofu+ERPn39RiC3bMrjNWVb+TlURHahB+6K6xpQ+7uWzi9pQq57T1Nd6nu8FejHSUSdRISDLiv9FnUrOSVWAFsnUdJBOJOYq4NwkxCpzUnkhMKErjiejklEOZeLb66NU1KjvIbsOSRzHYFotj/nwrAtYLM6vptp0klYJq4limf6NuPJzV+NA92zx5d55uSImQnFYeWPMASyxAm4E7jnuBjamufZVkSkK6exB+jJSUSdRMQWwk1As886iQY859yV0rHOVkuwIwaMqJOICK7sV5hutI+HtRv6u6mut54JzxESK3l0FOXrKa+Ii+O5jkFnUkyqp2cjTtMR5yZZ0p7x+IhpMiI1eqJE8xhS5vqcg8rHbiyb7FvpopNww3JkznPqJBnKdBEASR71FTJC8ZGrX8C100mWYIg0s2YqUrdK4FnUPcAaDqK43l+hLefRl0NZqYI6GO144AsVafe9bQsikWiCWTkv23O6r1LZiGhCGerM9xNq2rq+MfKrO9MW5wMX2iIVa+JTLBGUp7ywwc8n0ulsxPXphBsnJ0AeFXY0mcdyGgk6mz8+E5nW9pC2hyPOfRRTjKvctq+3V3dmP7GOE4rxZPqIM15+w2cA+OgzL+DEyUQXejaNimzffFhDFFoTmyadbtM8vIhOeFWT/JC43B0iElHctG6kupyXedkfmttcoP1SBNiS3qFmPE6RHZ5jvknBTcymCcdnE8ZJyjhJORzPMh8Eo7w2q3o7n4OlUC7pEHyrOg97X3GoS+zNVZRrKd/FaJxyw+EJf+L87/Mnzv8+nzs+z2yWzLmjXGFtNrCIgedZuvAZHNRiRXGbgu33wbK+g6FihxTXgyESIvIiEfnPIvIxEfmoiPxNT53XiMjTIvKRfPsHmxjrPiOmL+2AdIeWkxGd0NPj+mlVvXtIwf1gWOKmKfB3VPXDInIj8Osi8gFV/S2n3i+p6tdsYHzNWLcHdl1/6spYWo6tSVxVc03Jj8LmMkz3RvwUsugpkhAlHJ+Ni9Dh5yenXB4fMs2TEKUjQUZaBM2TQhyUNah5UqKQTsILi7uYm75qqayUZMgK6DcZz7jt6BoPnzwPgCsnh4Vlk+Ei7HSlFXNWtcbqPJveAf261O8SJnzd+oCh6x9C2KH1wWCIhKo+BjyW718WkY8BtwMukVigEyfP9Non9fXnuQ6hMJdtqWgv4g1BNmG6FVrIwiv5JQqxkxTKa8j+np6NuZZHhb0wOeVgMuV0nMv5p0bMJPN7sRTrYkROZmI2/4XGaBEHmIur1DaBtRTXJJnfRpITifMHp7zkwuf4zSu3A3B8NiHN4zXZ92uek++4Fp7yksiuowhqUN7Qi+S37tWfSxCX33+N5HArMYwZy4GIvBj448Cveor/lIj8hoj8jIj8sZo27haRD4nIh061ZXawIX08LVEK/OaWtQnyZ8OREVcCzQE6CpfZ/ZRWzA7HENwsBfZslnDt7IBrZwccjGa5XiLbKjoJo1TON020qleokQWrUxeZt2GU1CrMnflGc12E0Ue85uLH+PT1i3z6+kVOp6O5l7VRyJvNd9/Os6kER/T8JqHzlbImub9THBIn1r1ng8YmxtxHHzFgzmMwnISBiNwA/L/A31LVZ5ziDwNfqKpXROQNwE8BL/O1o6r3APcA3JTcuoVv9zAhszmhiGiACa0SsX8Y8KTfFYPiJERkQkYg/o2q/oRbrqrPqOqVfP8BYCIit611kH2tMla9otmgQrnOO9grU7dWzMWWUoSuMBZOJ9MxJ9NsHXPh4ITxeMZ4PCMZp3n4cObez6IlTiBrlLkuweYWHO7BrmNzERXrppxzMaKmw8mUw8mU246u8dnpRZ46PsdTx+eYno3mjnTGssle6buvQt2r4YjoNoJVv1t9v43ds44apHXTYDgJERHgXcDHVPUHAnWeB3xGVVVEXkU2TXx+ZYPapVDgPijFisf1mQj5UIS4iLoAf25YDrXOAXPFrjnOTUdPzrLX89p0wg2TU56cZKHDT8cp6XTuzCZppqy2/Say82YQUp6HfSaxMI/qavQR1rGtk5Bc1HSUO9C99IbP8quXv4hruW+E0UfYYUcqoTW0vO+G6VgkFIcv6F+BnZtXh4kYlmM1+HLgm4CvtExc3yAibxGRt+R1vhF4SER+A/gh4E2qAxGUDiFxUYsInk1y5bBnb0YFZFo+rm3b5SICE2PBVdj5JdKkiAp7NddLnJucATAap8gonTvXWSv+8nHevPFpMB+uKwc2vg82kbHyVxSJhSznufF4VsRqev3F3+SRazdzmhO1wjfCIoKFf4RJ3erjsDzH3mdqP3tz7KCJsFTbDVwwhM+ry7c1hPFC1EmsAqr6yzQ8KlV9O/D29Yxoh+BwRE3hPTp5YnsmtVLbjZNg+ZSkglpnDKEwODc54/rJZF5/pPOEPmQTuqT2sdNngIsIHbvPoSBMOQyhMDCEAsgIhaWTcJXCC4uPGglBU3lLQjEEDGER1gExLEdEe6wqrMe6MJfYNNexUJn8rXpG9GSH6cgsf/ITKZDrJSDLLwGZvwTAlfEh09GYWT5h60hz7sSSnYlYYqe867pZ0zJ5LVk3QcGZ2JzE4cGU552/DMCnpzfx+ePz5fwRM+b7dtwrRy9hVCeN4qWm+XvA83sr7JJ+YeCcQVdEIjFQiOo88RC01o2Y1aEbMjzEHZQCAFZW/VWdTNCyydVB2G07MnbX/NNWXkPGTaSzeehw4zNx8SCL5fTM5IiT8Zh0mo9tJEV6VXMzkmjh8VzEdgrNQ7biGgrRlBbiLM1zWmQDHOV5rO+44TEAfvmZP8K104OCqOlMSFIpiZgqyn3f87CeWW89QkDkV0sfvUS+46RtEfitNJVdNnaISAxJJxGxBZBZc519wV03/Xrp2BC1AnGu3Fv0DMsxSEQiMSTUZfBadmavBUS8QS9fe4XcorxiImtZORkLJ50Jp9NRprxOZhwkM84dnOWmsEoy1rmD2wgYMXeeK4LwgTZsRUpSEygwv86E4CA3ezWmrzcfXgcyQvGH157FcZ7PuvC0NpyRwzVUOJom0VJAlLcQlineb3pfd0mM1AVRcb2fUE0RN6xGXViLJWeeA0uc1PRW2crlPnCuL4X+tkT/pm5pKnAIhblWA8cuwZBU0HQuLppNE66fTpiey579jZMTrhwcFjqAdCqQjKxMcfnzKcJy5DfjKttt/bqlYC4RDShETSZW09FkyosuPMkvXnspwFwfYVlnzfNH+EVdwTAd7jN06tsWU6V2aq7tjBbXdxIpLauup0zXHdajLQYw6YvIHwX+JnAb8Auq+sN92tlPIuFRJmuaIskGGKu+SYpSwjkl6vqqCwjoKSvpLLDk/r56pi7UTzSeCVCtfBLkBMI4pKV56HATy+ni5CSzchplx7OxMku1SoTMl5pSmk3de8pOWsdOUiFJMi5iMs5kbTceHvPlF3+XX3r65QC5PmLuQDfnIMxDqxKF4POpIygBNMd+6jbxNqLPvLxBPYW6llGr5m5WKD4SkXuBrwEeV9Uvsc7fCfwg2azwI6r6far6MeAtkq1s/1XfPqO4KaITYqjw9ojPao+xOnHTfcCdpa5ERsA7gNcDdwB3icgdednXAb8M/ELfW4lEYpvRtDprdJzzlC8hGYzfAcz6HsyxVvd9YTrSWcLpdMTl00Munx6SSMoNByccHkw5PJgyGs8K5zop9BOKnZSoJFpyV3qFDiLf8jbE2kbjlPMHp5w/OOV55y/ztRce4ZFrN/HItZs4Ps0srWRmbbZlk6WbqNyv9XwW0T9URFaB37FRTLRoeUTBRK5Cca2qDwJPOKdfBTysqp9Q1VPgvcAb8/r3q+r/AvyVvvezn+KmNkhTWET81EbHsIywH13FVR55fAhBp7piUvPJbcrX211628DzN5fj2/GPZtOk8Jc4nk24ODnhmckRACenmc/ELDXiIckebaEUoRwRuqJUoaSDKHwi8mMjarpwkPlp3HHDY7zjyS/lqZNzAEynIzRNykmG7JSl7j3WzLNz3YOH0AKFt3WgjdZK7a5z/bKIQ5t2Fu1rCI53/T7r20TkQ9bxPXmg0ibcDnzKOr4EvFpEXgN8A3AIPNBrROwJkVBAU0W21aGtBUJe1GZF2+jfIDVEoag7z+Hgbcf6tm1nusJxzpHL2zoEO7ZTpqOQYiWsuc+E8Wa+cnbA885d5obcb+L62YRZmhSEIFXJ00fkOgJyYmxPtPk9F3/Fis2UEwiTL2I8mXH+4JRnn7sKwHfe+jt80x/8b1w9PQBgNh1lXt+5ebDkITkqegh73yEa4jy70HOtoG2YlSa9Rs28uut+D7oKHUW/qeaMLNL1+zpmp/Mu5VT1F4Ff7DUSC3tBJCKWB0lrCEVEGcogrFwi1oz+iuu+Af4uAS+yjl8IPNprBB5EIrEIFhEXpbr8UB19LJ4CqHAVXSa8wOq15IHtHBcWTlCE6ChiMM3megmAK6eHTI+uFh7YV04POZuOSE1mOxWUZG7xKpZewHuzmU6isG629BAAB7mo6aU3fBaA9155Fp++diMnhW+EFF7W2QnmoieHc6iN0Frz7GrR1OYiWIXkZpGV+7ZwNf0+7ZtE5B66cxIfBF4mIi8BHgHeBPzlXiPwIBKJtljFpL5OdAzyV7oOas1jK34S+Tl7343l5IaicMVNklrzQZ7VzY7ldOXskFsOrwFww8EJJ9Mxs1keFkMl80vM7TIEQe0BuDoJycxcKcRNaRHpFeDcwRnPPneVN9/yXwH4/s98FZdPjgo/DU3nDnTAPH+EFSq8LixHGwc7b7iO0vNsP3luVWC/Ntgdh71GTkJE3gO8hkx/cQn4blV9l4i8FXg/2TLxXlX96LIGFYnEstGkSDbL5VXluu7pd1FwDk3fm+QhsNOynqPiJ1HaoUwU8mP1lNnHhpPQmRZ6CYDTs8zS6eaDzOv55oNjrk8nTHMikebmIuk0zZsTxAmyNx80hU6iCOA3UsaTGYcHWVz0Gw+PeekNn+XR6Q0APHLtZo7PxgXRYpY57onFPYTu3XQrdhlOXZy67rk2ea1DOo62WDXhaOMEt+3EqzsaOQlVvStw/gEWUE7XIRKJiG7Y9URMy0TUSewtdinpUCORyL31fhF4SlW/zjp/HvjvwH8C/jHw94GvAJ4PPAb8O+Afqer15Q87og8qeoYmHYYzyXk9lQP9YImzKqImu/38b8naKV+VG6sTSSXTS+Qr9+l0xNXTA65MDwG4+eAaNx4ccJqnOp2lSd52HrZjli9cQzKxXNyU5JzEaDIr9BAAzzl3hW991n/lnZ//MwA8cXyek7NxMR6d5WE4ivFLSbxUsWbyDKNRVxFAUziPCpad0yLCjx1aHDQSCVVNReRbgN8UkW9T1Xvzon+cX/8dwJ8m+yL/OvC7wB8F7gFuBQZHGTeGNa7CRbUSLnyx9sgD37Vv0/aTqOSP0HJ56dgtVwq9BGQmp8dnE545zYjExckxtxxe49gQCZWypEKSLFhgESbDlntlBMKkJAWYjLMggiaI3x+94dN8enYDn7x6KwBXTVhwE58pd54rpSt1w4Pbz6SGaGTipW4z97In+rWavO6qSGm9iuuVopW4SVU/ISLfAfwzEfkF4IvJCMJrVPUq8LP5ZvAJEfk/ge8hEolhYUFCtc0msMk4LYXzdo8n41kpE97t55/mkWs3FcefPz7PrUfXimOdSTlTXUo5hkEUN+0n1m8Cu1K01kmo6jtF5OuBfw28GPiBPOVoCBeBJxcbXsRCCE1SIUJhn081S+bT1L65zqcYduvlcJ3rfFFiJbV0/JrtzwP+5eHDc07iyuEhtx5c5ZajbOV/lo5QFYrEojkHlKaSOchZX3AyTkkSJRnNuYijg7MiLent55/m7936EG/7zKuAjFCcnLqiJms1byybnPtzHeQqznOhBXXBebhWATWwrX265q7e0YX92rFDi4Ouiuu3AL+Xb38/VElEvoBMDPV/9R/amuBaA7W1DlqDSWyRnQ7mYbCLwuX0HcpkNy8nL7dOGqJQB89kEwzT4YpgKE+smkqW7MhYO6VJHqYje32fOjni4uQ6zzrIVvmns9FctAScnMFslsw97nPiKXkHiSjjUcrBJLdmOjjhOeeu8GUXfw+An7p6K5+8eitXTjKiNJ2O5nkjoBCFuWG8vffWRvdQRzSK8nKF5kixS579C6KVU/eCoK+ByrQ1ed2gqW9PTmJ7xU0Wvg24TubR90XAx9wKIvJcMnvdDwD/dNEBLh2ars78tC7eU1ei0iAWKghIk/Nc8WEsgagowVDhofqhfbHO+ZzrMHoIchGXkpmaQqHEPjnNYjldnRzy1Ol5nn/uaQBuObzGVJOCCIgo01nCdDYq2hfJiANAkqQcjOaK6lvPXePlN3yGu258BIC/89iX8/nj80VioUIfYeeMSJnPDI5Oxb5/ce699jmtE4v4WSzQFlA/6a8yDtMqc1HskLip9ScvIn8SeBvwjWQE4L48RK1d53nAfwYeAr5JdVe1UvuLGP46jMqziW///qIS1rjFNlC04iRE5Ah4N3Cfqv6MiPx34KPA3wW+N6/zfDIC8VHgLlWdrmbIa0bfpEB94UmIVIu+43P1FUYEY4mXGgP+5fBZ7JReeZ+oxZHZuzoK2wPbDduheZgO42F9/Wyci5wyPcLNk2ukKiTMxUmn0zGzUeZBPcu5DFM+HqUcjc8KncZLLnyeu276IP/syVcA8Kmrz+Lq6YGVCS8pdBGl+4ci/If57tV3303Phvm9N84hNdc19tkF5j1ru0jYhBf0QNakQ89Z3RVtxU3fCxwBfxtAVT8tIt8O/KiIvI8svvkvkgWV+ltkLuPm2s+q6myJY95dLGJ51CSecjykF23feF2Dv13jKwHkHs/W3OT5lkse22oRDjP2dD4pkyrM5mE6Ts/GXD095ImT8wCcG51y2+GVQpw0TlKujSac5eKmNKeOo7zBo/GUi5MTXnDuKQBed/F/8GvHX8BDl28H4MmTc1w/mWTRXiHTP+R5I4pjyw/Dp7Sui+xaVmJr7aLSy8ktODk2coeLtD+QiXvt2CedhIj8GeBvAK9V1cvmvKq+V0S+gSxT0juBl+XbHzpNvAT4ZJvB+FLwOeWSl78BuAZ8i6p+uE3bG0Pblf4SfChax2PqWR866iSKfih8JWC+7/pNuKvqIsy5ylwvAVlq07TsXHd8Nubp0yy/xNHoBl5w7imefXgFgMNkyuXpIcezTIeR5su8gyRbu9wwOea5h5d51YVMUX2sB/yXZ17G49ezMBxXTg5zZbUJE+I6zwEqYULg7Fcsm6CWMLhYRDLR3QdjCZP8OnJIDA07pJNo40z3YKieqv5F6/BHFhmIlYLvdWShbz8oIver6m9Z1V7PnBi9Gvjh/G/EmiBpP0IREbFP2Edx0zpQpOADEBGTgs8mEm8E3p0rxH9FRG4Wkeer6mPrH66DbYhp1GgxZb3crqxcAvXstm3rnkrfzvV2k1peSBYBBI1HdB4Rdh7wj0zUMzIe2EkucsqSAD2RnONgNOX5R88A8OyDy1ycHHMyy173M01IRDk/OgPgOZNnePnRYxynGafxS5dfyiPXbubp4yzz3MnpmJnJPkfet+NRLSlzeb1HvFQbdqPueVliu+D1bhsBUZYX27CCX9cYl2ntNPCpoAuGRCS8Kfha1LmdLFbUWqD5iyTLMqNdtmLcvOdt9Q8d6gdjN5VmfOucS2g6TYyO4toKHW50FHPnuoTpKOHYmMQmKU+MZoVi+rlHl7k4us4k94MYoUxkxk2jzK/ixtF1Hjl7Fh+/9jwA/vDaLTx5co7j3A9jNksyUVNJB1E2gbXHW1HkW6g1f3Wfm+/ZuO21nT+7zn/LtmJbwUSvqzRhXQTCfukk1givb3CPOllFkbvJQ4IccX6xkW0jAlxD2yB9gNfrWt2wE63H49l35fTWsctZGL0EAEnGTRRjy/UTZ6OM0l0/GzNKjua3gXAyGXPzJCMK50cnTIAnZhcA+P2TZ/OZk4t89iTTQXz++DxXTg45yYnE9CzjIgxRKohCwTlU9RHiIxq+ezf7bZ9lU72WVkW1xGUbuIuho5/iaDt1EmtEmxR8rdP05QnE7wG4mNwa3/olIeokOsDHYUXsBaJOYjVok4LvfuCtub7i1WSUd/P6iBCa9BQrDu3RKxJsQPxV0UPU3FsRi8l3Hvy+Ax4LoLk1E6VYTmhucpobVmsipEnCbJpdcCKT0tBSFY5nE56ZZtzFYTIlkZRpbrt7dXrAlbPDwjrq6ulBroeYx2eiYvLqWjeVx1/Z76AzaPStahOSoyc3sPKwGk2cTuRiBofBEAlVnfpS8InIW/Lyd5JlXnoD8DCZCey3LnUMaYqEwmrsGnzpTNsSFM8KueTnINY505fKXAduFNeu6MsRxdiK4ZJznYm2aufEtkxiZ4kW+achIxKnsxHXxpnOYiwpiSjTXBF9PBtzPJ0UocaPTye5HsIQCcd5LldSl9KTWorsWp2Ee2/FeQ2cDxEBz7kAdi5daQfoKsN67AkGQyQAbwq+nDiYfQW+fWkdNq3k62IxLYpVxpAq9ZP/7chQGM7Bnch9bHRrRz1Xz2APzeU81FPfiuWU+UrkZbOMmzBEbzbNPKpPz+axmqazhONpRiRGSZ7WNL+Zs9mI09moCBN+djZiejaqRHotFNWFDmV+XLo/30TvHNv32UZ83eTw5vO96IR10o1VKpybiMK6PMGjuCliX9E2VEdEfFb7jF363SORWBTrju3UF+qREbmnmhStzr3WmsTWWewELHxKK3JbJ6Hl8iJznfHITnPfhWRuEjuzxqQqpGnCKF9lmnAdxvM6TYWz6aiIBTWbjjI9hAnTnqdPlZJ4qyxusu83aNkUunf32dQsdqv+EU0y/objNm0MBdsyTqGvdVM0gd00NNV5ToE+6KJoXoVznfuRbIAwmax0VUU2YYIZUt7aE6Y4xy7RsKyqMh8FLWIYaz5pFyaqQh5Cw4ibJAv4lxMBE0LciJtSzcJ8zIoAflLoIcA4z5XzRbiKa3HHW0ccfM8hNJm3caRbN0wOCbO/ivbbYkHxka5K/LRDYTn2REsbEULnWD6biO45FKTlL7/W6zmiHtvCFfSF9NgGir3iJAaBjuKpInNcKhWSXspc18ZjOqBgDgb6c1e8ltVShYvogFJEWHNct7q2RTfkdW1FsmgWqiN/piqAJKRTo6BOstSleXY/l5NQhdTiHIzjnBTiJsqiJke0VHoG1tiL5+RRTnfiBjycRjCsR6Dd0GKgdX6QusxzaU/T2V0mFAOe9LsiEokho4vIakm6kU4e2U3tlCbOMiEqmcwyt3Aqju1ys28mqqR8nJnDlq2dENCcqqpmOaxnBVGx2s071FSK2EwYfYTtF2ETBmPu6ugkfOatPssmn/mrmz9iUSxVJNVlMt/lib81lvQjDgSRSAwVPR3tCs4jtJQJEZMUPzdixEujPCeEPaaAIhuX0yjGVr5s3k75b8FZ2JOuTXRSMp8Ok/c7zW11zaQtknEWpv0EUk3mnJDzAWsq2YCLAIJSBPEDCvPXUhgOxy/CVVAHFdbOcWguCYYQDznSdRUDNkzmsojOYZ9FkjB48VFXRCIR0QmSaplQREREVBBNYCMy+Fb7xpnH54TXhjtYlkltiDNoiyZRl6WjgLC+o/b6gM6jYibqqVsS9wiFyAlARRHmHt5oLqIqHAKlOG/+qhvV1VgwMecKSh7XPh2Ke2zuqa5uC3TOne2u/jcRAbYtB1LHdfgc47aFS4nipoitQEN8pWWtdoJtuZO8O/lb9WzltavIdsVP8/Ys5YIROZGZrGYakDkxsN1EfDqJIh4UzHUShSjHpCe1iIszHlcH0+SF3frZWM9gWdih+Wu4GAAnISJ/HvhzwHOAd6jqz/VpJ5rAGgwhvs2GlX6uhYov1ENmrdMwTtXG1a6R45f6YP5tlcq1fFxsqbtZgfdy3wljBWV8HcjzUKh1LKbcbDlRMO2Z9qkZi++eCutG5zk2TtJqnmFNRau8LiRHZ6ujZb+Dm/qO7H6bnuUKINJ9a9eu3Csij4vIQ875O0Xk4yLysIi8DUBVf0pV/yrwLcBf6nsvkUhEdEJchUZENMCsDLpu7XAfcGepu3nq59cDdwB3icgdVpX/Iy/vhShuWhc2lN60sHby9b3gmCrmsuY979ik10/C/mv27ePU6kclFznl7UlWrraSQ6zEQO7ATduWTqJq4mqH5fCM0x2fs9+buFY4g57tFO35G1h5iPA67KLZbL/P6jYR+ZB1fE+eF6eAqj4oIi92rvOmfhaRjwHfB/yMqn6414jYVyJRGNR3YKRWPcmvvH38L647sddN9G3KLFTDVMwnZ3UnVbcNj4zf1iHYfhUlgkFGECWZEw0l11cUdZwbcIiA5ATDDsNR8YOwxlcx0fURDee+5m1pRWTlvSaEUFnds+3S/jKwaiLQWaS2hvDh/T7lz6nqK3tcF0r9/DeA15LFhPpiO6J2F+wnkVgVNsQttIYvyF/jNc6hiZ+kNgdRve9OinFn4jQhyu2Fv51PovDDKE3KMpfR51yEFH4UgEhwPOL0Xyiq3SRHjuw/pAvwhuvoMI/5CYU2lLdvP9TuIDH08S0XfQP8eZdtqvpDwA8tOqhIJCI6IaYvbY9lWpBFbBt6Ebe+Af5ap3Xug/i5bxqpDsL2u86yqXtbnpNa3UL6OiNKqngyu9cbq6Zif27dVCQJskJqVK2hHFGS8ZVwxlfoKBo217LJ9yyC7XieYd/fYGELp1VgIO/52iA9tpyTEJGv7dhbkfpZRA7IUj/fv4S7ACIn0RuqKdJGpzGwfBNFDoaujnapwkgw4v2SuMf4H7ht25N7ze2bibM2XIctjlJX3EVpsi24HVt85NqilvqRyvV2JrqqHqF8XFG0O39bm73af5mPxVev1F6Pybd1YL91oQMh03XoFBaBUAn90hKNnISIvAd4DZmS+xLw3ar6Ll/q5z4D8GE/iMQQVlLLhpG/Q/vc1O71TfGbfHUCE35FtLKIfsaaCO182UalUnKuk3zCy4lWRrykCGmuSU40hGw8iZZDfucK7oJQJYa7kKL/0uo/70fs/i2iURDPRV65Rbi6Rb2tAyhyoG/A52DlWEUq4X6vfqNOQlXvCpyvpH5eFvaDSMBqXoSBIBjqewdgK7CB6uTrEC2XWBUExCB1CaOjcHcmVa8Suua4VL+Bg9o2DEJstSoslTvRlXESm8Buzprbgq4vZkp4JWfK1oVab+DqqbLfkNbK+NvK8UtinIqeQSo6h4rntKXTQP3Xlq5XZyzW/XrHDDXj1or5a+UZhh7xor9z19DfvtfUvIdd59ahi4qWhfXqJFaK/eEkthwFu98EWwzVsIz1Wd+4zncVs1TPsqKio2jRTxMEay5zJsyKDkND9Y1znObiJut6G8Xk7zjXGbFX6hzbxMJqoyjvdqv+MbUsD5/3F7RZ4EpHkdJOcxg90VPSOkhOIhKJiE6IJrBzNBG/aAK7p6gzkthCxM/dgaYp6gtR3Hih56VoCtDWtt2+K7UNc/aNohRHbFQR53g4CF+5K9oxoqV5HSmJmyqb4SJssZPLTQTGVeEoqJa3uTf7HitttcAiJrMl9H1nurynXb+Lnt9Ar+94CchsJLTzRhQ3DRTGfDCU56EuP8Sq0TM7XQlNJrihciUsN8nLbFGUa81UtXZiPuF1FT35xEtWe8bSqbCINTYKVr4JFam9H7s9r4mrdewjIF5i0Qae66qEQqtldX15CXPD4JYlMtqkL0QTUVjj2KK4ackQkf8b+FrgFPg94FtV9SlPvU8Cl4EZMO0Z52Qv0ajTqCMKppx24pNKAiJP25ncOzupzqRbTMhGReCZlN38Ej5zWduNwjjo2ffgWiLZE7Bt5VRRVrtEwSFSvvolAuSDSyi6LIIXnPuiTmH56GndNEgMRdz0AeBLVPV/Bn4H+M6aul+hqq+IBGIz2KF3PyJidehn3TRIDIJIqOrPqeo0P/wVstgjw8S6wgusYnXXsck6glDxB8AjFw/IkpdFaIKyfsrHro7Cawrbpg33/nqPu870VZ26zljo8LsUbdYMZhWv8ro4k6GG+ZCok1g1vg34d4EyBX5Osif6L91Y61uBJh3IqsN4FB9wtf1Gpzwz9lEbU9xyFyUdgnu5R3zjhhL36SBK8iRr0jVFxeX5ieCoHTl/SCdRqm+Vh8RRlXIPvMSiy9zXMFEGRUmbDt891Al+Sej59e63TkJEfh54nqfou1T1p/M63wVMgX8TaObLVfVREXkO8AER+W1VfTDQ393A3QBHnC/Oa6rIosrgNkjTzSi7lw2HaKlNIKyyOnNPb5k9cfqusydPCegg1LrcOXZoVP28W+GAqopsn2K7NB48k32gr1ZWTaGyTU3668SarJJ0RYRK6O1xPUisjUio6mvrykXkm4GvAb5K1f/Gq+qj+d/HReQnyTIyeYlEzmXcA3BRbtmdX2zDkJmWCUVEGC6litgb7BKRGMRSV0TuBP4e8HWqei1Q54KI3Gj2ga8GHvLVXSu2NARy0HrGTrYTurbpdt0VuH0+JHrx6QQ8/bbRQeDWMdfWheUgcG2gj9pn4Lumro5zj7XXWfVqx1DzOw4uAmxbbNF3JtJ9GyoGQSSAtwM3komQPiIi7wQQkReIiIls+Fzgl0XkN4BfA/6jqv7sZobbE+uyVW+LTqEXWrYXFIf42pzHL/L2V0c4HILhIyAh/4WQHiB0vduHVwzVQEDmbTmFDcSgDTotWof2ju2SmGxxRMV1CKr6xYHzjwJvyPc/AXzpOse1Dyg5xJXOB/wqjD7HEqX4dA4VXwlPv66znb2KrrSHNZ+Yvu1j668bRqomrFRpPK7Ow8ul4JzDL1GqW+k3Teqllb6JxeX2XXf9Ci3KItoh6ffA91txHbEjWIYXeETEDkNkt3QSkUjUYZHEOV3RZ/K1wzUAmkqWRKc4ueSx933v24TsqLF+Kupbxza3YcqDJrCe5qtjrLZvf+fuvjsHVDiPQNu+9rI6PR/uKuYiW2yY5pyJOe4zznXqEgYivopEYlfRRBTaEI1V+zkYUUTSPBZRzQgHlDPO+dA2C92oPNmDNYG74ifznYinPtXy0iTri/sUigVFed81gW31uTrirlpxk+9vC8LgG0zIRyIkXmprNFA+137CahWiY10T8TLG0iUt6hKJ2ZAV0V0RiURELSqr/pm2c6YbInyEpsOx+ywq3IpHl7KzSBmO2cvgoH11EoNE/JnXhSWsvhYKxNZqVRbq1znRY8XlNYWt6aNkWWTO5VvFughr5W+Litx9ynXs45LpqfPXK3Yy/Xru0yuKstFU3gaO6XVQUd6mbZNhrqtpbLrgO2n3v0MQiGE5dgYL5r1WTZFF82abdI5ryL/dOrtdV+Siqor4qejXv8KeT7jlcYUsn2wxlajVhTNhu1FefVKz6j1Ux1XxWXAJEv7j0rXFsVbbDNX3EqXVTKRriwC7pLSlumg7a0qfmvRTFkXrpogFsCwlunl3gyE0NK/m6ify/mvMXrv07VVe22Nrc2yPxb4n5x5LBCWAWk7HXaXXcB21f0N9dUQbpXfj5L9M2rBjnMDCGLhzXFdEIhHRDa2W5RER+wuht5/EIBGJxNDRFDW2L7Rhtk9ptoiyUBYH5eKnnLPX0fycfyzOUJxj18QVyatY4iRbi+xznis174ij5o3M9ysiJFc34Vwf8uCuWHL5YJsyp55zXeebJonKKlb+WxQyYx2IJrARy8E6/TA8aPKKrtS3dQ7QKLqqa6fJRNaepF0TWx8FcEVKJULh6kjMjmdiL8bh6iDs+h4CUqpnj6Oujz5o0H00YeNxm/ZCNLVb1k2RSCwTXl8DMyv0IAbrICI14/MpurVOvx7SOej8HHgmfbevEGdht28fqp+GVGhKiMj4iIGWj10uotKOy1WUjp0OfRxIiCvpSgQ2HUq8t1Pghse9ZEROImJvIWkDoYiYI+pv9hIiUSexvVjQ5HWX0claySPyWMSJLCR+CnEmlfr2NZ5xNqkFQmaplbYCq/3a6xcVL3n6C/bV9tqIOVZkEtvTBHapEJEvAr4LuElVv7FvO3HGDEDd7FjLzpbVpOgbCpvtyt8NasZf8S/AL0IpOYDVTIK+9lBK+eODznOe/tyt0r962sPqr+X9tb2XxmfTBqHfaRNYd+rSNC19n5qm1e93zejpTNeiXblXRB4XkYec83eKyMdF5GEReRuAqn5CVd+86L1EItEXu2jNUfNxV5TWXa61LXZ8oa8DfTVO4taxOSfWPj038bRXp3PwText7i+7Nn8eKWGlsvtsnXevdn4ZymJjmRh4oi/JFdddt5a4D7iz1J/ICHgH8HrgDuAuEbljWfcTiUREJ0TRRXuszZs5Ym+gqg8CTzinXwU8nHMOp8B7gTcuq89IJAyGsjKpy+7WcJ3YMXjydkz2t/bt0FpsUSsS6donlMfaRgRjVvyBNiuciI8DcMQ+FS4hh4CXowhxOvM61v20fSQtOLrmNjr0h/XszZa/R7LA+zgYLmYDnEdPTuI2EfmQtbUN0XE78Cnr+BJwu4jcmmf5/OMi8p1972W/FNfrRppCsmI63NbENqVVeHEDUcUbmqOmr1qRlAIhxXPAZLa2HKiYyVqnS/knTJWABjtohuojAObYrevW85Vb59uauTYShT4hOdzr24jv1zXhb1iXsAws4HF9BnwYeJ+qvq9jly5UVT8PvKXPQGxEIrEstE0atMx8Ez38KIrYTH37zwP51SYNyo8b9RjFmOqJhltuiEDRfjas0jxWEArPeLzwEQibyPi4pgAhaPR9qEEbRX6X9sL99GxgWcSiSztD4fLbor8JbN8Af5eAF1nHLwQe7TMAH6K4KaIbhiJCiIgYMBK080b/UOEfBF4mIi8RkQPgTcD9y7qXyEkMCV1SmC6a9EXNf8uL31Tfl8NZNHRtX1dJNWeXgb8dS1dR8akIoNFfwir3iou66Ava9tkVbeI2LdpXF4nQtnEBS4CxbuqBRk5CRN4DvIZMf3EJ+G5VfZeIvBV4P9kXe6+qfrTPAHzYSyJh0hTKsoPmrQELi4vc9uwgfG36s0NnlOqVxUoVkZSNUP6JQrzkhANxxVdQIgyic5GTad4N+Ff0471Hqx+rTzv8hk/UVKdXKIlzvEQhPIm4fdWJubz9+dpcsqh/my23lpmmNISeROImEbmHGp2Eqt4VOP8A8ECfTpuwl0QiYgFsOCjhRtGG+7FQSygjdhYLKK5j0qGIGrQJGbLKCbqp7VbiobBSvjbirE/R7fTn5VTs9rGkU2JxE572g2Ow9sVzrijrsNJvI06qXeUvkHa2Uxt9scUcxSqxS7GbouJ6VfB9PAP/oDq/1032566tvG0x1GLCDvpFtNEvOP2VfCJ8m92+25ZXvONc6463jijUiKmWHdJiEHNVXXyk0Hcy8G+lDsLaFdcrxSCIhIj8QxF5REQ+km9vCNSrxCdZOSoT3YZe4HUpAGvuT2zBv3u+w/G8L4JEoyLTD+gE7MnadYYrxVpqIkpq1W/Zfmh8rry+jU+ESxCbjr1tl8a0xvd0U8rpSgj2gRCWHo50OefxtKre3dFHYuUYkrjpn6rq/xMqtOKTvI7MLviDInK/qv7WugYYAZIquoUK/7XAFcm1EdFF7CSSjWd3Wh4GwUm0xErjkxTYclZ3aahdeS8mJw9yGu4qPSTOafPz2Ne57XlETaW5vGX73tV94LgtZ9C+/5r3NL6+GTb0HRvFdQ9OIoqbGvBWEfnNPBTuszzl3vgk6xlaGKopukhM+iaitC1Eq2mIHWTvbcRVrhgI9xgqRKBuq9T3tem233HMBWoy1fnrN5QPAWt4jxf6ztaMnjqJQYqb1kYkROTnReQhz/ZG4IeBlwKvAB4Dvt/XhOdc8K0TkbtNoKwzTtZiG71r8NnC26EyqvUb5OpOOOxaRW9I7t9hlV+5tqZ+xSKpLbcS0lEEjivExJn3WhPIALbZf2FT0FSXOj+sOFT42rE2nYSqvrZNPRH5V8B/8BR1ik+iqvcA9wBclFuG+wtsGaLtf3vEZ7W/6KmTaHSm2wQGobgWkeer6mP54dcDD3mqFfFJgEfI4pP85YU7NyzsqtOa9vFxWCSAXypZ1Fdz3j3Xtl3zrvt8HIIydWfclWO8fGHJTwLrXhAqPhNWE2a/GE5+0BT9NdQ/zl/DWdjlfg/saget9Q5tOYBQta5zkunPDQeeR4VdawDAdXE/axJXicAoOtMtHf9ERF5B9gl8EvhrACLyAuBHVPUNqjpdZXySpaFLlNcusZqW2S/ZJOBOvo31Q22nCqMF7sMlGvkEH3SeaxPzaQlhORphE5FAWaf2SvWdC2rTxXZoPEDMasexzEm8i1gnis4GgUEQCVX9psD5R4E3WMcri08yOLTlIhYNPV6kz5SwR3RRF7Qlw+X1kO5wbPrLCqvnncjiZc6iciLb7+JxbcZU9KfO+RrC0NVvpJPSO9BnHcQkoVoEXQnLniPZCmuDdhgEkYjYHkjanlDsPQJitYhdh0Y/iYiBYNnvYcsVYGuLG3fFXTl2RBmu7L9oW1utmOtMYH2+EcFjLG6hZb9BEU6T7qGFWKqNJVl4bEte0e7O3Lcy7JqfROQkumAd6Ug3DFd5XK2Q//WJwwLioYpoyQn2FxQ1edrxdVOoIELiK9+KvmZy9op/PMSnTindKFJqMH0NjrMlBmxRuVwMNN3pqN8PFxXX2wZNU6SOKLRRPIcIS0jn0FKZXatIXgZC42shQmll+um2EyAKrRMUeepXCAU4ZlDlY7d+JxPWuvotuAUXtX03cQcr1gm0UnzXvce+69suwBoU3zoAoiE7Jm6KRGId6MqBLNPqacvR1degUSHeMGEvpER26i/qJ7H13EBXB7UBTPDLwpCd47pi/4hEm7wNQ8ai1kyd+qJ2Bd8omiq1VeVMqn4R826rXEVusksNMxDgTNpM1hUPbcrHJYsm59i7srZ1G6F+irrtJ5TGuWddc9MuWDCtyG9iAT+JQWL/iMQqsYMcQMmhDfxEIzTxNRCFprp9V+ElsRM0m78W46iOtW//1bb9Dbau2zCenQ3HsaXhdJId0vBHIhHRDS31A3uJ+GwiMDqJ7SRuPmyx3GUF2PSqpYn93eT4mhZGnrEFzTQ95+pMaBsD7tl1fGKepuudMXmtmBpX8tV2Gj2y2z6Hpt99k4vWprFtOnLrhr6ZEWnnjWgCG7HtKERPAX1IWJxEVcfgy3ltJlVfG5RFX5VYTZ7+KuNpO9GH6tqTvloingYCVunHN28G2mhakO6smGmLYfwkeiCawO4dvL4EPRXPfQIEpviD/JGdzybaGl2DbxgizWIVH1Hw6RgsnUGbNuoHZvlNOGazTQp2X7C+0vmu33uIgwq01Zbb6lQeqF8E80udYyvgX2csM6jfThA9ZRRNYCP2FSv3z4iI2HIIMXZTxBag82Su2i7IX9F+vfWRn3Pwc1FZhNcaayjKZq+CFiInU14JJe5GAKRmdV+Mr9q3XVYO+Kce8VOoXa0tc9EsYqovL9XtEdwvirAWR+QkdgSaKjIUk9U+4qRQO9C/LRMVFmnhNd3QVx1R8Ooc8sv6DN2e13ze1m08wJ1xLLoYrJvMuxCNTuWm7XVGfV1nWwtiHRkqRaLH9X6iKZHOKvuFdn0te0ye9nzciWj9fFqrj/D2S71i2mmjouj2Ka5bzA1BxbXhElrOL22ssWr7bhpbHZY9IbdOhrRGQlAJkrikvpdoidUzdtMgEYlERCdEnURExHZARC4A/wI4BX5RVf9Nn3ain8SSoJu2B1/VSs4196xDqmG79IApaCYzr2/WlvlXzE89Y3X7a7MK93IRAY7A6CPK4wnXt9vqYv4KZM+z8flofd+LYsPiItV0899XB2QmsGnnrVXbIveKyOMi8pBz/k4R+biIPCwib8tPfwPw46r6V4Gv63s/kUiEsIoPwxv9sm5ibTMxsz5nqpZ9hSbluhwIhRy9lzlls8OaaP3m1q+OrfuwzP300U3UPasSBvbbB3+/0Hu+qu9so4RNGfXYWuI+4E77hIiMgHcArwfuAO4SkTuAFwKfyqvN+t5NFDdFdEPSUuRUpyxu0J0UOo4GXwzJPywzlmCAwLqx2HXc9p1zrZznAtdW69RUaMX57I7Me9dgOIlVQFUfFJEXO6dfBTysqp8AEJH3Am8ELpERio+wAEMQiUREN6RE/jMiogGjfizebSLyIev4HlW9p8V1tzPnGCAjDq8Gfgh4u4j8OeB9fQYEkUisDmuwftqoErnF6rzOpLXJIqp0Pa6fRMBaqo576LLw9nAV3rFpeb9tpNleprEN41kX1sLBbDmXtECAv8+p6it7dVmFqupV4Fv7DMRGXBMOFY2B0zb0IVlvzFxxu9hY2yiva6932m/SMQThTvw1fXSCWkr6YJ0GOXoL/cba0PQsNh0ocwBYc4C/S8CLrOMXAo8u614iJ7FncLmPztzIPombmriTBiyamS5iOyH09rjuG+Dvg8DLROQlwCPAm4C/3GcAPuzL5w40eFtukYndohAT4M14V1shJvq32aK8YWVvcya17WmgvRAn0MRROOWdstQF2zTPtqFeizoLcw+a/cZucL+9gabB73s1HthK0mOjBSchIu8B/hvwchG5JCJvVtUp8Fbg/cDHgB9T1Y8u624iJ7Eu+PJc94rsmr/UXcOJtPHczuM3AWjSY2wtv7fWK2wzZMqL+FAY8YIjyisvK1R4SNRUjuWEn0A19dXCUqozVOeireDYdV7eR5TWd3L1moHvFtXK0peuhpNQ1bsC5x8AHujTaRMGQSRE5N8BL88PbwaeUtVXeOp9ErhMZvM77ankWQzmhXYn/KHCvKs9hlsE3oM5wWjTTh4Tq5YQ6DxE3yJpSmFu8lrKd2FRlc6hwovzVkFbxXQD2uhwROk+CXfgXLzYxnl6wMSlZ/rSm0TkHuB9qtrbGmnZGASRUNW/ZPZF5PuBp2uqf4Wqfm71o4rwYp90EhERPSAoo5h0aDUQEQH+IvCVmx5La/QV/4RgZKey+Ey8VBNZtZzXWmcoMgNp0XYKam65hWmtuvVaKJXbB8lrPm5tqiodrbc6zi3L0CdV2lsGVqHj2xKrKaG3n0TkJFrgTwOfUdXfDZQr8HMiosC/rHM0EZG7gbsBjjjfe0Cas7TSVry0bKJRGkwHPcEq/TRSkKQ9AQrli2iqX4ijKPtJNGW9K4ijPad09LgOmtV6rqn4SdiK91bEpJtpq6xa+dyFUKzSFLsjUdABiZ96elzvNychIj8PPM9T9F2q+tP5/l3Ae2qa+XJVfVREngN8QER+W1Uf9FXMCcg9ABflluEtQVY1iS+aT6Kp+VK49Lwr2ukVgmEzeg/GNFw9rnBRPoLRpIto4iiWgLbEYVk5LlpjlZP/ljvLNUHQvpzEILE2IqGqr60rF5ExWdTCP1HTxqP538dF5CfJYpZ4iUT9YNKliHP2ETFUeEREM3rqJKK4qQGvBX5bVS/5CvPY6ImqXs73vxr4R717GyKhqOMu2nAe60qEtIS+vZZTrfuqci++THQlayfnev+Y1Fun6n0dbqMRRrTU5fpNrrzb9N3kKT40rNgnKstxvTvipiHNkm/CETWJyAtExNj+Phf4ZRH5DeDXgP+oqj+79FHUhe7ug42HLV4f7PwKXa8rha4o2tHSpBoMn93QX+FI1rM81Ec1n4QzXud+Oi0uXT3HrmMV38mWKLqHjsFwEqr6LZ5zjwJvyPc/AXzpmoe1PViTaWprUZOVP3xVoSlc/4dKutKKgrvlpGErpD39rRK9fCRWjd0Rr68J2teZbpAYEicRsQWIeQwiIuphTGDXGOBvpRgMJxERQIO1UlDu3rH9bPE9NzSVVPqF5vD20d4Cqg2a/CQK/cRCJrDhsmWLgRbScbgwITksCy1ZgiinWRwXFw82kn4/6CB1EpFIrBuhST/V1fhWBCCqaCqNvOTcbNWehXt2WhAk6WYGm4fwKD67kJ9EyQTWIUqdTGCr5Y1+El1gE5lFJ9dcF9J6DOkGuMF1pS0dCER2S9wUiUQXuBY8zrFqiqzTYqqrNZP9YfblEGROYCRZjIuxCZBNBIz3dV1Av/zCYky17fsSFNXUD6Ku3HGgs72s7XhNi3IgxpFu4YneDvDX57o1QV1LJLf/gRKbDjmrB49IJCK6waMQjoiImGMBE9hBYu8V15rqimLKLwHLHNdKwziUTUBXDo9OoE6H0CunQ42oaak6hCbYfhWr6nOZ78ZQvyVY43eeiZu6bkTF9R5ipfGT2ukwFlZsVxpsUUVXM5/Vio/c/BEOx9PZR8G9xnP9ykxiN+AfMRirtaGMYwFk1k1RcR2xr9hjcVNjsqQ9fjYRZSQ75AW5P0SiTxiOTYa5WAY2Pf6cvZdEGjPLeWFW9OCNItuUTKh1qHK3fg38Vk7+qK9tMvBVIshuWlzTZiW/6THWoQ8nsuQwHTHAX8RyseKorYOEY9JqiEApNLidX8LbBs1+EnYZhIlFkyipRl/hg8+yqZrqdMAT7SqwR/e7gLhpkIhEYp3w5bkOIcT5tCUqq+IiNknLbD8LX3A/ixh4V/Etv9s2fhI7PdEvHNSvwyp6QDkgloldEjftvXVTRAMqdumbGUZExLbAcBJdN6J1U8RasArxlREPbZCLqOUMujrT1fRRi75OeCvGSriaXeWS1oRo3bSrGGKOiXVBTeSmecyKLH5TuU6GTKOgaVZXEluj0EExXTueXLZPrptoTFlKJZ+EXVY3rqbw462iwVrjXSa3NVds540ab+tCJFburIjbVOhE9nyyX3HuCB+EtUbYWTkikegLV7+wjthLTXqGPuHCzTfU9ro2HIobaykPG16rqM7PZeUNXEslPAqNfhLQYcXfxk+iYfKthOJw2qwoso3F0CoU24vkxG66bh1EyLWmGrweQ6PiOmKPsWmz2oiIgSMLy7E72KV7aYW1uea36ce3CltbaIsW2dgGBje7W2O2N2sF3woN9Tv3H+xnc899GWHDWyHUT1s/jDV8p4MNxzMwRE6iJTRNkbbmqxuAqLYLvbGFfhmFz0TJ98EvcoJ1+ElULb62LjJ0B0Ix9MWEDk38JDDans+rEZFIrAuLimmWIeZZRorTNRGXUvyntkrrdftJuPDpMoaOZcyvywpbviMQhNEOxWcZ7tI4YpjYsQ96ldgqYhGxVCQ9tnVBRL5IRN4lIj/epn4kEtuOljLmoYsMwJLxh8baUcfQVWfQWcdQN54lJRlaB1q9G+vSZewABBiJdN5atS1yr4g8LiIPOefvFJGPi8jDIvK2ujZU9ROq+ua29xPFTUPCmlOY9sIyc17bJq/ZTrhbXywnS7+yFj+JwCTZqI/QchtrzUexaeypcjhZnbjpPuDtwLvNCREZAe8AXgdcAj4oIvcDI+B7neu/TVUf79LhfhOJOue5VU/YbXUMdeNoM8Zlmazmvg+ooonkE32uLE/nDnWFPgDLSzv3A5Ckp9d2wOeha9huc41NFLxtNMxrTUSlCwofiXy8dkBA8+xMylISy5Eu/12X5lPR5X0MnW/7vayaI6kbyxqc67KwHKuZO1T1QRF5sXP6VcDDqvoJABF5L/BGVf1e4GsW7TOKm7YBi67GVhiyoSKqaFDwblr8MrTxlNA0Nnd+G1oojiFxDRseS4J03oDbRORD1tY2RMftwKes40v5OS9E5FYReSfwx0XkO5sa329OYtexCBeh2X+Sh96oDdu9CqjlkR3IRGfXLVbVdv0m09dKn+XDshlsPTGs1F8z5iFBtDdXU1wfsRCE9joGB2fAh4H3qer7OnVZRfCHVNXPA29p2/haP30R+Qsi8lERSUXklU7Zd+ZKl4+LyJ8NXH+LiHxARH43//us9Yy8Bo5CTzVFNxAvZinIHezqFJnrCPIXmmwlLcv/S4rmEnfjXNikA/CU+whExZnOGY9df9McSvE7bvGkX/mOtuhekh7/yAP8dSQQkHEOL7KOXwg8urx7WS8eAr4BeNA+KSJ3AG8C/hhwJ/AvcmWMi7cBv6CqLwN+IT+OWCM2PfltFeKzao8tIgBNyMJy9BI39Q0V/kHgZSLyEhE5IJtL71/W/ayVSKjqx1T1456iNwLvVdUTVf194GEyZYyv3o/m+z8K/PmVDHSZWNVqrqnNrn1umPkpLH6CJqUNx772fG34tjbX9h2LDoCwdv1tl/1uLavfrYEwkqTzRgtOQkTeA/w34OUicklE3qyqU+CtwPuBjwE/pqofXdbdDEUncTvwK9ZxSPHyXFV9DEBVHxOR54QazJU+dwMccX55I11VWAuf/mANocuNaKlVSI9lQueCVF/+ahtNlkxzS6Dyb7NwPok6EVbTtS48EWHXibX5yfhErXtGVDJOotd3e5OI3EONTkJV7wqcfwB4oE+nTVg6kRCRnwee5yn6LlX96dBlnnMLvQGqeg9wTz6myz+vP/7xosUtVRksCbcBn9v0IAaC+CwyxOcwx8sXbeDXf/Pk/aPn/+5tPS793F4kHVLV1/a4rK3i5TMi8vyci3g+0NYp5OOq+srmarsPEflQfBYZ4rPIEJ/DHCLyoUXbUNU7lzGWoWAofhL3A28SkUMReQnwMuDXAvW+Od//ZiDEmURERERELAHrNoH9ehG5BPwp4D+KyPsBciXLjwG/Bfws8O2qOsuv+RHLXPb7gNeJyO+SuaB/3zrHHxEREbFvEB2o8meZEJG7cx3F3iM+iznis8gQn8Mc8VlUsRdEIiIiIiKiH4aik4iIiIiIGCB2mkgsGgZkVyEi/1BEHhGRj+TbGzY9pnWiS+z9XYeIfFJE/kf+Hixs2bNN8OVmGGTonw1jp4kEi4cB2WX8U1V9Rb6txAlniLBi778euAO4K38f9hlfkb8H+2YGex/Z928jhv5xsNNEYglhQCJ2D0XsfVU9Bd5L9j5E7BlU9UHgCef09oX+WTF2mkjUoFP89R3FW0XkN3OWe59Y6vjbl6HAz4nIr3fIX7DLKIX+AYKhf/YFQ4nd1BtDCQMyNNQ9F+CHge8hu+fvAb4f+Lb1jW6j2PnfviO+XFUfzeOgfUBEfjtfYUdEADtAJFYcBmRr0fa5iMi/Av7DioczJOz8b98Fqvpo/vdxEflJMnHcPhOJvqF/dhb7Km5qGwZkJ5G//AZfT6bg3xesNPb+NkFELojIjWYf+Gr2613wIYb+cbD1nEQdROTrgX8OPJssDMhHVPXPqupHRcSEAZlihQHZE/wTEXkFmZjlk8Bf2+ho1ghVnYqIib0/Au5dZuz9LcNzgZ+ULLT6GPi3qvqzmx3S+pDnZngNWW7pS8B3k4X6+TEReTPwh8Bf2NwIh4HocR0REREREcS+ipsiIiIiIlogEomIiIiIiCAikYiIiIiICCISiYiIiIiIICKRiIiIiIgIIhKJiIiIiIggIpGI2EmISCIiD4rI/c7583mY8B/Oj79LRP6LiFwVkWgPHhHhIBKJiJ2EqqbAtwBfKSJ2XKp/TOY49h358SHwE8A/W+f4IiK2BdGZLmKnISJvAf4J8D8BX0zmaf0aVf1lp943Av9eVX0BACMi9hY7HZYjIkJV35mHZ/nXwIuBH3AJRERERBhR3BSxD3gL8L8CJ8Df3/BYIiK2CpFIROwDvg24ThYW/Is2PJaIiK1CJBIROw0R+ZNkeYq/EfgAcN8e5jOPiOiNSCQidhYicgS8G7hPVX8GuJtMef13NzqwiIgtQiQSEbuM7wWOgL8NoKqfBr4d+Ici8iUAIvIFeW6NF+fHr8i3GzYy4oiIgSGawEbsJETkzwD/CXitqv6iU/ZjZLqJLwN+hHkmMhtf4V4XEbGPiEQiIiIiIiKIKG6KiIiIiAgiEomIiIiIiCAikYiIiIiICCISiYiIiIiIICKRiIiIiIgIIhKJiIiIiIggIpGIiIiIiAgiEomIiIiIiCAikYiIiIiICOL/B5NeRRX/wzbmAAAAAElFTkSuQmCC\\n\",\n      \"text/plain\": [\n       \"<Figure size 432x360 with 2 Axes>\"\n      ]\n     },\n     \"metadata\": {\n      \"needs_background\": \"light\"\n     },\n     \"output_type\": \"display_data\"\n    }\n   ],\n   \"source\": [\n    \"x, y = torch.meshgrid(\\n\",\n    \"    torch.linspace(-10,10,200), \\n\",\n    \"    torch.linspace(-10,10,200)\\n\",\n    \")\\n\",\n    \"xy = torch.stack([x, y], -1)\\n\",\n    \"z = rosen(xy, reduce=False)\\n\",\n    \"\\n\",\n    \"fig, ax = plt.subplots(figsize=(6,5))\\n\",\n    \"c = ax.pcolormesh(x, y, z, shading='auto', cmap='viridis_r', \\n\",\n    \"                  norm=LogNorm(vmin=z.min(), vmax=z.max()))\\n\",\n    \"ax.set_xlabel('X1', fontsize=14)\\n\",\n    \"ax.set_ylabel('X2', fontsize=14, rotation=0)\\n\",\n    \"ax.yaxis.set_label_coords(-0.15, 0.5)\\n\",\n    \"fig.colorbar(c, ax=ax)\\n\",\n    \"plt.show()\"\n   ]\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# 1. Minimize (single point)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 5,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"tensor(4900.)\"\n      ]\n     },\n     \"execution_count\": 5,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"x0 = torch.tensor([1., 8.])\\n\",\n    \"\\n\",\n    \"rosen(x0)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 6,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 4900.0000\\n\",\n      \"iter   1 - fval: 119.3775\\n\",\n      \"iter   2 - fval: 26.0475\\n\",\n      \"iter   3 - fval: 2.2403\\n\",\n      \"iter   4 - fval: 0.9742\\n\",\n      \"iter   5 - fval: 0.9085\\n\",\n      \"iter   6 - fval: 0.9070\\n\",\n      \"iter   7 - fval: 0.8999\\n\",\n      \"iter   8 - fval: 0.8847\\n\",\n      \"iter   9 - fval: 0.8506\\n\",\n      \"iter  10 - fval: 0.8048\\n\",\n      \"iter  11 - fval: 0.7286\\n\",\n      \"iter  12 - fval: 0.5654\\n\",\n      \"iter  13 - fval: 0.4128\\n\",\n      \"iter  14 - fval: 0.3506\\n\",\n      \"iter  15 - fval: 0.2667\\n\",\n      \"iter  16 - fval: 0.1814\\n\",\n      \"iter  17 - fval: 0.1401\\n\",\n      \"iter  18 - fval: 0.1074\\n\",\n      \"iter  19 - fval: 0.0681\\n\",\n      \"iter  20 - fval: 0.0385\\n\",\n      \"iter  21 - fval: 0.0196\\n\",\n      \"iter  22 - fval: 0.0157\\n\",\n      \"iter  23 - fval: 0.0063\\n\",\n      \"iter  24 - fval: 0.0030\\n\",\n      \"iter  25 - fval: 0.0009\\n\",\n      \"iter  26 - fval: 0.0002\\n\",\n      \"iter  27 - fval: 0.0000\\n\",\n      \"iter  28 - fval: 0.0000\\n\",\n      \"iter  29 - fval: 0.0000\\n\",\n      \"iter  30 - fval: 0.0000\\n\",\n      \"Optimization terminated successfully.\\n\",\n      \"         Current function value: 0.000000\\n\",\n      \"         Iterations: 30\\n\",\n      \"         Function evaluations: 39\\n\",\n      \"\\n\",\n      \"final x: tensor([1.0000, 1.0000])\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# BFGS\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='bfgs', \\n\",\n    \"    options=dict(line_search='strong-wolfe'),\\n\",\n    \"    max_iter=50,\\n\",\n    \"    disp=2\\n\",\n    \")\\n\",\n    \"print()\\n\",\n    \"print('final x: {}'.format(res.x))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 7,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 4900.0000\\n\",\n      \"iter   1 - fval: 119.3775\\n\",\n      \"iter   2 - fval: 2.7829\\n\",\n      \"iter   3 - fval: 2.7823\\n\",\n      \"iter   4 - fval: 2.7822\\n\",\n      \"iter   5 - fval: 2.7818\\n\",\n      \"iter   6 - fval: 2.7810\\n\",\n      \"iter   7 - fval: 2.7785\\n\",\n      \"iter   8 - fval: 2.7723\\n\",\n      \"iter   9 - fval: 2.7563\\n\",\n      \"iter  10 - fval: 2.7187\\n\",\n      \"iter  11 - fval: 2.6477\\n\",\n      \"iter  12 - fval: 2.5353\\n\",\n      \"iter  13 - fval: 2.2997\\n\",\n      \"iter  14 - fval: 1.8811\\n\",\n      \"iter  15 - fval: 1.5526\\n\",\n      \"iter  16 - fval: 1.1877\\n\",\n      \"iter  17 - fval: 1.0779\\n\",\n      \"iter  18 - fval: 0.9352\\n\",\n      \"iter  19 - fval: 0.6669\\n\",\n      \"iter  20 - fval: 0.5938\\n\",\n      \"iter  21 - fval: 0.4380\\n\",\n      \"iter  22 - fval: 0.3308\\n\",\n      \"iter  23 - fval: 0.2343\\n\",\n      \"iter  24 - fval: 0.1972\\n\",\n      \"iter  25 - fval: 0.1279\\n\",\n      \"iter  26 - fval: 0.0869\\n\",\n      \"iter  27 - fval: 0.0695\\n\",\n      \"iter  28 - fval: 0.0473\\n\",\n      \"iter  29 - fval: 0.0298\\n\",\n      \"iter  30 - fval: 0.0158\\n\",\n      \"iter  31 - fval: 0.0065\\n\",\n      \"iter  32 - fval: 0.0029\\n\",\n      \"iter  33 - fval: 0.0004\\n\",\n      \"iter  34 - fval: 0.0001\\n\",\n      \"iter  35 - fval: 0.0000\\n\",\n      \"iter  36 - fval: 0.0000\\n\",\n      \"iter  37 - fval: 0.0000\\n\",\n      \"iter  38 - fval: 0.0000\\n\",\n      \"Optimization terminated successfully.\\n\",\n      \"         Current function value: 0.000000\\n\",\n      \"         Iterations: 39\\n\",\n      \"         Function evaluations: 48\\n\",\n      \"\\n\",\n      \"final x: tensor([1.0000, 1.0000])\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# L-BFGS\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='l-bfgs', \\n\",\n    \"    options=dict(line_search='strong-wolfe'),\\n\",\n    \"    max_iter=50,\\n\",\n    \"    disp=2\\n\",\n    \")\\n\",\n    \"print()\\n\",\n    \"print('final x: {}'.format(res.x))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 8,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 4900.0000\\n\",\n      \"iter   1 - fval: 6.0505\\n\",\n      \"iter   2 - fval: 2.8156\\n\",\n      \"iter   3 - fval: 2.8144\\n\",\n      \"iter   4 - fval: 2.3266\\n\",\n      \"iter   5 - fval: 2.1088\\n\",\n      \"iter   6 - fval: 1.7060\\n\",\n      \"iter   7 - fval: 1.5851\\n\",\n      \"iter   8 - fval: 1.2548\\n\",\n      \"iter   9 - fval: 1.1625\\n\",\n      \"iter  10 - fval: 0.8967\\n\",\n      \"iter  11 - fval: 0.8249\\n\",\n      \"iter  12 - fval: 0.6160\\n\",\n      \"iter  13 - fval: 0.5591\\n\",\n      \"iter  14 - fval: 0.4051\\n\",\n      \"iter  15 - fval: 0.3299\\n\",\n      \"iter  16 - fval: 0.2217\\n\",\n      \"iter  17 - fval: 0.1886\\n\",\n      \"iter  18 - fval: 0.1167\\n\",\n      \"iter  19 - fval: 0.0987\\n\",\n      \"iter  20 - fval: 0.0543\\n\",\n      \"iter  21 - fval: 0.0442\\n\",\n      \"iter  22 - fval: 0.0210\\n\",\n      \"iter  23 - fval: 0.0118\\n\",\n      \"iter  24 - fval: 0.0035\\n\",\n      \"iter  25 - fval: 0.0021\\n\",\n      \"iter  26 - fval: 0.0005\\n\",\n      \"iter  27 - fval: 0.0000\\n\",\n      \"iter  28 - fval: 0.0000\\n\",\n      \"iter  29 - fval: 0.0000\\n\",\n      \"Optimization terminated successfully.\\n\",\n      \"         Current function value: 0.000000\\n\",\n      \"         Iterations: 29\\n\",\n      \"         Function evaluations: 84\\n\",\n      \"         CG iterations: 41\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# Newton CG\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='newton-cg',\\n\",\n    \"    options=dict(line_search='strong-wolfe'),\\n\",\n    \"    max_iter=50, \\n\",\n    \"    disp=2\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 10,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 4900.0000\\n\",\n      \"iter   1 - fval: 0.0000\\n\",\n      \"iter   2 - fval: 0.0000\\n\",\n      \"Optimization terminated successfully.\\n\",\n      \"         Current function value: 0.000000\\n\",\n      \"         Iterations: 2\\n\",\n      \"         Function evaluations: 3\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# Newton Exact\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='newton-exact',\\n\",\n    \"    options=dict(line_search='strong-wolfe', tikhonov=1e-4),\\n\",\n    \"    max_iter=50, \\n\",\n    \"    disp=2\\n\",\n    \")\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"markdown\",\n   \"metadata\": {},\n   \"source\": [\n    \"# minimize (batch of points)\\n\",\n    \"\\n\",\n    \"In addition to optimizing a single point, we can also optimize a batch of points.\\n\",\n    \"\\n\",\n    \"Results for batch inputs may differ from sequential point-wise optimization due to convergence stopping. Assuming that all points run for `max_iter` iterations, then they should be equivalent up to two conditions:\\n\",\n    \"1. When using line search, the optimal step size at each iteration may differ accross points. Batch mode will select a single step size for all points, whereas sequential optimization will select one per point.\\n\",\n    \"2. When using conjugate gradient (e.g. Newton-CG), there is also convergence stopping for linear inverse sub-problems.\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 11,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"data\": {\n      \"text/plain\": [\n       \"tensor(602.9989)\"\n      ]\n     },\n     \"execution_count\": 11,\n     \"metadata\": {},\n     \"output_type\": \"execute_result\"\n    }\n   ],\n   \"source\": [\n    \"torch.manual_seed(337)\\n\",\n    \"x0 = torch.randn(4,2)\\n\",\n    \"\\n\",\n    \"rosen(x0)\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 12,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 602.9989\\n\",\n      \"iter   1 - fval: 339.5845\\n\",\n      \"iter   2 - fval: 146.6088\\n\",\n      \"iter   3 - fval: 92.8062\\n\",\n      \"iter   4 - fval: 88.3703\\n\",\n      \"iter   5 - fval: 79.7213\\n\",\n      \"iter   6 - fval: 33.5407\\n\",\n      \"iter   7 - fval: 31.6904\\n\",\n      \"iter   8 - fval: 22.7846\\n\",\n      \"iter   9 - fval: 7.9474\\n\",\n      \"iter  10 - fval: 6.1061\\n\",\n      \"iter  11 - fval: 4.0852\\n\",\n      \"iter  12 - fval: 3.6830\\n\",\n      \"iter  13 - fval: 3.5514\\n\",\n      \"iter  14 - fval: 3.2786\\n\",\n      \"iter  15 - fval: 2.9614\\n\",\n      \"iter  16 - fval: 2.3316\\n\",\n      \"iter  17 - fval: 1.9711\\n\",\n      \"iter  18 - fval: 1.8731\\n\",\n      \"iter  19 - fval: 1.5464\\n\",\n      \"iter  20 - fval: 1.2185\\n\",\n      \"iter  21 - fval: 1.1106\\n\",\n      \"iter  22 - fval: 0.9249\\n\",\n      \"iter  23 - fval: 0.7769\\n\",\n      \"iter  24 - fval: 0.6506\\n\",\n      \"iter  25 - fval: 0.6083\\n\",\n      \"iter  26 - fval: 0.5571\\n\",\n      \"iter  27 - fval: 0.5008\\n\",\n      \"iter  28 - fval: 0.4542\\n\",\n      \"iter  29 - fval: 0.4272\\n\",\n      \"iter  30 - fval: 0.4088\\n\",\n      \"iter  31 - fval: 0.3964\\n\",\n      \"iter  32 - fval: 0.3924\\n\",\n      \"iter  33 - fval: 0.3894\\n\",\n      \"iter  34 - fval: 0.3876\\n\",\n      \"iter  35 - fval: 0.3837\\n\",\n      \"iter  36 - fval: 0.3795\\n\",\n      \"iter  37 - fval: 0.3708\\n\",\n      \"iter  38 - fval: 0.3569\\n\",\n      \"iter  39 - fval: 0.3319\\n\",\n      \"iter  40 - fval: 0.3180\\n\",\n      \"iter  41 - fval: 0.3149\\n\",\n      \"iter  42 - fval: 0.2971\\n\",\n      \"iter  43 - fval: 0.2936\\n\",\n      \"iter  44 - fval: 0.2769\\n\",\n      \"iter  45 - fval: 0.2584\\n\",\n      \"iter  46 - fval: 0.2416\\n\",\n      \"iter  47 - fval: 0.2346\\n\",\n      \"iter  48 - fval: 0.2293\\n\",\n      \"iter  49 - fval: 0.2224\\n\",\n      \"iter  50 - fval: 0.2204\\n\",\n      \"Warning: Maximum number of iterations has been exceeded.\\n\",\n      \"         Current function value: 0.220413\\n\",\n      \"         Iterations: 50\\n\",\n      \"         Function evaluations: 91\\n\",\n      \"\\n\",\n      \"final x: \\n\",\n      \"tensor([[1.3291, 1.7689],\\n\",\n      \"        [1.1972, 1.4332],\\n\",\n      \"        [1.2357, 1.5273],\\n\",\n      \"        [0.8715, 0.7573]])\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# BFGS\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='bfgs', \\n\",\n    \"    options=dict(line_search='strong-wolfe'),\\n\",\n    \"    max_iter=50,\\n\",\n    \"    disp=2\\n\",\n    \")\\n\",\n    \"print()\\n\",\n    \"print('final x: \\\\n{}'.format(res.x))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 14,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 602.9989\\n\",\n      \"iter   1 - fval: 339.5845\\n\",\n      \"iter   2 - fval: 95.0195\\n\",\n      \"iter   3 - fval: 21.5655\\n\",\n      \"iter   4 - fval: 5.2430\\n\",\n      \"iter   5 - fval: 4.9395\\n\",\n      \"iter   6 - fval: 4.9042\\n\",\n      \"iter   7 - fval: 4.8668\\n\",\n      \"iter   8 - fval: 4.7709\\n\",\n      \"iter   9 - fval: 4.6042\\n\",\n      \"iter  10 - fval: 4.2993\\n\",\n      \"iter  11 - fval: 3.7640\\n\",\n      \"iter  12 - fval: 3.0566\\n\",\n      \"iter  13 - fval: 3.0339\\n\",\n      \"iter  14 - fval: 2.3893\\n\",\n      \"iter  15 - fval: 2.2195\\n\",\n      \"iter  16 - fval: 2.0010\\n\",\n      \"iter  17 - fval: 1.5127\\n\",\n      \"iter  18 - fval: 1.2743\\n\",\n      \"iter  19 - fval: 1.0382\\n\",\n      \"iter  20 - fval: 0.8332\\n\",\n      \"iter  21 - fval: 0.7181\\n\",\n      \"iter  22 - fval: 0.5824\\n\",\n      \"iter  23 - fval: 0.4413\\n\",\n      \"iter  24 - fval: 0.3279\\n\",\n      \"iter  25 - fval: 0.2649\\n\",\n      \"iter  26 - fval: 0.1784\\n\",\n      \"iter  27 - fval: 0.1088\\n\",\n      \"iter  28 - fval: 0.0634\\n\",\n      \"iter  29 - fval: 0.0492\\n\",\n      \"iter  30 - fval: 0.0307\\n\",\n      \"iter  31 - fval: 0.0207\\n\",\n      \"iter  32 - fval: 0.0144\\n\",\n      \"iter  33 - fval: 0.0130\\n\",\n      \"iter  34 - fval: 0.0120\\n\",\n      \"iter  35 - fval: 0.0119\\n\",\n      \"iter  36 - fval: 0.0118\\n\",\n      \"iter  37 - fval: 0.0116\\n\",\n      \"iter  38 - fval: 0.0112\\n\",\n      \"iter  39 - fval: 0.0101\\n\",\n      \"iter  40 - fval: 0.0078\\n\",\n      \"iter  41 - fval: 0.0044\\n\",\n      \"iter  42 - fval: 0.0030\\n\",\n      \"iter  43 - fval: 0.0024\\n\",\n      \"iter  44 - fval: 0.0010\\n\",\n      \"iter  45 - fval: 0.0002\\n\",\n      \"iter  46 - fval: 0.0000\\n\",\n      \"iter  47 - fval: 0.0000\\n\",\n      \"iter  48 - fval: 0.0000\\n\",\n      \"iter  49 - fval: 0.0000\\n\",\n      \"iter  50 - fval: 0.0000\\n\",\n      \"Warning: Maximum number of iterations has been exceeded.\\n\",\n      \"         Current function value: 0.000027\\n\",\n      \"         Iterations: 50\\n\",\n      \"         Function evaluations: 56\\n\",\n      \"\\n\",\n      \"final x: \\n\",\n      \"tensor([[1.0013, 1.0026],\\n\",\n      \"        [1.0032, 1.0064],\\n\",\n      \"        [0.9962, 0.9923],\\n\",\n      \"        [0.9997, 0.9995]])\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# L-BFGS\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='l-bfgs', \\n\",\n    \"    options=dict(line_search='strong-wolfe'),\\n\",\n    \"    max_iter=50,\\n\",\n    \"    disp=2\\n\",\n    \")\\n\",\n    \"print()\\n\",\n    \"print('final x: \\\\n{}'.format(res.x))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 15,\n   \"metadata\": {\n    \"scrolled\": true\n   },\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 602.9989\\n\",\n      \"iter   1 - fval: 367.2567\\n\",\n      \"iter   2 - fval: 47.2528\\n\",\n      \"iter   3 - fval: 17.9457\\n\",\n      \"iter   4 - fval: 5.0277\\n\",\n      \"iter   5 - fval: 4.5732\\n\",\n      \"iter   6 - fval: 4.2605\\n\",\n      \"iter   7 - fval: 3.5691\\n\",\n      \"iter   8 - fval: 3.1882\\n\",\n      \"iter   9 - fval: 3.0478\\n\",\n      \"iter  10 - fval: 2.9619\\n\",\n      \"iter  11 - fval: 2.5264\\n\",\n      \"iter  12 - fval: 2.1677\\n\",\n      \"iter  13 - fval: 1.8351\\n\",\n      \"iter  14 - fval: 1.7114\\n\",\n      \"iter  15 - fval: 1.3834\\n\",\n      \"iter  16 - fval: 1.1907\\n\",\n      \"iter  17 - fval: 0.8378\\n\",\n      \"iter  18 - fval: 0.7662\\n\",\n      \"iter  19 - fval: 0.5443\\n\",\n      \"iter  20 - fval: 0.4112\\n\",\n      \"iter  21 - fval: 0.2527\\n\",\n      \"iter  22 - fval: 0.2030\\n\",\n      \"iter  23 - fval: 0.1118\\n\",\n      \"iter  24 - fval: 0.0870\\n\",\n      \"iter  25 - fval: 0.0403\\n\",\n      \"iter  26 - fval: 0.0301\\n\",\n      \"iter  27 - fval: 0.0104\\n\",\n      \"iter  28 - fval: 0.0071\\n\",\n      \"iter  29 - fval: 0.0023\\n\",\n      \"iter  30 - fval: 0.0000\\n\",\n      \"iter  31 - fval: 0.0000\\n\",\n      \"iter  32 - fval: 0.0000\\n\",\n      \"Optimization terminated successfully.\\n\",\n      \"         Current function value: 0.000000\\n\",\n      \"         Iterations: 32\\n\",\n      \"         Function evaluations: 83\\n\",\n      \"         CG iterations: 44\\n\",\n      \"\\n\",\n      \"final x: \\n\",\n      \"tensor([[1.0000, 1.0000],\\n\",\n      \"        [1.0000, 1.0000],\\n\",\n      \"        [1.0000, 1.0000],\\n\",\n      \"        [0.9999, 0.9999]])\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# Newton CG\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='newton-cg', \\n\",\n    \"    options=dict(line_search='strong-wolfe'),\\n\",\n    \"    max_iter=50,\\n\",\n    \"    disp=2\\n\",\n    \")\\n\",\n    \"print()\\n\",\n    \"print('final x: \\\\n{}'.format(res.x))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": 16,\n   \"metadata\": {},\n   \"outputs\": [\n    {\n     \"name\": \"stdout\",\n     \"output_type\": \"stream\",\n     \"text\": [\n      \"initial fval: 602.9989\\n\",\n      \"iter   1 - fval: 5.2148\\n\",\n      \"iter   2 - fval: 4.4287\\n\",\n      \"iter   3 - fval: 3.6888\\n\",\n      \"iter   4 - fval: 2.7710\\n\",\n      \"iter   5 - fval: 1.9319\\n\",\n      \"iter   6 - fval: 1.5444\\n\",\n      \"iter   7 - fval: 1.0819\\n\",\n      \"iter   8 - fval: 0.8638\\n\",\n      \"iter   9 - fval: 0.5068\\n\",\n      \"iter  10 - fval: 0.3768\\n\",\n      \"iter  11 - fval: 0.2248\\n\",\n      \"iter  12 - fval: 0.1487\\n\",\n      \"iter  13 - fval: 0.0758\\n\",\n      \"iter  14 - fval: 0.0412\\n\",\n      \"iter  15 - fval: 0.0125\\n\",\n      \"iter  16 - fval: 0.0055\\n\",\n      \"iter  17 - fval: 0.0003\\n\",\n      \"iter  18 - fval: 0.0000\\n\",\n      \"iter  19 - fval: 0.0000\\n\",\n      \"iter  20 - fval: 0.0000\\n\",\n      \"Optimization terminated successfully.\\n\",\n      \"         Current function value: 0.000000\\n\",\n      \"         Iterations: 20\\n\",\n      \"         Function evaluations: 27\\n\",\n      \"\\n\",\n      \"final x: \\n\",\n      \"tensor([[1., 1.],\\n\",\n      \"        [1., 1.],\\n\",\n      \"        [1., 1.],\\n\",\n      \"        [1., 1.]])\\n\"\n     ]\n    }\n   ],\n   \"source\": [\n    \"# Newton Exact\\n\",\n    \"res = minimize(\\n\",\n    \"    rosen, x0, \\n\",\n    \"    method='newton-exact', \\n\",\n    \"    options=dict(line_search='strong-wolfe', tikhonov=1e-4),\\n\",\n    \"    max_iter=50,\\n\",\n    \"    disp=2\\n\",\n    \")\\n\",\n    \"print()\\n\",\n    \"print('final x: \\\\n{}'.format(res.x))\"\n   ]\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  },\n  {\n   \"cell_type\": \"code\",\n   \"execution_count\": null,\n   \"metadata\": {},\n   \"outputs\": [],\n   \"source\": []\n  }\n ],\n \"metadata\": {\n  \"kernelspec\": {\n   \"display_name\": \"Python 3\",\n   \"language\": \"python\",\n   \"name\": \"python3\"\n  },\n  \"language_info\": {\n   \"codemirror_mode\": {\n    \"name\": \"ipython\",\n    \"version\": 3\n   },\n   \"file_extension\": \".py\",\n   \"mimetype\": \"text/x-python\",\n   \"name\": \"python\",\n   \"nbconvert_exporter\": \"python\",\n   \"pygments_lexer\": \"ipython3\",\n   \"version\": \"3.8.5\"\n  }\n },\n \"nbformat\": 4,\n \"nbformat_minor\": 4\n}"
  },
  {
    "path": "examples/scipy_benchmark.py",
    "content": "\"\"\"\nA comparison of pytorch-minimize solvers to the analogous solvers from\nscipy.optimize.\n\nPytorch-minimize uses autograd to compute 1st- and 2nd-order derivatives\nimplicitly, therefore derivative functions need not be provided or known.\nIn contrast, scipy.optimize requires that they be provided, or else it will\nuse imprecise numerical approximations. For fair comparison I am providing\nderivative functions to scipy.optimize in this script. In general, however,\nwe will not have access to these functions, so applications of scipy.optimize\nare far more limited.\n\n\"\"\"\nimport torch\nfrom torchmin import minimize\nfrom torchmin.benchmarks import rosen\nfrom scipy import optimize\n\n# Many scipy optimizers convert the data to double-precision, so\n# we will use double precision in torch for a fair comparison\ntorch.set_default_dtype(torch.float64)\n\n\ndef print_header(title, num_breaks=1):\n    print('\\n'*num_breaks + '='*50)\n    print(' '*20 + title)\n    print('='*50 + '\\n')\n\n\ndef main():\n    torch.manual_seed(991)\n    x0 = torch.randn(100)\n    x0_np = x0.numpy()\n\n    print('\\ninitial loss: %0.4f\\n' % rosen(x0))\n\n\n    # ---- BFGS ----\n    print_header('BFGS')\n\n    print('-'*19 + ' pytorch ' + '-'*19)\n    res = minimize(rosen, x0, method='bfgs', tol=1e-5, disp=True)\n\n    print('\\n' + '-'*20 + ' scipy ' + '-'*20)\n    res = optimize.minimize(\n        optimize.rosen, x0_np,\n        method='bfgs',\n        jac=optimize.rosen_der,\n        tol=1e-5,\n        options=dict(disp=True)\n    )\n\n\n    # ---- Newton CG ----\n    print_header('Newton-CG')\n\n    print('-'*19 + ' pytorch ' + '-'*19)\n    res = minimize(rosen, x0, method='newton-cg', tol=1e-5, disp=True)\n\n    print('\\n' + '-'*20 + ' scipy ' + '-'*20)\n    res = optimize.minimize(\n        optimize.rosen, x0_np,\n        method='newton-cg',\n        jac=optimize.rosen_der,\n        hessp=optimize.rosen_hess_prod,\n        tol=1e-5,\n        options=dict(disp=True)\n    )\n\n\n    # ---- Newton Exact ----\n    # NOTE: Scipy does not have a precise analogue to \"newton-exact,\" but they\n    # have something very close called \"trust-exact.\" Like newton-exact,\n    # trust-exact also uses Cholesky factorization of the explicit Hessian\n    # matrix. However, whereas newton-exact first computes the newton direction\n    # and then uses line search to determine a step size, trust-exact first\n    # specifies a step size boundary and then solves for the optimal newton\n    # step within this boundary (a constrained optimization problem).\n\n    print_header('Newton-Exact')\n\n    print('-'*19 + ' pytorch ' + '-'*19)\n    res = minimize(rosen, x0, method='newton-exact', tol=1e-5, disp=True)\n\n    print('\\n' + '-'*20 + ' scipy ' + '-'*20)\n    res = optimize.minimize(\n        optimize.rosen, x0_np,\n        method='trust-exact',\n        jac=optimize.rosen_der,\n        hess=optimize.rosen_hess,\n        options=dict(gtol=1e-5, disp=True)\n    )\n\n    print()\n\n\nif __name__ == '__main__':\n    main()"
  },
  {
    "path": "examples/train_mnist_Minimizer.py",
    "content": "import argparse\nimport matplotlib.pyplot as plt\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom torchvision import datasets\n\nfrom torchmin import Minimizer\n\n\ndef MLPClassifier(input_size, hidden_sizes, num_classes):\n    layers = []\n    for i, hidden_size in enumerate(hidden_sizes):\n        layers.append(nn.Linear(input_size, hidden_size))\n        layers.append(nn.ReLU())\n        input_size = hidden_size\n    layers.append(nn.Linear(input_size, num_classes))\n    layers.append(nn.LogSoftmax(-1))\n\n    return nn.Sequential(*layers)\n\n\n@torch.no_grad()\ndef evaluate(model):\n    train_output = model(X_train)\n    test_output = model(X_test)\n    train_loss = F.nll_loss(train_output, y_train)\n    test_loss = F.nll_loss(test_output, y_test)\n    print('Loss (cross-entropy):\\n  train: {:.4f}  -  test: {:.4f}'.format(train_loss, test_loss))\n    train_accuracy = (train_output.argmax(-1) == y_train).float().mean()\n    test_accuracy = (test_output.argmax(-1) == y_test).float().mean()\n    print('Accuracy:\\n  train: {:.4f}  -  test: {:.4f}'.format(train_accuracy, test_accuracy))\n\n\nif __name__ == '__main__':\n    parser = argparse.ArgumentParser()\n    parser.add_argument('--mnist_root', type=str, required=True,\n                        help='root path for the MNIST dataset')\n    parser.add_argument('--method', type=str, default='newton-cg',\n                        help='optimization method to use')\n    parser.add_argument('--device', type=str, default='cpu',\n                        help='device to use for training')\n    parser.add_argument('--quiet', action='store_true',\n                        help='whether to train in quiet mode (no loss printing)')\n    parser.add_argument('--plot_weight', action='store_true',\n                        help='whether to plot the learned weights')\n    args = parser.parse_args()\n\n    device = torch.device(args.device)\n\n\n    # --------------------------------------------\n    #            Load MNIST dataset\n    # --------------------------------------------\n\n    train_data = datasets.MNIST(args.mnist_root, train=True)\n    X_train = (train_data.data.float().view(-1, 784) / 255.).to(device)\n    y_train = train_data.targets.to(device)\n\n    test_data = datasets.MNIST(args.mnist_root, train=False)\n    X_test = (test_data.data.float().view(-1, 784) / 255.).to(device)\n    y_test = test_data.targets.to(device)\n\n\n    # --------------------------------------------\n    #           Initialize model\n    # --------------------------------------------\n    mlp = MLPClassifier(784, hidden_sizes=[50], num_classes=10)\n    mlp = mlp.to(device)\n\n    print('-------- Initial evaluation ---------')\n    evaluate(mlp)\n\n\n    # --------------------------------------------\n    #         Fit model with Minimizer\n    # --------------------------------------------\n    optimizer = Minimizer(mlp.parameters(),\n                          method=args.method,\n                          tol=1e-6,\n                          max_iter=200,\n                          disp=0 if args.quiet else 2)\n\n    def closure():\n        optimizer.zero_grad()\n        output = mlp(X_train)\n        loss = F.nll_loss(output, y_train)\n        # loss.backward()  <-- do not call backward!\n        return loss\n\n    loss = optimizer.step(closure)\n\n    # --------------------------------------------\n    #          Evaluate fitted model\n    # --------------------------------------------\n    print('-------- Final evaluation ---------')\n    evaluate(mlp)\n\n    if args.plot_weight:\n        weight = mlp[0].weight.data.cpu().view(-1, 28, 28)\n        vmin, vmax = weight.min(), weight.max()\n        fig, axes = plt.subplots(4, 4, figsize=(6, 6))\n        axes = axes.ravel()\n        for i in range(len(axes)):\n            axes[i].matshow(weight[i], cmap='gray', vmin=0.5 * vmin, vmax=0.5 * vmax)\n            axes[i].set_xticks(())\n            axes[i].set_yticks(())\n        plt.show()"
  },
  {
    "path": "pyproject.toml",
    "content": "[build-system]\nrequires = [\"setuptools>=61.0\", \"wheel\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"pytorch-minimize\"\ndynamic = [\"version\"]\ndescription = \"Newton and Quasi-Newton optimization with PyTorch\"\nreadme = \"README.md\"\nrequires-python = \">=3.7\"\nlicense = {text = \"MIT License\"}\nauthors = [\n    {name = \"Reuben Feinman\", email = \"reuben.feinman@nyu.edu\"}\n]\nclassifiers = [\n    \"Programming Language :: Python :: 3\",\n    \"License :: OSI Approved :: MIT License\",\n    \"Operating System :: OS Independent\",\n]\ndependencies = [\n    \"numpy>=1.18.0\",\n    \"scipy>=1.6\",\n    \"torch>=1.9.0\",\n]\n\n[project.optional-dependencies]\ndev = [\n    \"pytest\",\n]\ndocs = [\n    \"sphinx==3.5.3\",\n    \"jinja2<3.1\",\n    \"sphinx_rtd_theme==0.5.2\",\n    \"readthedocs-sphinx-search==0.3.2\",\n]\n\n[project.urls]\nDocumentation = \"https://pytorch-minimize.readthedocs.io\"\nHomepage = \"https://github.com/rfeinman/pytorch-minimize\"\n\n[tool.setuptools]\ninclude-package-data = false  # Only include .py files\n\n[tool.setuptools.dynamic]\nversion = {attr = \"torchmin._version.__version__\"}\n\n[tool.setuptools.packages.find]\nexclude = [\"tests*\", \"docs*\", \"examples*\", \"tmp*\"]\n\n[tool.pytest.ini_options]\ntestpaths = [\"tests\"]\npython_files = [\"test_*.py\"]\npython_classes = [\"Test*\"]\npython_functions = [\"test_*\"]\naddopts = [\n    \"-v\",\n    \"--strict-markers\",\n    \"--tb=short\",\n]\nmarkers = [\n    \"slow: marks tests as slow (deselect with '-m \\\"not slow\\\"')\",\n    \"cuda: marks tests that require CUDA\",\n]"
  },
  {
    "path": "tests/__init__.py",
    "content": ""
  },
  {
    "path": "tests/conftest.py",
    "content": "\"\"\"Shared pytest fixtures for torchmin tests.\"\"\"\nimport pytest\nimport torch\n\nfrom torchmin.benchmarks import rosen\n\n\n@pytest.fixture\ndef random_seed():\n    \"\"\"Set random seed for reproducibility.\"\"\"\n    torch.manual_seed(42)\n    yield 42\n\n\n# =============================================================================\n# Objective Function Fixtures\n# =============================================================================\n# To add a new test problem, create a fixture that returns a dict with:\n#   - 'objective': callable, the objective function\n#   - 'x0': Tensor, initial point\n#   - 'solution': Tensor, known optimal solution\n#   - 'name': str, descriptive name for the problem\n\n\n@pytest.fixture(scope='session')\ndef least_squares_problem():\n    \"\"\"\n    Generate a least squares problem for testing optimization algorithms.\n\n    Creates a linear regression problem: min ||Y - X @ B||^2\n    where X is N x D, Y is N x M, and B is D x M.\n\n    This is a session-scoped fixture, so the same problem instance is used\n    across all tests for consistency.\n\n    Returns\n    -------\n    dict\n        Dictionary containing:\n        - objective: callable, the objective function\n        - x0: Tensor, initial parameter values (zeros)\n        - solution: Tensor, the true solution\n        - X: Tensor, design matrix\n        - Y: Tensor, target values\n    \"\"\"\n    torch.manual_seed(42)\n    N, D, M = 100, 7, 5\n    X = torch.randn(N, D)\n    Y = torch.randn(N, M)\n\n    def objective(B):\n        return torch.sum((Y - X @ B) ** 2)\n\n    # target B\n    #trueB = torch.linalg.inv(X.T @ X) @ X.T @ Y\n    trueB = torch.linalg.lstsq(X, Y).solution # XB = Y (solve for B)\n\n    # initial B\n    B0 = torch.zeros(D, M)\n\n    return {\n        'objective': objective,\n        'x0': B0,\n        'solution': trueB,\n        'X': X,\n        'Y': Y,\n        'name': 'least_squares',\n    }\n\n\n@pytest.fixture(scope='session')\ndef rosenbrock_problem():\n    \"\"\"Rosenbrock function (banana function).\"\"\"\n\n    torch.manual_seed(42)\n    D = 10\n    x0 = torch.zeros(D)\n    x_sol = torch.ones(D)\n\n    return {\n        'objective': rosen,\n        'x0': x0,\n        'solution': x_sol,\n        'name': 'rosenbrock',\n    }\n\n\n# =============================================================================\n# Other Fixtures\n# =============================================================================\n\n\n@pytest.fixture(params=['cpu', 'cuda'])\ndef device(request):\n    \"\"\"\n    Parametrize tests across CPU and CUDA devices.\n\n    Automatically skips CUDA tests if CUDA is not available.\n    \"\"\"\n    if request.param == 'cuda' and not torch.cuda.is_available():\n        pytest.skip('CUDA not available')\n    return torch.device(request.param)\n"
  },
  {
    "path": "tests/test_imports.py",
    "content": "\"\"\"Test that all public APIs are importable and accessible.\"\"\"\nimport pytest\n\n\ndef test_import_main_package():\n    \"\"\"Test importing the main torchmin package.\"\"\"\n    import torchmin\n    assert hasattr(torchmin, '__version__')\n\n\ndef test_import_core_functions():\n    \"\"\"Test importing core minimize functions.\"\"\"\n    from torchmin import minimize, minimize_constr, Minimizer\n\n\ndef test_import_benchmarks():\n    \"\"\"Test importing benchmark functions.\"\"\"\n    from torchmin.benchmarks import rosen\n\n\n@pytest.mark.parametrize('method', [\n    'bfgs',\n    'l-bfgs',\n    'cg',\n    'newton-cg',\n    'newton-exact',\n    'trust-ncg',\n    # 'trust-krylov',\n    'trust-exact',\n    'dogleg',\n])\ndef test_method_available(method):\n    \"\"\"Test that all advertised methods are available and callable.\"\"\"\n    import torch\n    from torchmin import minimize\n\n    # Simple quadratic objective: f(x) = ||x||^2\n    x0 = torch.zeros(2)\n    result = minimize(lambda x: x.square().sum(), x0, method=method, max_iter=1)\n    assert result is not None\n"
  },
  {
    "path": "tests/torchmin/__init__.py",
    "content": ""
  },
  {
    "path": "tests/torchmin/test_bounds.py",
    "content": "import pytest\nimport torch\nfrom scipy.optimize import Bounds\n\nfrom torchmin import minimize, minimize_constr\nfrom torchmin.benchmarks import rosen\n\n\n@pytest.mark.parametrize(\n    'method',\n    ['l-bfgs-b', 'trust-constr'],\n)\ndef test_equivalent_bounds(method):\n    x0 = torch.tensor([-1.0, 1.5])\n\n    def minimize_with_bounds(bounds):\n        return minimize_constr(\n            rosen,\n            x0,\n            method=method,\n            bounds=bounds,\n            tol=1e-6,\n        )\n\n    def assert_equivalent(src_result, tgt_result):\n        return torch.testing.assert_close(\n            src_result.x,\n            tgt_result.x,\n            rtol=1e-5,\n            atol=1e-3,\n            msg=f\"Solution {src_result.x} not close to expected {tgt_result.x}\"\n        )\n\n    result_0 = minimize_with_bounds(\n        bounds=(torch.tensor([-2.0, -2.0]), torch.tensor([2.0, 2.0]))\n    )\n\n    equivalent_bounds_to_test = [\n        ([-2.0, -2.0], [2.0, 2.0]),\n        (-2.0, 2.0),\n        Bounds(-2.0, 2.0),\n    ]\n    for bounds in equivalent_bounds_to_test:\n        result = minimize_with_bounds(bounds)\n        assert_equivalent(result, result_0)\n        print(f'Test passed with bounds: {bounds}')\n\n\ndef test_invalid_bounds():\n    x0 = torch.tensor([-1.0, 1.5])\n\n    invalid_bounds_to_test = [\n        (torch.tensor([-2.0]), torch.tensor([2.0, 2.0])),\n        (-2.0,),\n        torch.tensor([-2.0, -2.0, 2.0, 2.0]),\n    ]\n\n    for bounds in invalid_bounds_to_test:\n        with pytest.raises(Exception):\n            result = minimize_constr(\n                rosen,\n                x0,\n                method='l-bfgs-b',\n                bounds=bounds,\n            )\n\n\n# TODO: remove this block\nif __name__ == '__main__':\n    test_equivalent_bounds(method='l-bfgs-b')\n    test_invalid_bounds()\n"
  },
  {
    "path": "tests/torchmin/test_minimize.py",
    "content": "\"\"\"\nTest unconstrained minimization methods on various objective functions.\n\nThis module tests all unconstrained optimization methods provided by torchmin\non a variety of test problems. To add a new test problem:\n1. Create a fixture in conftest.py (or here) following the standard format\n2. Add the fixture name to the PROBLEMS list below\n\nNOTE: The problem fixtures are defined in `conftest.py`\n\"\"\"\nimport pytest\nimport torch\n\nfrom torchmin import minimize\n\n\n# All unconstrained optimization methods\nALL_METHODS = [\n    'bfgs',\n    'l-bfgs',\n    'cg',\n    'newton-cg',\n    'newton-exact',\n    'trust-ncg',\n    # 'trust-krylov',  # TODO: fix trust-krylov solver and add this back\n    'trust-exact',\n    'dogleg',\n]\n\n# All test problems - add new problem fixture names here\nPROBLEMS = [\n    'least_squares_problem',\n    'rosenbrock_problem',\n]\n\n\n# =============================================================================\n# Fixtures\n# =============================================================================\n\n@pytest.fixture\ndef problem(request):\n    \"\"\"\n    Indirect fixture that routes to specific problem fixtures.\n\n    This allows parametrizing over multiple problem fixtures without\n    duplicating test code.\n    \"\"\"\n    return request.getfixturevalue(request.param)\n\n\n# =============================================================================\n# Tests\n# =============================================================================\n\n@pytest.mark.parametrize('problem', PROBLEMS, indirect=True)\n@pytest.mark.parametrize('method', ALL_METHODS)\ndef test_minimize(method, problem):\n    \"\"\"Test minimization methods on various optimization problems.\"\"\"\n    result = minimize(problem['objective'], problem['x0'], method=method)\n\n    # TODO: should we check result.success??\n    # assert result.success, (\n    #     f\"Optimization failed for method {method} on {problem['name']}: \"\n    #     f\"{result.message}\"\n    # )\n\n    torch.testing.assert_close(\n        result.x, problem['solution'],\n        rtol=1e-4, atol=1e-3,\n        msg=f\"Solution incorrect for method {method} on {problem['name']}\"\n    )\n"
  },
  {
    "path": "tests/torchmin/test_minimize_constr.py",
    "content": "\"\"\"\nTest constrained minimization methods.\n\nThis module tests the minimize_constr function on various types of constraints,\nincluding inactive constraints (that don't affect the solution) and active\nconstraints (that bind at the optimum).\n\"\"\"\nimport pytest\nimport torch\n\nfrom torchmin import minimize, minimize_constr\n# from torchmin.constrained.trust_constr import _minimize_trust_constr as minimize_constr\nfrom torchmin.benchmarks import rosen\n\n\n# Test constants\nRTOL = 1e-2\nATOL = 1e-2\nMAX_ITER = 50\nTOLERANCE = 1e-6  # Numerical tolerance for constraint satisfaction\n\n\n# =============================================================================\n# Fixtures\n# =============================================================================\n\n@pytest.fixture(scope='session')\ndef rosen_start():\n    \"\"\"Starting point for Rosenbrock optimization tests.\"\"\"\n    return torch.tensor([1., 8.])\n\n\n@pytest.fixture(scope='session')\ndef rosen_unconstrained_solution(rosen_start):\n    \"\"\"Compute the unconstrained Rosenbrock solution for comparison.\"\"\"\n    result = minimize(\n        rosen,\n        rosen_start,\n        method='l-bfgs',\n        options=dict(line_search='strong-wolfe'),\n        max_iter=MAX_ITER,\n        disp=0\n    )\n    return result\n\n\n# =============================================================================\n# Constraint Functions\n# =============================================================================\n\ndef sum_constraint(x):\n    \"\"\"Sum constraint: sum(x).\"\"\"\n    return x.sum()\n\n\ndef norm_constraint(x):\n    \"\"\"L2 norm squared constraint: ||x||^2.\"\"\"\n    return x.square().sum()\n\n\n# =============================================================================\n# Tests\n# =============================================================================\n\nclass TestUnconstrainedBaseline:\n    \"\"\"Test unconstrained optimization as a baseline.\"\"\"\n\n    def test_rosen_unconstrained(self, rosen_start):\n        \"\"\"Test unconstrained Rosenbrock minimization.\"\"\"\n        result = minimize(\n            rosen,\n            rosen_start,\n            method='l-bfgs',\n            options=dict(line_search='strong-wolfe'),\n            max_iter=MAX_ITER,\n            disp=0\n        )\n        assert result.success\n\n\nclass TestInactiveConstraints:\n    \"\"\"\n    Test constraints that are inactive (non-binding) at the optimum.\n\n    When the constraint is loose enough, the constrained solution should\n    match the unconstrained solution.\n    \"\"\"\n\n    @pytest.mark.parametrize('constraint_fun,constraint_name', [\n        (sum_constraint, 'sum'),\n        (norm_constraint, 'norm'),\n    ])\n    def test_loose_constraints(\n        self,\n        rosen_start,\n        rosen_unconstrained_solution,\n        constraint_fun,\n        constraint_name\n    ):\n        \"\"\"Test that loose constraints don't affect the solution.\"\"\"\n        # Upper bound of 10 is loose enough to not affect the solution\n        result = minimize_constr(\n            rosen,\n            rosen_start,\n            method='trust-constr',\n            constr=dict(fun=constraint_fun, ub=10.),\n            max_iter=MAX_ITER,\n            disp=0\n        )\n\n        torch.testing.assert_close(\n            result.x,\n            rosen_unconstrained_solution.x,\n            rtol=RTOL,\n            atol=ATOL,\n            msg=f\"Loose {constraint_name} constraint affected the solution\"\n        )\n\n\nclass TestActiveConstraints:\n    \"\"\"\n    Test constraints that are active (binding) at the optimum.\n\n    When the constraint is tight, it should bind at the specified bound\n    and produce a different solution than the unconstrained case.\n    \"\"\"\n\n    @pytest.mark.parametrize('constraint_fun,ub', [\n        (sum_constraint, 1.),\n        (norm_constraint, 1.),\n    ])\n    def test_tight_constraints(self, rosen_start, constraint_fun, ub):\n        \"\"\"Test that tight constraints bind at the specified bound.\"\"\"\n        result = minimize_constr(\n            rosen,\n            rosen_start,\n            method='trust-constr',\n            constr=dict(fun=constraint_fun, ub=ub),\n            max_iter=MAX_ITER,\n            disp=0\n        )\n\n        # Verify the constraint is satisfied (with numerical tolerance)\n        constraint_value = constraint_fun(result.x)\n        assert constraint_value <= ub + TOLERANCE, (\n            f\"Constraint violated: {constraint_value:.6f} > {ub}\"\n        )\n\n\ndef test_frankwolfe_birkhoff_polytope():\n    n, d = 5, 10\n    X = torch.randn(n, d)\n    Y = torch.flipud(torch.eye(n)) @ X\n\n    def fun(P):\n        return torch.sum((X @ X.T @ P - P @ Y @ Y.T) ** 2)\n\n    init_P = torch.eye(n)\n    init_err = torch.sum((X - init_P @ Y) ** 2)\n    res = minimize_constr(\n        fun,\n        init_P,\n        method='frank-wolfe',\n        constr='birkhoff',\n    )\n    est_P = res.x\n    final_err = torch.sum((X - est_P @ Y) ** 2)\n    torch.testing.assert_close(est_P.sum(0), torch.ones(n))\n    torch.testing.assert_close(est_P.sum(1), torch.ones(n))\n    assert final_err < 0.01 * init_err\n\n\ndef test_frankwolfe_tracenorm():\n    dim = 5\n    init_X = torch.zeros((dim, dim))\n    eye = torch.eye(dim)\n\n    def fun(X):\n        return torch.sum((X - eye) ** 2)\n\n    res = minimize_constr(\n        fun,\n        init_X,\n        method='frank-wolfe',\n        constr='tracenorm',\n        options=dict(t=5.0),\n    )\n    est_X = res.x\n    torch.testing.assert_close(est_X, eye, rtol=1e-2, atol=1e-2)\n\n    res = minimize_constr(\n        fun,\n        init_X,\n        method='frank-wolfe',\n        constr='tracenorm',\n        options=dict(t=1.0),\n    )\n    est_X = res.x\n    torch.testing.assert_close(est_X, 0.2 * eye, rtol=1e-2, atol=1e-2)\n\n\ndef test_lbfgsb_simple_quadratic():\n    \"\"\"Test L-BFGS-B on a simple bounded quadratic problem.\n\n    Minimize: f(x) = (x1 - 2)^2 + (x2 - 1)^2\n    Subject to: 0 <= x1 <= 1.5, 0 <= x2 <= 2\n\n    The unconstrained minimum is at (2, 1), but x1 is constrained,\n    so the optimal solution should be at (1.5, 1).\n    \"\"\"\n\n    def fun(x):\n        return (x[0] - 2)**2 + (x[1] - 1)**2\n\n    x0 = torch.tensor([0.5, 0.5])\n    lb = torch.tensor([0.0, 0.0])\n    ub = torch.tensor([1.5, 2.0])\n\n    result = minimize_constr(\n        fun,\n        x0,\n        method='l-bfgs-b',\n        bounds=(lb, ub),\n        options=dict(gtol=1e-6, ftol=1e-9),\n    )\n\n    # Check if close to expected solution\n    expected_x = torch.tensor([1.5, 1.0])\n    expected_f = 0.25\n\n    torch.testing.assert_close(\n        result.x,\n        expected_x,\n        rtol=1e-5,\n        atol=1e-4,\n        msg=f\"Solution {result.x} not close to expected {expected_x}\"\n    )\n\n    assert abs(result.fun - expected_f) < 1e-4, \\\n        f\"Function value {result.fun} not close to expected {expected_f}\"\n\n\ndef test_lbfgsb_rosenbrock():\n    \"\"\"Test L-BFGS-B on Rosenbrock function with bounds.\n\n    Minimize: f(x,y) = (1-x)^2 + 100(y-x^2)^2\n    Subject to: -2 <= x <= 2, -2 <= y <= 2\n\n    The unconstrained minimum is at (1, 1).\n    \"\"\"\n\n    x0 = torch.tensor([-1.0, 1.5])\n    lb = torch.tensor([-2.0, -2.0])\n    ub = torch.tensor([2.0, 2.0])\n\n    result = minimize_constr(\n        rosen,\n        x0,\n        method='l-bfgs-b',\n        bounds=(lb, ub),\n        options=dict(gtol=1e-6, ftol=1e-9, max_iter=100),\n    )\n\n    # Check if close to expected solution\n    expected_x = torch.tensor([1.0, 1.0])\n\n    torch.testing.assert_close(\n        result.x,\n        expected_x,\n        rtol=1e-5,\n        atol=1e-3,\n        msg=f\"Solution {result.x} not close to expected {expected_x}\"\n    )\n\n    assert result.fun < 1e-6, \\\n        f\"Function value {result.fun} not close to 0\"\n\n\ndef test_lbfgsb_active_constraints():\n    \"\"\"Test L-BFGS-B with multiple active constraints.\n\n    Minimize: f(x) = sum(x_i^2)\n    Subject to: x_i >= 1 for all i\n\n    The solution should be all ones (on the boundary).\n    \"\"\"\n\n    def fun(x):\n        return (x**2).sum()\n\n    n = 5\n    x0 = torch.ones(n) * 2.0\n    lb = torch.ones(n)\n    ub = torch.ones(n) * 10.0\n\n    result = minimize_constr(\n        fun,\n        x0,\n        method='l-bfgs-b',\n        bounds=(lb, ub),\n        options=dict(gtol=1e-6, ftol=1e-9),\n    )\n\n    # Check if close to expected solution\n    expected_x = torch.ones(n)\n    expected_f = float(n)\n\n    torch.testing.assert_close(\n        result.x,\n        expected_x,\n        rtol=1e-5,\n        atol=1e-4,\n        msg=f\"Solution {result.x} not close to expected {expected_x}\"\n    )\n\n    assert abs(result.fun - expected_f) < 1e-4, \\\n        f\"Function value {result.fun} not close to expected {expected_f}\"\n"
  },
  {
    "path": "torchmin/__init__.py",
    "content": "from ._version import __version__\nfrom .minimize import minimize\nfrom .minimize_constr import minimize_constr\nfrom .lstsq import least_squares\nfrom .optim import Minimizer, ScipyMinimizer\n\n__all__ = ['minimize', 'minimize_constr', 'least_squares',\n           'Minimizer', 'ScipyMinimizer']\n"
  },
  {
    "path": "torchmin/_optimize.py",
    "content": "# **** Optimization Utilities ****\n#\n# This module contains general utilies for optimization such as\n# `_status_message` and `OptimizeResult` (coming soon).\n\n\n# standard status messages of optimizers (derived from SciPy)\n_status_message = {\n    'success': 'Optimization terminated successfully.',\n    'maxfev': 'Maximum number of function evaluations has been exceeded.',\n    'maxiter': 'Maximum number of iterations has been exceeded.',\n    'pr_loss': 'Desired error not necessarily achieved due to precision loss.',\n    'nan': 'NaN result encountered.',\n    'out_of_bounds': 'The result is outside of the provided bounds.',\n    'callback_stop': 'Stopped by the user through the callback function.',\n}\n"
  },
  {
    "path": "torchmin/_version.py",
    "content": "__version__ = \"0.1.0\""
  },
  {
    "path": "torchmin/benchmarks.py",
    "content": "import torch\n\n__all__ = ['rosen', 'rosen_der', 'rosen_hess', 'rosen_hess_prod']\n\n\n# =============================\n#     Rosenbrock function\n# =============================\n\n\ndef rosen(x, reduce=True):\n    val = 100. * (x[...,1:] - x[...,:-1]**2)**2 + (1 - x[...,:-1])**2\n    if reduce:\n        return val.sum()\n    else:\n        # don't reduce batch dimensions\n        return val.sum(-1)\n\n\ndef rosen_der(x):\n    xm = x[..., 1:-1]\n    xm_m1 = x[..., :-2]\n    xm_p1 = x[..., 2:]\n    der = torch.zeros_like(x)\n    der[..., 1:-1] = (200 * (xm - xm_m1**2) -\n                 400 * (xm_p1 - xm**2) * xm - 2 * (1 - xm))\n    der[..., 0] = -400 * x[..., 0] * (x[..., 1] - x[..., 0]**2) - 2 * (1 - x[..., 0])\n    der[..., -1] = 200 * (x[..., -1] - x[..., -2]**2)\n    return der\n\n\ndef rosen_hess(x):\n    H = torch.diag_embed(-400*x[..., :-1], 1) - \\\n        torch.diag_embed(400*x[..., :-1], -1)\n    diagonal = torch.zeros_like(x)\n    diagonal[..., 0] = 1200*x[..., 0].square() - 400*x[..., 1] + 2\n    diagonal[..., -1] = 200\n    diagonal[..., 1:-1] = 202 + 1200*x[..., 1:-1].square() - 400*x[..., 2:]\n    H.diagonal(dim1=-2, dim2=-1).add_(diagonal)\n    return H\n\n\ndef rosen_hess_prod(x, p):\n    Hp = torch.zeros_like(x)\n    Hp[..., 0] = (1200 * x[..., 0]**2 - 400 * x[..., 1] + 2) * p[..., 0] - \\\n                 400 * x[..., 0] * p[..., 1]\n    Hp[..., 1:-1] = (-400 * x[..., :-2] * p[..., :-2] +\n                     (202 + 1200 * x[..., 1:-1]**2 - 400 * x[..., 2:]) * p[..., 1:-1] -\n                     400 * x[..., 1:-1] * p[..., 2:])\n    Hp[..., -1] = -400 * x[..., -2] * p[..., -2] + 200*p[..., -1]\n    return Hp"
  },
  {
    "path": "torchmin/bfgs.py",
    "content": "from abc import ABC, abstractmethod\nimport torch\nfrom torch import Tensor\nfrom scipy.optimize import OptimizeResult\n\nfrom ._optimize import _status_message\nfrom .function import ScalarFunction\nfrom .line_search import strong_wolfe\n\n\nclass HessianUpdateStrategy(ABC):\n    def __init__(self):\n        self.n_updates = 0\n\n    @abstractmethod\n    def solve(self, grad):\n        pass\n\n    @abstractmethod\n    def _update(self, s, y, rho_inv):\n        pass\n\n    def update(self, s, y):\n        rho_inv = y.dot(s)\n        if rho_inv <= 1e-10:\n            # curvature is negative; do not update\n            return\n        self._update(s, y, rho_inv)\n        self.n_updates += 1\n\n\nclass L_BFGS(HessianUpdateStrategy):\n    def __init__(self, x, history_size=100):\n        super().__init__()\n        self.y = []\n        self.s = []\n        self.rho = []\n        self.H_diag = 1.\n        self.alpha = x.new_empty(history_size)\n        self.history_size = history_size\n\n    def solve(self, grad):\n        mem_size = len(self.y)\n        d = grad.neg()\n        for i in reversed(range(mem_size)):\n            self.alpha[i] = self.s[i].dot(d) * self.rho[i]\n            d.add_(self.y[i], alpha=-self.alpha[i])\n        d.mul_(self.H_diag)\n        for i in range(mem_size):\n            beta_i = self.y[i].dot(d) * self.rho[i]\n            d.add_(self.s[i], alpha=self.alpha[i] - beta_i)\n\n        return d\n\n    def _update(self, s, y, rho_inv):\n        if len(self.y) == self.history_size:\n            self.y.pop(0)\n            self.s.pop(0)\n            self.rho.pop(0)\n        self.y.append(y)\n        self.s.append(s)\n        self.rho.append(rho_inv.reciprocal())\n        self.H_diag = rho_inv / y.dot(y)\n\n\nclass BFGS(HessianUpdateStrategy):\n    def __init__(self, x, inverse=True):\n        super().__init__()\n        self.inverse = inverse\n        if inverse:\n            self.I = torch.eye(x.numel(), device=x.device, dtype=x.dtype)\n            self.H = self.I.clone()\n        else:\n            self.B = torch.eye(x.numel(), device=x.device, dtype=x.dtype)\n\n    def solve(self, grad):\n        if self.inverse:\n            return torch.matmul(self.H, grad.neg())\n        else:\n            return torch.cholesky_solve(grad.neg().unsqueeze(1),\n                                        torch.linalg.cholesky(self.B)).squeeze(1)\n\n    def _update(self, s, y, rho_inv):\n        rho = rho_inv.reciprocal()\n        if self.inverse:\n            if self.n_updates == 0:\n                self.H.mul_(rho_inv / y.dot(y))\n            R = torch.addr(self.I, s, y, alpha=-rho)\n            torch.addr(\n                torch.linalg.multi_dot((R, self.H, R.t())),\n                s, s, alpha=rho, out=self.H)\n        else:\n            if self.n_updates == 0:\n                self.B.mul_(rho * y.dot(y))\n            Bs = torch.mv(self.B, s)\n            self.B.addr_(y, y, alpha=rho)\n            self.B.addr_(Bs, Bs, alpha=-1./s.dot(Bs))\n\n\n@torch.no_grad()\ndef _minimize_bfgs_core(\n        fun, x0, lr=1., low_mem=False, history_size=100, inv_hess=True,\n        max_iter=None, line_search='strong-wolfe', gtol=1e-5, xtol=1e-8,\n        gtd_tol=1e-10, normp=float('inf'), callback=None, disp=0,\n        return_all=False):\n    \"\"\"Minimize a multivariate function with BFGS or L-BFGS.\n\n    We choose from BFGS/L-BFGS with the `low_mem` argument.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize\n    x0 : Tensor\n        Initialization point\n    lr : float\n        Step size for parameter updates. If using line search, this will be\n        used as the initial step size for the search.\n    low_mem : bool\n        Whether to use L-BFGS, the \"low memory\" variant of the BFGS algorithm.\n    history_size : int\n        History size for L-BFGS hessian estimates. Ignored if `low_mem=False`.\n    inv_hess : bool\n        Whether to parameterize the inverse hessian vs. the hessian with BFGS.\n        Ignored if `low_mem=True` (L-BFGS always parameterizes the inverse).\n    max_iter : int, optional\n        Maximum number of iterations to perform. Defaults to 200 * x0.numel()\n    line_search : str\n        Line search specifier. Currently the available options are\n        {'none', 'strong-wolfe'}.\n    gtol : float\n        Termination tolerance on 1st-order optimality (gradient norm).\n    xtol : float\n        Termination tolerance on function/parameter changes.\n    gtd_tol : float\n        Tolerence used to verify that the search direction is a *descent\n        direction*. The directional derivative `gtd` should be negative for\n        descent; this check ensures that `gtd < -xtol` (sufficiently negative).\n    normp : Number or str\n        The norm type to use for termination conditions. Can be any value\n        supported by `torch.norm` p argument.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool, optional\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n    \"\"\"\n    lr = float(lr)\n    disp = int(disp)\n    if max_iter is None:\n        max_iter = x0.numel() * 200\n    if low_mem and not inv_hess:\n        raise ValueError('inv_hess=False is not available for L-BFGS.')\n\n    # construct scalar objective function\n    sf = ScalarFunction(fun, x0.shape)\n    closure = sf.closure\n    if line_search == 'strong-wolfe':\n        dir_evaluate = sf.dir_evaluate\n\n    # compute initial f(x) and f'(x)\n    x = x0.detach().view(-1).clone(memory_format=torch.contiguous_format)\n    f, g, _, _ = closure(x)\n    if disp > 1:\n        print('initial fval: %0.4f' % f)\n    if return_all:\n        allvecs = [x]\n\n    # initial settings\n    if low_mem:\n        hess = L_BFGS(x, history_size)\n    else:\n        hess = BFGS(x, inv_hess)\n    d = g.neg()\n    t = min(1., g.norm(p=1).reciprocal()) * lr\n    n_iter = 0\n\n    # BFGS iterations\n    for n_iter in range(1, max_iter+1):\n\n        # ==================================\n        #   compute Quasi-Newton direction\n        # ==================================\n\n        if n_iter > 1:\n            d = hess.solve(g)\n\n        # directional derivative\n        gtd = g.dot(d)\n\n        # check if directional derivative is below tolerance\n        if gtd > -gtd_tol:\n            warnflag = 4\n            msg = 'A non-descent direction was encountered.'\n            break\n\n        # ======================\n        #   update parameter\n        # ======================\n\n        if line_search == 'none':\n            # no line search, move with fixed-step\n            x_new = x + d.mul(t)\n            f_new, g_new, _, _ = closure(x_new)\n        elif line_search == 'strong-wolfe':\n            #  Determine step size via strong-wolfe line search\n            f_new, g_new, t, ls_evals = \\\n                strong_wolfe(dir_evaluate, x, t, d, f, g, gtd)\n            x_new = x + d.mul(t)\n        else:\n            raise ValueError('invalid line_search option {}.'.format(line_search))\n\n        if disp > 1:\n            print('iter %3d - fval: %0.4f' % (n_iter, f_new))\n        if return_all:\n            allvecs.append(x_new)\n        if callback is not None:\n            if callback(x_new):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n\n\n        # ================================\n        #   update hessian approximation\n        # ================================\n\n        s = x_new.sub(x)\n        y = g_new.sub(g)\n\n        hess.update(s, y)\n\n        # =========================================\n        #   check conditions and update buffers\n        # =========================================\n\n        # convergence by insufficient progress\n        if (s.norm(p=normp) <= xtol) | ((f_new - f).abs() <= xtol):\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n        # update state\n        f[...] = f_new\n        x.copy_(x_new)\n        g.copy_(g_new)\n        t = lr\n\n        # convergence by 1st-order optimality\n        if g.norm(p=normp) <= gtol:\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n        # precision loss; exit\n        if ~f.isfinite():\n            warnflag = 2\n            msg = _status_message['pr_loss']\n            break\n\n    else:\n        # if we get to the end, the maximum num. iterations was reached\n        warnflag = 1\n        msg = _status_message['maxiter']\n\n    if disp:\n        print(msg)\n        print(\"         Current function value: %f\" % f)\n        print(\"         Iterations: %d\" % n_iter)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n    result = OptimizeResult(fun=f, x=x.view_as(x0), grad=g.view_as(x0),\n                            status=warnflag, success=(warnflag==0),\n                            message=msg, nit=n_iter, nfev=sf.nfev)\n    if not low_mem:\n        if inv_hess:\n            result['hess_inv'] = hess.H.view(2 * x0.shape)\n        else:\n            result['hess'] = hess.B.view(2 * x0.shape)\n    if return_all:\n        result['allvecs'] = allvecs\n\n    return result\n\n\ndef _minimize_bfgs(\n        fun, x0, lr=1., inv_hess=True, max_iter=None,\n        line_search='strong-wolfe', gtol=1e-5, xtol=1e-8, gtd_tol=1e-10,\n        normp=float('inf'), callback=None, disp=0, return_all=False):\n    \"\"\"Minimize a multivariate function with BFGS\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    lr : float\n        Step size for parameter updates. If using line search, this will be\n        used as the initial step size for the search.\n    inv_hess : bool\n        Whether to parameterize the inverse hessian vs. the hessian with BFGS.\n    max_iter : int, optional\n        Maximum number of iterations to perform. Defaults to\n        ``200 * x0.numel()``.\n    line_search : str\n        Line search specifier. Currently the available options are\n        {'none', 'strong-wolfe'}.\n    gtol : float\n        Termination tolerance on 1st-order optimality (gradient norm).\n    xtol : float\n        Termination tolerance on function/parameter changes.\n    gtd_tol : float\n        Tolerence used to verify that the search direction is a *descent\n        direction*. The directional derivative `gtd` should be negative for\n        descent; this check ensures that `gtd < -xtol` (sufficiently negative).\n    normp : Number or str\n        The norm type to use for termination conditions. Can be any value\n        supported by :func:`torch.norm`.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool, optional\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n    \"\"\"\n    return _minimize_bfgs_core(\n        fun, x0, lr, low_mem=False, inv_hess=inv_hess, max_iter=max_iter,\n        line_search=line_search, gtol=gtol, xtol=xtol, gtd_tol=gtd_tol,\n        normp=normp, callback=callback, disp=disp, return_all=return_all)\n\n\ndef _minimize_lbfgs(\n        fun, x0, lr=1., history_size=100, max_iter=None,\n        line_search='strong-wolfe', gtol=1e-5, xtol=1e-8, gtd_tol=1e-10,\n        normp=float('inf'), callback=None, disp=0, return_all=False):\n    \"\"\"Minimize a multivariate function with L-BFGS\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    lr : float\n        Step size for parameter updates. If using line search, this will be\n        used as the initial step size for the search.\n    history_size : int\n        History size for L-BFGS hessian estimates.\n    max_iter : int, optional\n        Maximum number of iterations to perform. Defaults to\n        ``200 * x0.numel()``.\n    line_search : str\n        Line search specifier. Currently the available options are\n        {'none', 'strong-wolfe'}.\n    gtol : float\n        Termination tolerance on 1st-order optimality (gradient norm).\n    xtol : float\n        Termination tolerance on function/parameter changes.\n    gtd_tol : float\n        Tolerence used to verify that the search direction is a *descent\n        direction*. The directional derivative `gtd` should be negative for\n        descent; this check ensures that `gtd < -xtol` (sufficiently negative).\n    normp : Number or str\n        The norm type to use for termination conditions. Can be any value\n        supported by :func:`torch.norm`.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool, optional\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n    \"\"\"\n    return _minimize_bfgs_core(\n        fun, x0, lr, low_mem=True, history_size=history_size,\n        max_iter=max_iter, line_search=line_search, gtol=gtol, xtol=xtol,\n        gtd_tol=gtd_tol, normp=normp, callback=callback, disp=disp,\n        return_all=return_all)\n"
  },
  {
    "path": "torchmin/cg.py",
    "content": "import torch\nfrom scipy.optimize import OptimizeResult\n\nfrom ._optimize import _status_message\nfrom .function import ScalarFunction\nfrom .line_search import strong_wolfe\n\n\ndot = lambda u,v: torch.dot(u.view(-1), v.view(-1))\n\n\n@torch.no_grad()\ndef _minimize_cg(fun, x0, max_iter=None, gtol=1e-5, normp=float('inf'),\n                 callback=None, disp=0, return_all=False):\n    \"\"\"Minimize a scalar function of one or more variables using\n    nonlinear conjugate gradient.\n\n    The algorithm is described in Nocedal & Wright (2006) chapter 5.2.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    max_iter : int\n        Maximum number of iterations to perform. Defaults to\n        ``200 * x0.numel()``.\n    gtol : float\n        Termination tolerance on 1st-order optimality (gradient norm).\n    normp : float\n        The norm type to use for termination conditions. Can be any value\n        supported by :func:`torch.norm`.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool, optional\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    \"\"\"\n    disp = int(disp)\n    if max_iter is None:\n        max_iter = x0.numel() * 200\n\n    # Construct scalar objective function\n    sf = ScalarFunction(fun, x_shape=x0.shape)\n    closure = sf.closure\n    dir_evaluate = sf.dir_evaluate\n\n    # initialize\n    x = x0.detach().flatten()\n    f, g, _, _ = closure(x)\n    if disp > 1:\n        print('initial fval: %0.4f' % f)\n    if return_all:\n        allvecs = [x]\n    d = g.neg()\n    grad_norm = g.norm(p=normp)\n    old_f = f + g.norm() / 2  # Sets the initial step guess to dx ~ 1\n\n    for niter in range(1, max_iter + 1):\n        # delta/gtd\n        delta = dot(g, g)\n        gtd = dot(g, d)\n\n        # compute initial step guess based on (f - old_f) / gtd\n        t0 = torch.clamp(2.02 * (f - old_f) / gtd, max=1.0)\n        if t0 <= 0:\n            warnflag = 4\n            msg = 'Initial step guess is negative.'\n            break\n        old_f = f\n\n        # buffer to store next direction vector\n        cached_step = [None]\n\n        def polak_ribiere_powell_step(t, g_next):\n            y = g_next - g\n            beta = torch.clamp(dot(y, g_next) / delta, min=0)\n            d_next = -g_next + d.mul(beta)\n            torch.norm(g_next, p=normp, out=grad_norm)\n            return t, d_next\n\n        def descent_condition(t, f_next, g_next):\n            # Polak-Ribiere+ needs an explicit check of a sufficient\n            # descent condition, which is not guaranteed by strong Wolfe.\n            cached_step[:] = polak_ribiere_powell_step(t, g_next)\n            t, d_next = cached_step\n\n            # Accept step if it leads to convergence.\n            cond1 = grad_norm <= gtol\n\n            # Accept step if sufficient descent condition applies.\n            cond2 = dot(d_next, g_next) <= -0.01 * dot(g_next, g_next)\n\n            return cond1 | cond2\n\n        # Perform CG step\n        f, g, t, ls_evals = \\\n                strong_wolfe(dir_evaluate, x, t0, d, f, g, gtd,\n                             c2=0.4, extra_condition=descent_condition)\n\n        # Update x and then update d (in that order)\n        x = x + d.mul(t)\n        if t == cached_step[0]:\n            # Reuse already computed results if possible\n            d = cached_step[1]\n        else:\n            d = polak_ribiere_powell_step(t, g)[1]\n\n        if disp > 1:\n            print('iter %3d - fval: %0.4f' % (niter, f))\n        if return_all:\n            allvecs.append(x)\n        if callback is not None:\n            if callback(x):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n\n        # check optimality\n        if grad_norm <= gtol:\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n    else:\n        # if we get to the end, the maximum iterations was reached\n        warnflag = 1\n        msg = _status_message['maxiter']\n\n    if disp:\n        print(\"%s%s\" % (\"Warning: \" if warnflag != 0 else \"\", msg))\n        print(\"         Current function value: %f\" % f)\n        print(\"         Iterations: %d\" % niter)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n\n    result = OptimizeResult(fun=f, x=x.view_as(x0), grad=g.view_as(x0),\n                            status=warnflag, success=(warnflag == 0),\n                            message=msg, nit=niter, nfev=sf.nfev)\n    if return_all:\n        result['allvecs'] = allvecs\n    return result"
  },
  {
    "path": "torchmin/constrained/frankwolfe.py",
    "content": "import warnings\nimport numpy as np\nimport torch\nfrom numbers import Number\nfrom scipy.optimize import (\n    linear_sum_assignment,\n    OptimizeResult,\n)\nfrom scipy.sparse.linalg import svds\n\nfrom .._optimize import _status_message\nfrom ..function import ScalarFunction\n\n\n@torch.no_grad()\ndef _minimize_frankwolfe(\n        fun, x0, constr='tracenorm', t=None, max_iter=None, gtol=1e-5,\n        normp=float('inf'), callback=None, disp=0):\n    \"\"\"Minimize a scalar function of a matrix with Frank-Wolfe (a.k.a.\n    conditional gradient).\n\n    The algorithm is described in [1]_. The following constraints are currently \n    supported:\n\n        - Trace norm. The matrix is constrained to have trace norm (a.k.a.\n          nuclear norm) less than t.\n        - Birkhoff polytope. The matrix is constrained to lie in the Birkhoff\n          polytope, i.e. over the space of doubly stochastic matrices. Requires\n          a square matrix.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    constr : str\n        Which constraint to use. Must be either 'tracenorm' or 'birkhoff'.\n    t : float, optional\n        Maximum allowed trace norm. Required when using the 'tracenorm' constr;\n        otherwise unused.\n    max_iter : int, optional\n        Maximum number of iterations to perform.\n    gtol : float\n        Termination tolerance on 1st-order optimality (gradient norm).\n    normp : float\n        The norm type to use for termination conditions. Can be any value\n        supported by :func:`torch.norm`.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    References\n    ----------\n    .. [1] Martin Jaggi, \"Revisiting Frank-Wolfe: Projection-Free Sparse Convex\n       Optimization\", ICML 2013.\n\n    \"\"\"\n    assert isinstance(constr, str)\n    constr = constr.lower()\n    if constr in {'tracenorm', 'trace-norm'}:\n        assert t is not None, \\\n            f'Argument `t` is required when using the trace-norm constraint.'\n        assert isinstance(t, Number), \\\n            f'Argument `t` must be a Number but got {type(t)}'\n        constr = 'tracenorm'\n    elif constr in {'birkhoff', 'birkhoff-polytope'}:\n        if t is not None:\n            warnings.warn(\n                'Argument `t` was provided but is unused for the'\n                'birkhoff-polytope constraint.'\n            )\n        constr = 'birkhoff'\n    else:\n        raise ValueError(f'Invalid constr: \"{constr}\".')\n\n    if x0.ndim != 2:\n        raise ValueError(\n            f'Optimization variable `x` must be a matrix to use Frank-Wolfe.'\n        )\n\n    m, n = x0.shape\n\n    if constr == 'birkhoff':\n        if m != n:\n            raise RuntimeError('Initial iterate must be a square matrix.')\n\n        if not ((x0.sum(0) == 1).all() and (x0.sum(1) == 1).all()):\n            raise RuntimeError('Initial iterate must be doubly stochastic.')\n\n    disp = int(disp)\n    if max_iter is None:\n        max_iter = m * 100\n\n    # Construct scalar objective function\n    sf = ScalarFunction(fun, x_shape=x0.shape)\n    closure = sf.closure\n    dir_evaluate = sf.dir_evaluate\n\n    x = x0.detach()\n    for niter in range(max_iter):\n        f, g, _, _ = closure(x)\n\n        if constr == 'tracenorm':\n            u, s, vh = svds(g.detach().numpy(), k=1)\n            uvh = x.new_tensor(u @ vh)\n            alpha = 2. / (niter + 2.)\n            x = torch.lerp(x, -t * uvh, weight=alpha)\n        elif constr == 'birkhoff':\n            row_ind, col_ind = linear_sum_assignment(g.detach().numpy())\n            alpha = 2. / (niter + 2.)\n            x = (1 - alpha) * x\n            x[row_ind, col_ind] += alpha\n        else:\n            raise ValueError\n\n        if disp > 1:\n            print('iter %3d - fval: %0.4f' % (niter, f))\n\n        if callback is not None:\n            if callback(x):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n\n        # check optimality\n        grad_norm = g.norm(p=normp)\n        if grad_norm <= gtol:\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n    else:\n        # if we get to the end, the maximum iterations was reached\n        warnflag = 1\n        msg = _status_message['maxiter']\n\n    if disp:\n        print(\"%s%s\" % (\"Warning: \" if warnflag != 0 else \"\", msg))\n        print(\"         Current function value: %f\" % f)\n        print(\"         Iterations: %d\" % niter)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n\n    result = OptimizeResult(fun=f, x=x.view_as(x0), grad=g.view_as(x0),\n                            status=warnflag, success=(warnflag == 0),\n                            message=msg, nit=niter, nfev=sf.nfev)\n\n    return result\n"
  },
  {
    "path": "torchmin/constrained/lbfgsb.py",
    "content": "import torch\nfrom torch import Tensor\nfrom scipy.optimize import OptimizeResult\n\nfrom .._optimize import _status_message\nfrom ..function import ScalarFunction\n\n\nclass L_BFGS_B:\n    \"\"\"Limited-memory BFGS Hessian approximation for bounded optimization.\n\n    This class maintains the L-BFGS history and provides methods for\n    computing search directions within bound constraints.\n    \"\"\"\n    def __init__(self, x, history_size=10):\n        self.y = []\n        self.s = []\n        self.rho = []\n        self.theta = 1.0  # scaling factor\n        self.history_size = history_size\n        self.n_updates = 0\n\n    def solve(self, grad, x, lb, ub, theta=None):\n        \"\"\"Compute search direction: -H * grad, respecting bounds.\n\n        Parameters\n        ----------\n        grad : Tensor\n            Current gradient\n        x : Tensor\n            Current point\n        lb : Tensor\n            Lower bounds\n        ub : Tensor\n            Upper bounds\n        theta : float, optional\n            Scaling factor. If None, uses stored value.\n\n        Returns\n        -------\n        d : Tensor\n            Search direction\n        \"\"\"\n        if theta is not None:\n            self.theta = theta\n\n        mem_size = len(self.y)\n        if mem_size == 0:\n            # No history yet, use scaled steepest descent\n            return grad.neg() * self.theta\n\n        # Two-loop recursion\n        alpha = torch.zeros(mem_size, dtype=grad.dtype, device=grad.device)\n        q = grad.clone()\n\n        # First loop: backward pass\n        for i in reversed(range(mem_size)):\n            alpha[i] = self.rho[i] * self.s[i].dot(q)\n            q.add_(self.y[i], alpha=-alpha[i])\n\n        # Apply initial Hessian approximation\n        r = q * self.theta\n\n        # Second loop: forward pass\n        for i in range(mem_size):\n            beta = self.rho[i] * self.y[i].dot(r)\n            r.add_(self.s[i], alpha=alpha[i] - beta)\n\n        return -r\n\n    def update(self, s, y):\n        \"\"\"Update the L-BFGS history with new correction pair.\n\n        Parameters\n        ----------\n        s : Tensor\n            Step vector (x_new - x)\n        y : Tensor\n            Gradient difference (g_new - g)\n        \"\"\"\n        # Check curvature condition\n        sy = s.dot(y)\n        if sy <= 1e-10:\n            # Skip update if curvature is too small\n            return False\n\n        yy = y.dot(y)\n\n        # Update scaling factor (theta = s'y / y'y)\n        if yy > 1e-10:\n            self.theta = sy / yy\n\n        # Update history\n        if len(self.y) >= self.history_size:\n            self.y.pop(0)\n            self.s.pop(0)\n            self.rho.pop(0)\n\n        self.y.append(y.clone())\n        self.s.append(s.clone())\n        self.rho.append(1.0 / sy)\n        self.n_updates += 1\n\n        return True\n\n\ndef _project_bounds(x, lb, ub):\n    \"\"\"Project x onto the box [lb, ub].\"\"\"\n    return torch.clamp(x, lb, ub)\n\n\ndef _gradient_projection(x, g, lb, ub):\n    \"\"\"Compute the projected gradient.\n\n    Returns the projected gradient and identifies the active set.\n    \"\"\"\n    # Project gradient: if at bound and gradient points out, set to zero\n    g_proj = g.clone()\n\n    # At lower bound with positive gradient\n    at_lb = (x <= lb + 1e-10) & (g > 0)\n    g_proj[at_lb] = 0\n\n    # At upper bound with negative gradient\n    at_ub = (x >= ub - 1e-10) & (g < 0)\n    g_proj[at_ub] = 0\n\n    return g_proj\n\n\n@torch.no_grad()\ndef _minimize_lbfgsb(\n        fun, x0, bounds=None, lr=1.0, history_size=10,\n        max_iter=None, gtol=1e-5, ftol=1e-9,\n        normp=float('inf'), callback=None, disp=0, return_all=False):\n    \"\"\"Minimize a scalar function with L-BFGS-B.\n\n    L-BFGS-B [1]_ is a limited-memory quasi-Newton method for bound-constrained\n    optimization. It extends L-BFGS to handle box constraints.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    bounds : tuple of Tensor, optional\n        Bounds for variables as (lb, ub) where lb and ub are Tensors\n        of the same shape as x0. Use float('-inf') and float('inf')\n        for unbounded variables. If None, equivalent to unbounded.\n    lr : float\n        Step size for parameter updates (used as initial step in line search).\n    history_size : int\n        History size for L-BFGS Hessian estimates.\n    max_iter : int, optional\n        Maximum number of iterations. Defaults to 200 * x0.numel().\n    gtol : float\n        Termination tolerance on projected gradient norm.\n    ftol : float\n        Termination tolerance on function/parameter changes.\n    normp : Number or str\n        Norm type for termination conditions. Can be any value\n        supported by torch.norm.\n    callback : callable, optional\n        Function to call after each iteration: callback(x).\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool, optional\n        Set to True to return a list of the best solution at each iteration.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    References\n    ----------\n    .. [1] Byrd, R. H., Lu, P., Nocedal, J., & Zhu, C. (1995). A limited memory\n       algorithm for bound constrained optimization. SIAM Journal on\n       Scientific Computing, 16(5), 1190-1208.\n    \"\"\"\n    lr = float(lr)\n    disp = int(disp)\n    if max_iter is None:\n        max_iter = x0.numel() * 200\n\n    # Set up bounds\n    x = x0.detach().view(-1).clone(memory_format=torch.contiguous_format)\n    n = x.numel()\n\n    if bounds is None:\n        lb = torch.full_like(x, float('-inf'))\n        ub = torch.full_like(x, float('inf'))\n    else:\n        lb, ub = bounds\n        lb = lb.detach().view(-1).clone(memory_format=torch.contiguous_format)\n        ub = ub.detach().view(-1).clone(memory_format=torch.contiguous_format)\n\n        if lb.shape != x.shape or ub.shape != x.shape:\n            raise ValueError('Bounds must have the same shape as x0')\n\n    # Project initial point onto feasible region\n    x = _project_bounds(x, lb, ub)\n\n    # Construct scalar objective function\n    sf = ScalarFunction(fun, x0.shape)\n    closure = sf.closure\n\n    # Compute initial function and gradient\n    f, g, _, _ = closure(x)\n\n    if disp > 1:\n        print('initial fval: %0.4f' % f)\n        print('initial gnorm: %0.4e' % g.norm(p=normp))\n\n    if return_all:\n        allvecs = [x.clone()]\n\n    # Initialize L-BFGS approximation\n    hess = L_BFGS_B(x, history_size)\n\n    # Main iteration loop\n    for n_iter in range(1, max_iter + 1):\n\n        # ========================================\n        #   Check projected gradient convergence\n        # ========================================\n\n        g_proj = _gradient_projection(x, g, lb, ub)\n        g_proj_norm = g_proj.norm(p=normp)\n\n        if disp > 1:\n            print('iter %3d - fval: %0.4f, gnorm: %0.4e' %\n                  (n_iter, f, g_proj_norm))\n\n        if g_proj_norm <= gtol:\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n        # ========================================\n        #   Compute search direction\n        # ========================================\n\n        # Use projected gradient for search direction computation\n        # This ensures we only move in directions away from active constraints\n        d = hess.solve(g_proj, x, lb, ub)\n\n        # Ensure direction is a descent direction w.r.t. original gradient\n        gtd = g.dot(d)\n        if gtd > -1e-10:\n            # Not a descent direction, use projected steepest descent\n            d = -g_proj\n            gtd = g.dot(d)\n\n        # Find maximum step length that keeps us feasible\n        alpha_max = 1.0\n        for i in range(x.numel()):\n            if d[i] > 1e-10:\n                # Moving toward upper bound\n                if ub[i] < float('inf'):\n                    alpha_max = min(alpha_max, (ub[i] - x[i]) / d[i])\n            elif d[i] < -1e-10:\n                # Moving toward lower bound\n                if lb[i] > float('-inf'):\n                    alpha_max = min(alpha_max, (lb[i] - x[i]) / d[i])\n\n        # Take a step with line search on the feasible segment\n        # Simple backtracking: try alpha_max, 0.5*alpha_max, etc.\n        alpha = alpha_max\n        for _ in range(10):\n            x_new = x + alpha * d\n            x_new = _project_bounds(x_new, lb, ub)\n            f_new, g_new, _, _ = closure(x_new)\n\n            # Armijo condition (sufficient decrease)\n            if f_new <= f + 1e-4 * alpha * gtd:\n                break\n            alpha *= 0.5\n        else:\n            # Line search failed, take a small step\n            x_new = x + 0.01 * alpha_max * d\n            x_new = _project_bounds(x_new, lb, ub)\n            f_new, g_new, _, _ = closure(x_new)\n\n        if return_all:\n            allvecs.append(x_new.clone())\n\n        if callback is not None:\n            if callback(x_new.view_as(x0)):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n\n        # ========================================\n        #   Update Hessian approximation\n        # ========================================\n\n        s = x_new - x\n        y = g_new - g\n\n        # Update L-BFGS (skip if curvature condition fails)\n        hess.update(s, y)\n\n        # ========================================\n        #   Check convergence by small progress\n        # ========================================\n\n        # Convergence by insufficient progress (be more lenient than gtol)\n        if (s.norm(p=normp) <= ftol) and ((f_new - f).abs() <= ftol):\n            # Double check with projected gradient\n            g_proj_new = _gradient_projection(x_new, g_new, lb, ub)\n            if g_proj_new.norm(p=normp) <= gtol:\n                warnflag = 0\n                msg = _status_message['success']\n                break\n\n        # Check for precision loss\n        if not f_new.isfinite():\n            warnflag = 2\n            msg = _status_message['pr_loss']\n            break\n\n        # Update state\n        f = f_new\n        x = x_new\n        g = g_new\n\n    else:\n        # Maximum iterations reached\n        warnflag = 1\n        msg = _status_message['maxiter']\n\n    if disp:\n        print(msg)\n        print(\"         Current function value: %f\" % f)\n        print(\"         Iterations: %d\" % n_iter)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n\n    result = OptimizeResult(\n        fun=f,\n        x=x.view_as(x0),\n        grad=g.view_as(x0),\n        status=warnflag,\n        success=(warnflag == 0),\n        message=msg,\n        nit=n_iter,\n        nfev=sf.nfev\n    )\n\n    if return_all:\n        result['allvecs'] = [v.view_as(x0) for v in allvecs]\n\n    return result\n"
  },
  {
    "path": "torchmin/constrained/trust_constr.py",
    "content": "import warnings\nimport numbers\nimport torch\nimport numpy as np\nfrom scipy.optimize import minimize, Bounds, NonlinearConstraint\nfrom scipy.sparse.linalg import LinearOperator\n\n_constr_keys = {'fun', 'lb', 'ub', 'jac', 'hess', 'hessp', 'keep_feasible'}\n_bounds_keys = {'lb', 'ub', 'keep_feasible'}\n\n\ndef _build_obj(f, x0):\n    numel = x0.numel()\n\n    def to_tensor(x):\n        return torch.tensor(x, dtype=x0.dtype, device=x0.device).view_as(x0)\n\n    def f_with_jac(x):\n        x = to_tensor(x).requires_grad_(True)\n        with torch.enable_grad():\n            fval = f(x)\n        grad, = torch.autograd.grad(fval, x)\n        return fval.detach().cpu().numpy(), grad.view(-1).cpu().numpy()\n\n    def f_hess(x):\n        x = to_tensor(x).requires_grad_(True)\n        with torch.enable_grad():\n            fval = f(x)\n            grad, = torch.autograd.grad(fval, x, create_graph=True)\n        def matvec(p):\n            p = to_tensor(p)\n            hvp, = torch.autograd.grad(grad, x, p, retain_graph=True)\n            return hvp.view(-1).cpu().numpy()\n        return LinearOperator((numel, numel), matvec=matvec)\n\n    return f_with_jac, f_hess\n\n\ndef _build_constr(constr, x0):\n    assert isinstance(constr, dict)\n    assert set(constr.keys()).issubset(_constr_keys)\n    assert 'fun' in constr\n    assert 'lb' in constr or 'ub' in constr\n    if 'lb' not in constr:\n        constr['lb'] = -np.inf\n    if 'ub' not in constr:\n        constr['ub'] = np.inf\n    f_ = constr['fun']\n    numel = x0.numel()\n\n    def to_tensor(x):\n        return torch.tensor(x, dtype=x0.dtype, device=x0.device).view_as(x0)\n\n    def f(x):\n        x = to_tensor(x)\n        return f_(x).cpu().numpy()\n\n    def f_jac(x):\n        x = to_tensor(x)\n        if 'jac' in constr:\n            grad = constr['jac'](x)\n        else:\n            x.requires_grad_(True)\n            with torch.enable_grad():\n                grad, = torch.autograd.grad(f_(x), x)\n        return grad.view(-1).cpu().numpy()\n\n    def f_hess(x, v):\n        x = to_tensor(x)\n        if 'hess' in constr:\n            hess = constr['hess'](x)\n            return v[0] * hess.view(numel, numel).cpu().numpy()\n        elif 'hessp' in constr:\n            def matvec(p):\n                p = to_tensor(p)\n                hvp = constr['hessp'](x, p)\n                return v[0] * hvp.view(-1).cpu().numpy()\n            return LinearOperator((numel, numel), matvec=matvec)\n        else:\n            x.requires_grad_(True)\n            with torch.enable_grad():\n                if 'jac' in constr:\n                    grad = constr['jac'](x)\n                else:\n                    grad, = torch.autograd.grad(f_(x), x, create_graph=True)\n            def matvec(p):\n                p = to_tensor(p)\n                if grad.grad_fn is None:\n                    # If grad_fn is None, then grad is constant wrt x, and hess is 0.\n                    hvp = torch.zeros_like(grad)\n                else:\n                    hvp, = torch.autograd.grad(grad, x, p, retain_graph=True)\n                return v[0] * hvp.view(-1).cpu().numpy()\n            return LinearOperator((numel, numel), matvec=matvec)\n\n    return NonlinearConstraint(\n        fun=f, lb=constr['lb'], ub=constr['ub'],\n        jac=f_jac, hess=f_hess,\n        keep_feasible=constr.get('keep_feasible', False))\n\n\ndef _check_bound(val, x0):\n    if isinstance(val, numbers.Number):\n        return np.full(x0.numel(), val)\n    elif isinstance(val, torch.Tensor):\n        assert val.numel() == x0.numel()\n        return val.detach().cpu().numpy().flatten()\n    elif isinstance(val, np.ndarray):\n        assert val.size == x0.numel()\n        return val.flatten()\n    else:\n        raise ValueError('Bound value has unrecognized format.')\n\n\ndef _build_bounds(bounds, x0):\n    assert isinstance(bounds, dict)\n    assert set(bounds.keys()).issubset(_bounds_keys)\n    assert 'lb' in bounds or 'ub' in bounds\n    lb = _check_bound(bounds.get('lb', -np.inf), x0)\n    ub = _check_bound(bounds.get('ub', np.inf), x0)\n    keep_feasible = bounds.get('keep_feasible', False)\n\n    return Bounds(lb, ub, keep_feasible)\n\n\n@torch.no_grad()\ndef _minimize_trust_constr(\n        f, x0, constr=None, bounds=None, max_iter=None, tol=None, callback=None,\n        disp=0, **kwargs):\n    \"\"\"Minimize a scalar function of one or more variables subject to\n    bounds and/or constraints.\n\n    .. note::\n        This is a wrapper for SciPy's\n        `'trust-constr' <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-trustconstr.html>`_\n        method. It uses autograd behind the scenes to build Jacobian & Hessian\n        callables before invoking scipy. Inputs and objectives should use\n        PyTorch tensors like other routines. CUDA is supported; however,\n        data will be transferred back-and-forth between GPU/CPU.\n\n    Parameters\n    ----------\n    f : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    constr : dict, optional\n        Constraint specifications. Should be a dictionary with the\n        following fields:\n\n            * fun (callable) - Constraint function\n            * lb (Tensor or float, optional) - Constraint lower bounds\n            * ub (Tensor or float, optional) - Constraint upper bounds\n\n        One of either `lb` or `ub` must be provided. When `lb` == `ub` it is\n        interpreted as an equality constraint.\n    bounds : dict, optional\n        Bounds on variables. Should a dictionary with at least one\n        of the following fields:\n\n            * lb (Tensor or float) - Lower bounds\n            * ub (Tensor or float) - Upper bounds\n\n        Bounds of `-inf`/`inf` are interpreted as no bound. When `lb` == `ub`\n        it is interpreted as an equality constraint.\n    max_iter : int, optional\n        Maximum number of iterations to perform. If unspecified, this will\n        be set to the default of the selected method.\n    tol : float, optional\n        Tolerance for termination. For detailed control, use solver-specific\n        options.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int\n        Level of algorithm's verbosity:\n\n            * 0 : work silently (default).\n            * 1 : display a termination report.\n            * 2 : display progress during iterations.\n            * 3 : display progress during iterations (more complete report).\n    **kwargs\n        Additional keyword arguments passed to SciPy's trust-constr solver.\n        See options `here <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-trustconstr.html>`_.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    \"\"\"\n    if max_iter is None:\n        max_iter = 1000\n    x0 = x0.detach()\n    if x0.is_cuda:\n        warnings.warn('GPU is not recommended for trust-constr. '\n                      'Data will be moved back-and-forth from CPU.')\n\n    # handle callbacks\n    if callback is not None:\n        callback_ = callback\n\n        def callback(x, state):\n            # x = state.x\n            x = x0.new_tensor(x).view_as(x0)\n            return callback_(x)\n\n    # handle bounds\n    if bounds is not None:\n        bounds = _build_bounds(bounds, x0)\n\n    # build objective function (and hessian)\n    f_with_jac, f_hess = _build_obj(f, x0)\n\n    # build constraints\n    if constr is not None:\n        constraints = [_build_constr(constr, x0)]\n    else:\n        constraints = []\n\n    # optimize\n    x0_np = x0.cpu().numpy().flatten().copy()\n    result = minimize(\n        f_with_jac, x0_np, method='trust-constr', jac=True,\n        hess=f_hess, callback=callback, tol=tol,\n        bounds=bounds,\n        constraints=constraints,\n        options=dict(verbose=int(disp), maxiter=max_iter, **kwargs)\n    )\n\n    # convert the important things to torch tensors\n    for key in ['fun', 'grad', 'x']:\n        result[key] = torch.tensor(result[key], dtype=x0.dtype, device=x0.device)\n    result['x'] = result['x'].view_as(x0)\n\n    return result\n\n"
  },
  {
    "path": "torchmin/function.py",
    "content": "from typing import List, Optional\nfrom torch import Tensor\nfrom collections import namedtuple\nimport torch\nimport torch.autograd as autograd\nfrom torch._vmap_internals import _vmap\n\nfrom .optim.minimizer import Minimizer\n\n__all__ = ['ScalarFunction', 'VectorFunction']\n\n\n\n# scalar function result (value)\nsf_value = namedtuple('sf_value', ['f', 'grad', 'hessp', 'hess'])\n\n# directional evaluate result\nde_value = namedtuple('de_value', ['f', 'grad'])\n\n# vector function result (value)\nvf_value = namedtuple('vf_value', ['f', 'jacp', 'jac'])\n\n\n@torch.jit.script\nclass JacobianLinearOperator(object):\n    def __init__(self,\n                 x: Tensor,\n                 f: Tensor,\n                 gf: Optional[Tensor] = None,\n                 gx: Optional[Tensor] = None,\n                 symmetric: bool = False) -> None:\n        self.x = x\n        self.f = f\n        self.gf = gf\n        self.gx = gx\n        self.symmetric = symmetric\n        # tensor-like properties\n        self.shape = (f.numel(), x.numel())\n        self.dtype = x.dtype\n        self.device = x.device\n\n    def mv(self, v: Tensor) -> Tensor:\n        if self.symmetric:\n            return self.rmv(v)\n        assert v.shape == self.x.shape\n        gx, gf = self.gx, self.gf\n        assert (gx is not None) and (gf is not None)\n        outputs: List[Tensor] = [gx]\n        inputs: List[Tensor] = [gf]\n        grad_outputs: List[Optional[Tensor]] = [v]\n        jvp = autograd.grad(outputs, inputs, grad_outputs, retain_graph=True)[0]\n        if jvp is None:\n            raise Exception\n        return jvp\n\n    def rmv(self, v: Tensor) -> Tensor:\n        assert v.shape == self.f.shape\n        outputs: List[Tensor] = [self.f]\n        inputs: List[Tensor] = [self.x]\n        grad_outputs: List[Optional[Tensor]] = [v]\n        vjp = autograd.grad(outputs, inputs, grad_outputs, retain_graph=True)[0]\n        if vjp is None:\n            raise Exception\n        return vjp\n\n\ndef jacobian_linear_operator(x, f, symmetric=False):\n    if symmetric:\n        # Use vector-jacobian product (more efficient)\n        gf = gx = None\n    else:\n        # Apply the \"double backwards\" trick to get true\n        # jacobian-vector product\n        with torch.enable_grad():\n            gf = torch.zeros_like(f, requires_grad=True)\n            gx = autograd.grad(f, x, gf, create_graph=True)[0]\n    return JacobianLinearOperator(x, f, gf, gx, symmetric)\n\n\n\nclass ScalarFunction(object):\n    \"\"\"Scalar-valued objective function with autograd backend.\n\n    This class provides a general-purpose objective wrapper which will\n    compute first- and second-order derivatives via autograd as specified\n    by the parameters of __init__.\n    \"\"\"\n    def __new__(cls, fun, x_shape, hessp=False, hess=False, twice_diffable=True):\n        if isinstance(fun, Minimizer):\n            assert fun._hessp == hessp\n            assert fun._hess == hess\n            return fun\n        return super(ScalarFunction, cls).__new__(cls)\n\n    def __init__(self, fun, x_shape, hessp=False, hess=False, twice_diffable=True):\n        self._fun = fun\n        self._x_shape = x_shape\n        self._hessp = hessp\n        self._hess = hess\n        self._I = None\n        self._twice_diffable = twice_diffable\n        self.nfev = 0\n\n    def fun(self, x):\n        if x.shape != self._x_shape:\n            x = x.view(self._x_shape)\n        f = self._fun(x)\n        if f.numel() != 1:\n            raise RuntimeError('ScalarFunction was supplied a function '\n                               'that does not return scalar outputs.')\n        self.nfev += 1\n\n        return f\n\n    def closure(self, x):\n        \"\"\"Evaluate the function, gradient, and hessian/hessian-product\n\n        This method represents the core function call. It is used for\n        computing newton/quasi newton directions, etc.\n        \"\"\"\n        x = x.detach().requires_grad_(True)\n        with torch.enable_grad():\n            f = self.fun(x)\n            grad = autograd.grad(f, x, create_graph=self._hessp or self._hess)[0]\n        if (self._hessp or self._hess) and grad.grad_fn is None:\n            raise RuntimeError('A 2nd-order derivative was requested but '\n                               'the objective is not twice-differentiable.')\n        hessp = None\n        hess = None\n        if self._hessp:\n            hessp = jacobian_linear_operator(x, grad, symmetric=self._twice_diffable)\n        if self._hess:\n            if self._I is None:\n                self._I = torch.eye(x.numel(), dtype=x.dtype, device=x.device)\n            hvp = lambda v: autograd.grad(grad, x, v, retain_graph=True)[0]\n            hess = _vmap(hvp)(self._I)\n\n        return sf_value(f=f.detach(), grad=grad.detach(), hessp=hessp, hess=hess)\n\n    def dir_evaluate(self, x, t, d):\n        \"\"\"Evaluate a direction and step size.\n\n        We define a separate \"directional evaluate\" function to be used\n        for strong-wolfe line search. Only the function value and gradient\n        are needed for this use case, so we avoid computational overhead.\n        \"\"\"\n        x = x + d.mul(t)\n        x = x.detach().requires_grad_(True)\n        with torch.enable_grad():\n            f = self.fun(x)\n        grad = autograd.grad(f, x)[0]\n\n        return de_value(f=float(f), grad=grad)\n\n\nclass VectorFunction(object):\n    \"\"\"Vector-valued objective function with autograd backend.\"\"\"\n    def __init__(self, fun, x_shape, jacp=False, jac=False):\n        self._fun = fun\n        self._x_shape = x_shape\n        self._jacp = jacp\n        self._jac = jac\n        self._I = None\n        self.nfev = 0\n\n    def fun(self, x):\n        if x.shape != self._x_shape:\n            x = x.view(self._x_shape)\n        f = self._fun(x)\n        if f.dim() == 0:\n            raise RuntimeError('VectorFunction expected vector outputs but '\n                               'received a scalar.')\n        elif f.dim() > 1:\n            f = f.view(-1)\n        self.nfev += 1\n\n        return f\n\n    def closure(self, x):\n        x = x.detach().requires_grad_(True)\n        with torch.enable_grad():\n            f = self.fun(x)\n        jacp = None\n        jac = None\n        if self._jacp:\n            jacp = jacobian_linear_operator(x, f)\n        if self._jac:\n            if self._I is None:\n                self._I = torch.eye(f.numel(), dtype=x.dtype, device=x.device)\n            vjp = lambda v: autograd.grad(f, x, v, retain_graph=True)[0]\n            jac = _vmap(vjp)(self._I)\n\n        return vf_value(f=f.detach(), jacp=jacp, jac=jac)"
  },
  {
    "path": "torchmin/line_search.py",
    "content": "import warnings\nimport torch\nfrom torch.optim.lbfgs import _strong_wolfe, _cubic_interpolate\nfrom scipy.optimize import minimize_scalar\n\n__all__ = ['strong_wolfe', 'brent', 'backtracking']\n\n\ndef _strong_wolfe_extra(\n        obj_func, x, t, d, f, g, gtd, c1=1e-4, c2=0.9,\n        tolerance_change=1e-9, max_ls=25, extra_condition=None):\n    \"\"\"A modified variant of pytorch's strong-wolfe line search that supports\n    an \"extra_condition\" argument (callable).\n\n    This is required for methods such as Conjugate Gradient (polak-ribiere)\n    where the strong wolfe conditions do not guarantee that we have a\n    descent direction.\n\n    Code borrowed from pytorch::\n        Copyright (c) 2016 Facebook, Inc.\n        All rights reserved.\n    \"\"\"\n    # ported from https://github.com/torch/optim/blob/master/lswolfe.lua\n    if extra_condition is None:\n        extra_condition = lambda *args: True\n    d_norm = d.abs().max()\n    g = g.clone(memory_format=torch.contiguous_format)\n    # evaluate objective and gradient using initial step\n    f_new, g_new = obj_func(x, t, d)\n    ls_func_evals = 1\n    gtd_new = g_new.dot(d)\n\n    # bracket an interval containing a point satisfying the Wolfe criteria\n    t_prev, f_prev, g_prev, gtd_prev = 0, f, g, gtd\n    done = False\n    ls_iter = 0\n    while ls_iter < max_ls:\n        # check conditions\n        if f_new > (f + c1 * t * gtd) or (ls_iter > 1 and f_new >= f_prev):\n            bracket = [t_prev, t]\n            bracket_f = [f_prev, f_new]\n            bracket_g = [g_prev, g_new.clone(memory_format=torch.contiguous_format)]\n            bracket_gtd = [gtd_prev, gtd_new]\n            break\n\n        if abs(gtd_new) <= -c2 * gtd and extra_condition(t, f_new, g_new):\n            bracket = [t]\n            bracket_f = [f_new]\n            bracket_g = [g_new]\n            done = True\n            break\n\n        if gtd_new >= 0:\n            bracket = [t_prev, t]\n            bracket_f = [f_prev, f_new]\n            bracket_g = [g_prev, g_new.clone(memory_format=torch.contiguous_format)]\n            bracket_gtd = [gtd_prev, gtd_new]\n            break\n\n        # interpolate\n        min_step = t + 0.01 * (t - t_prev)\n        max_step = t * 10\n        tmp = t\n        t = _cubic_interpolate(\n            t_prev,\n            f_prev,\n            gtd_prev,\n            t,\n            f_new,\n            gtd_new,\n            bounds=(min_step, max_step))\n\n        # next step\n        t_prev = tmp\n        f_prev = f_new\n        g_prev = g_new.clone(memory_format=torch.contiguous_format)\n        gtd_prev = gtd_new\n        f_new, g_new = obj_func(x, t, d)\n        ls_func_evals += 1\n        gtd_new = g_new.dot(d)\n        ls_iter += 1\n\n    # reached max number of iterations?\n    if ls_iter == max_ls:\n        bracket = [0, t]\n        bracket_f = [f, f_new]\n        bracket_g = [g, g_new]\n\n    # zoom phase: we now have a point satisfying the criteria, or\n    # a bracket around it. We refine the bracket until we find the\n    # exact point satisfying the criteria\n    insuf_progress = False\n    # find high and low points in bracket\n    low_pos, high_pos = (0, 1) if bracket_f[0] <= bracket_f[-1] else (1, 0)\n    while not done and ls_iter < max_ls:\n        # line-search bracket is so small\n        if abs(bracket[1] - bracket[0]) * d_norm < tolerance_change:\n            break\n\n        # compute new trial value\n        t = _cubic_interpolate(bracket[0], bracket_f[0], bracket_gtd[0],\n                               bracket[1], bracket_f[1], bracket_gtd[1])\n\n        # test that we are making sufficient progress:\n        # in case `t` is so close to boundary, we mark that we are making\n        # insufficient progress, and if\n        #   + we have made insufficient progress in the last step, or\n        #   + `t` is at one of the boundary,\n        # we will move `t` to a position which is `0.1 * len(bracket)`\n        # away from the nearest boundary point.\n        eps = 0.1 * (max(bracket) - min(bracket))\n        if min(max(bracket) - t, t - min(bracket)) < eps:\n            # interpolation close to boundary\n            if insuf_progress or t >= max(bracket) or t <= min(bracket):\n                # evaluate at 0.1 away from boundary\n                if abs(t - max(bracket)) < abs(t - min(bracket)):\n                    t = max(bracket) - eps\n                else:\n                    t = min(bracket) + eps\n                insuf_progress = False\n            else:\n                insuf_progress = True\n        else:\n            insuf_progress = False\n\n        # Evaluate new point\n        f_new, g_new = obj_func(x, t, d)\n        ls_func_evals += 1\n        gtd_new = g_new.dot(d)\n        ls_iter += 1\n\n        if f_new > (f + c1 * t * gtd) or f_new >= bracket_f[low_pos]:\n            # Armijo condition not satisfied or not lower than lowest point\n            bracket[high_pos] = t\n            bracket_f[high_pos] = f_new\n            bracket_g[high_pos] = g_new.clone(memory_format=torch.contiguous_format)\n            bracket_gtd[high_pos] = gtd_new\n            low_pos, high_pos = (0, 1) if bracket_f[0] <= bracket_f[1] else (1, 0)\n        else:\n            if abs(gtd_new) <= -c2 * gtd and extra_condition(t, f_new, g_new):\n                # Wolfe conditions satisfied\n                done = True\n            elif gtd_new * (bracket[high_pos] - bracket[low_pos]) >= 0:\n                # old high becomes new low\n                bracket[high_pos] = bracket[low_pos]\n                bracket_f[high_pos] = bracket_f[low_pos]\n                bracket_g[high_pos] = bracket_g[low_pos]\n                bracket_gtd[high_pos] = bracket_gtd[low_pos]\n\n            # new point becomes new low\n            bracket[low_pos] = t\n            bracket_f[low_pos] = f_new\n            bracket_g[low_pos] = g_new.clone(memory_format=torch.contiguous_format)\n            bracket_gtd[low_pos] = gtd_new\n\n    # return stuff\n    t = bracket[low_pos]\n    f_new = bracket_f[low_pos]\n    g_new = bracket_g[low_pos]\n    return f_new, g_new, t, ls_func_evals\n\n\ndef strong_wolfe(fun, x, t, d, f, g, gtd=None, **kwargs):\n    \"\"\"\n    Expects `fun` to take arguments {x, t, d} and return {f(x1), f'(x1)},\n    where x1 is the new location after taking a step from x in direction d\n    with step size t.\n    \"\"\"\n    if gtd is None:\n        gtd = g.mul(d).sum()\n\n    # use python floats for scalars as per torch.optim.lbfgs\n    f, t = float(f), float(t)\n\n    if 'extra_condition' in kwargs:\n        f, g, t, ls_nevals = _strong_wolfe_extra(\n            fun, x.view(-1), t, d.view(-1), f, g.view(-1), gtd, **kwargs)\n    else:\n        # in theory we shouldn't need to use pytorch's native _strong_wolfe\n        # since the custom implementation above is equivalent with\n        # extra_codition=None. But we will keep this in case they make any\n        # changes.\n        f, g, t, ls_nevals = _strong_wolfe(\n            fun, x.view(-1), t, d.view(-1), f, g.view(-1), gtd, **kwargs)\n\n    # convert back to torch scalar\n    f = torch.as_tensor(f, dtype=x.dtype, device=x.device)\n\n    return f, g.view_as(x), t, ls_nevals\n\n\ndef brent(fun, x, d, bounds=(0,10)):\n    \"\"\"\n    Expects `fun` to take arguments {x} and return {f(x)}\n    \"\"\"\n    def line_obj(t):\n        return float(fun(x + t * d))\n    res = minimize_scalar(line_obj, bounds=bounds, method='bounded')\n    return res.x\n\n\ndef backtracking(fun, x, t, d, f, g, mu=0.1, decay=0.98, max_ls=500, tmin=1e-5):\n    \"\"\"\n    Expects `fun` to take arguments {x, t, d} and return {f(x1), x1},\n    where x1 is the new location after taking a step from x in direction d\n    with step size t.\n\n    We use a generalized variant of the armijo condition that supports\n    arbitrary step functions x' = step(x,t,d). When step(x,t,d) = x + t * d\n    it is equivalent to the standard condition.\n    \"\"\"\n    x_new = x\n    f_new = f\n    success = False\n    for i in range(max_ls):\n        f_new, x_new = fun(x, t, d)\n        if f_new <= f + mu * g.mul(x_new-x).sum():\n            success = True\n            break\n        if t <= tmin:\n            warnings.warn('step size has reached the minimum threshold.')\n            break\n        t = t.mul(decay)\n    else:\n        warnings.warn('backtracking did not converge.')\n\n    return x_new, f_new, t, success"
  },
  {
    "path": "torchmin/lstsq/__init__.py",
    "content": "\"\"\"\nThis module represents a pytorch re-implementation of scipy's\n`scipy.optimize._lsq` module. Some of the code is borrowed directly\nfrom the scipy library (all rights reserved).\n\"\"\"\nfrom .least_squares import least_squares"
  },
  {
    "path": "torchmin/lstsq/cg.py",
    "content": "import torch\n\nfrom .linear_operator import aslinearoperator, TorchLinearOperator\n\n\ndef cg(A, b, x0=None, max_iter=None, tol=1e-5):\n    if max_iter is None:\n        max_iter = 20 * b.numel()\n    if x0 is None:\n        x = torch.zeros_like(b)\n        r = b.clone()\n    else:\n        x = x0.clone()\n        r = b - A.mv(x)\n    p = r.clone()\n    rs = r.dot(r)\n    rs_new = b.new_tensor(0.)\n    alpha = b.new_tensor(0.)\n    for n_iter in range(1, max_iter+1):\n        Ap = A.mv(p)\n        torch.div(rs, p.dot(Ap), out=alpha)\n        x.add_(p, alpha=alpha)\n        r.sub_(Ap, alpha=alpha)\n        torch.dot(r, r, out=rs_new)\n        p.mul_(rs_new / rs).add_(r)\n        if n_iter % 10 == 0:\n            r_norm = rs.sqrt()\n            if r_norm < tol:\n                break\n        rs.copy_(rs_new, non_blocking=True)\n\n    return x\n\n\ndef cgls(A, b, alpha=0., **kwargs):\n    A = aslinearoperator(A)\n    m, n = A.shape\n    Atb = A.rmv(b)\n    AtA = TorchLinearOperator(shape=(n,n),\n                              matvec=lambda x: A.rmv(A.mv(x)) + alpha * x,\n                              rmatvec=None)\n    return cg(AtA, Atb, **kwargs)"
  },
  {
    "path": "torchmin/lstsq/common.py",
    "content": "import numpy as np\nimport torch\nfrom scipy.sparse.linalg import LinearOperator\n\nfrom .linear_operator import TorchLinearOperator\n\nEPS = torch.finfo(float).eps\n\n\ndef in_bounds(x, lb, ub):\n    \"\"\"Check if a point lies within bounds.\"\"\"\n    return torch.all((x >= lb) & (x <= ub))\n\n\ndef find_active_constraints(x, lb, ub, rtol=1e-10):\n    \"\"\"Determine which constraints are active in a given point.\n    The threshold is computed using `rtol` and the absolute value of the\n    closest bound.\n    Returns\n    -------\n    active : ndarray of int with shape of x\n        Each component shows whether the corresponding constraint is active:\n             *  0 - a constraint is not active.\n             * -1 - a lower bound is active.\n             *  1 - a upper bound is active.\n    \"\"\"\n    active = torch.zeros_like(x, dtype=torch.long)\n\n    if rtol == 0:\n        active[x <= lb] = -1\n        active[x >= ub] = 1\n        return active\n\n    lower_dist = x - lb\n    upper_dist = ub - x\n    lower_threshold = rtol * lb.abs().clamp(1, None)\n    upper_threshold = rtol * ub.abs().clamp(1, None)\n\n    lower_active = (lb.isfinite() &\n                    (lower_dist <= torch.minimum(upper_dist, lower_threshold)))\n    active[lower_active] = -1\n\n    upper_active = (ub.isfinite() &\n                    (upper_dist <= torch.minimum(lower_dist, upper_threshold)))\n    active[upper_active] = 1\n\n    return active\n\n\ndef make_strictly_feasible(x, lb, ub, rstep=1e-10):\n    \"\"\"Shift a point to the interior of a feasible region.\n    Each element of the returned vector is at least at a relative distance\n    `rstep` from the closest bound. If ``rstep=0`` then `np.nextafter` is used.\n    \"\"\"\n    x_new = x.clone()\n\n    active = find_active_constraints(x, lb, ub, rstep)\n    lower_mask = torch.eq(active, -1)\n    upper_mask = torch.eq(active, 1)\n\n    if rstep == 0:\n        torch.nextafter(lb[lower_mask], ub[lower_mask], out=x_new[lower_mask])\n        torch.nextafter(ub[upper_mask], lb[upper_mask], out=x_new[upper_mask])\n    else:\n        x_new[lower_mask] = lb[lower_mask].add(lb[lower_mask].abs().clamp(1,None), alpha=rstep)\n        x_new[upper_mask] = ub[upper_mask].sub(ub[upper_mask].abs().clamp(1,None), alpha=rstep)\n\n    tight_bounds = (x_new < lb) | (x_new > ub)\n    x_new[tight_bounds] = 0.5 * (lb[tight_bounds] + ub[tight_bounds])\n\n    return x_new\n\n\ndef solve_lsq_trust_region(n, m, uf, s, V, Delta, initial_alpha=None,\n                           rtol=0.01, max_iter=10):\n    \"\"\"Solve a trust-region problem arising in least-squares minimization.\n    This function implements a method described by J. J. More [1]_ and used\n    in MINPACK, but it relies on a single SVD of Jacobian instead of series\n    of Cholesky decompositions. Before running this function, compute:\n    ``U, s, VT = svd(J, full_matrices=False)``.\n    \"\"\"\n    def phi_and_derivative(alpha, suf, s, Delta):\n        \"\"\"Function of which to find zero.\n        It is defined as \"norm of regularized (by alpha) least-squares\n        solution minus `Delta`\". Refer to [1]_.\n        \"\"\"\n        denom = s.pow(2) + alpha\n        p_norm = (suf / denom).norm()\n        phi = p_norm - Delta\n        phi_prime = -(suf.pow(2) / denom.pow(3)).sum() / p_norm\n        return phi, phi_prime\n\n    def set_alpha(alpha_lower, alpha_upper):\n        new_alpha = (alpha_lower * alpha_upper).sqrt()\n        return new_alpha.clamp_(0.001 * alpha_upper, None)\n\n    suf = s * uf\n\n    # Check if J has full rank and try Gauss-Newton step.\n    eps = torch.finfo(s.dtype).eps\n    full_rank = m >= n and s[-1] > eps * m * s[0]\n\n    if full_rank:\n        p = -V.mv(uf / s)\n        if p.norm() <= Delta:\n            return p, 0.0, 0\n        phi, phi_prime = phi_and_derivative(0., suf, s, Delta)\n        alpha_lower = -phi / phi_prime\n    else:\n        alpha_lower = s.new_tensor(0.)\n\n    alpha_upper = suf.norm() / Delta\n\n    if initial_alpha is None or not full_rank and initial_alpha == 0:\n        alpha = set_alpha(alpha_lower, alpha_upper)\n    else:\n        alpha = initial_alpha.clone()\n\n    for it in range(max_iter):\n        # if alpha is outside of bounds, set new value (5.5)(a)\n        alpha = torch.where((alpha < alpha_lower) | (alpha > alpha_upper),\n                            set_alpha(alpha_lower, alpha_upper),\n                            alpha)\n\n        # compute new phi and phi' (5.5)(b)\n        phi, phi_prime = phi_and_derivative(alpha, suf, s, Delta)\n\n        # if phi is negative, update our upper bound  (5.5)(b)\n        alpha_upper = torch.where(phi < 0, alpha, alpha_upper)\n\n        # update lower bound  (5.5)(b)\n        ratio = phi / phi_prime\n        alpha_lower.clamp_(alpha-ratio, None)\n\n        # compute new alpha (5.5)(c)\n        alpha.addcdiv_((phi + Delta) * ratio, Delta, value=-1)\n\n        if phi.abs() < rtol * Delta:\n            break\n\n    p = -V.mv(suf / (s.pow(2) + alpha))\n\n    # Make the norm of p equal to Delta, p is changed only slightly during\n    # this. It is done to prevent p lie outside the trust region (which can\n    # cause problems later).\n    p.mul_(Delta / p.norm())\n\n    return p, alpha, it + 1\n\n\ndef right_multiplied_operator(J, d):\n    \"\"\"Return J diag(d) as LinearOperator.\"\"\"\n    if isinstance(J, LinearOperator):\n        if torch.is_tensor(d):\n            d = d.data.cpu().numpy()\n        return LinearOperator(J.shape,\n                              matvec=lambda x: J.matvec(np.ravel(x) * d),\n                              matmat=lambda X: J.matmat(X * d[:, np.newaxis]),\n                              rmatvec=lambda x: d * J.rmatvec(x))\n    elif isinstance(J, TorchLinearOperator):\n        return TorchLinearOperator(J.shape,\n                                   matvec=lambda x: J.matvec(x.view(-1) * d),\n                                   rmatvec=lambda x: d * J.rmatvec(x))\n    else:\n        raise ValueError('Expected J to be a LinearOperator or '\n                         'TorchLinearOperator but found {}'.format(type(J)))\n\n\ndef build_quadratic_1d(J, g, s, diag=None, s0=None):\n    \"\"\"Parameterize a multivariate quadratic function along a line.\n\n    The resulting univariate quadratic function is given as follows:\n    ::\n        f(t) = 0.5 * (s0 + s*t).T * (J.T*J + diag) * (s0 + s*t) +\n               g.T * (s0 + s*t)\n    \"\"\"\n    v = J.mv(s)\n    a = v.dot(v)\n    if diag is not None:\n        a += s.dot(s * diag)\n    a *= 0.5\n\n    b = g.dot(s)\n\n    if s0 is not None:\n        u = J.mv(s0)\n        b += u.dot(v)\n        c = 0.5 * u.dot(u) + g.dot(s0)\n        if diag is not None:\n            b += s.dot(s0 * diag)\n            c += 0.5 * s0.dot(s0 * diag)\n        return a, b, c\n    else:\n        return a, b\n\n\ndef minimize_quadratic_1d(a, b, lb, ub, c=0):\n    \"\"\"Minimize a 1-D quadratic function subject to bounds.\n\n    The free term `c` is 0 by default. Bounds must be finite.\n    \"\"\"\n    t = [lb, ub]\n    if a != 0:\n        extremum = -0.5 * b / a\n        if lb < extremum < ub:\n            t.append(extremum)\n    t = a.new_tensor(t)\n    y = t * (a * t + b) + c\n    min_index = torch.argmin(y)\n    return t[min_index], y[min_index]\n\n\ndef evaluate_quadratic(J, g, s, diag=None):\n    \"\"\"Compute values of a quadratic function arising in least squares.\n    The function is 0.5 * s.T * (J.T * J + diag) * s + g.T * s.\n    \"\"\"\n    if s.dim() == 1:\n        Js = J.mv(s)\n        q = Js.dot(Js)\n        if diag is not None:\n            q += s.dot(s * diag)\n    else:\n        Js = J.matmul(s.T)\n        q = Js.square().sum(0)\n        if diag is not None:\n            q += (diag * s.square()).sum(1)\n\n    l = s.matmul(g)\n\n    return 0.5 * q + l\n\ndef solve_trust_region_2d(B, g, Delta):\n    \"\"\"Solve a general trust-region problem in 2 dimensions.\n    The problem is reformulated as a 4th order algebraic equation,\n    the solution of which is found by numpy.roots.\n    \"\"\"\n    try:\n        L = torch.linalg.cholesky(B)\n        p = - torch.cholesky_solve(g.unsqueeze(1), L).squeeze(1)\n        if p.dot(p) <= Delta**2:\n            return p, True\n    except RuntimeError as exc:\n        if not 'cholesky' in exc.args[0]:\n            raise\n\n    # move things to numpy\n    device = B.device\n    dtype = B.dtype\n    B = B.data.cpu().numpy()\n    g = g.data.cpu().numpy()\n    Delta = float(Delta)\n\n    a = B[0, 0] * Delta**2\n    b = B[0, 1] * Delta**2\n    c = B[1, 1] * Delta**2\n    d = g[0] * Delta\n    f = g[1] * Delta\n\n    coeffs = np.array([-b + d, 2 * (a - c + f), 6 * b, 2 * (-a + c + f), -b - d])\n    t = np.roots(coeffs)  # Can handle leading zeros.\n    t = np.real(t[np.isreal(t)])\n\n    p = Delta * np.vstack((2 * t / (1 + t**2), (1 - t**2) / (1 + t**2)))\n    value = 0.5 * np.sum(p * B.dot(p), axis=0) + np.dot(g, p)\n    p = p[:, np.argmin(value)]\n\n    # convert back to torch\n    p = torch.tensor(p, device=device, dtype=dtype)\n\n    return p, False\n\n\ndef update_tr_radius(Delta, actual_reduction, predicted_reduction,\n                     step_norm, bound_hit):\n    \"\"\"Update the radius of a trust region based on the cost reduction.\n    \"\"\"\n    if predicted_reduction > 0:\n        ratio = actual_reduction / predicted_reduction\n    elif predicted_reduction == actual_reduction == 0:\n        ratio = 1\n    else:\n        ratio = 0\n\n    if ratio < 0.25:\n        Delta = 0.25 * step_norm\n    elif ratio > 0.75 and bound_hit:\n        Delta *= 2.0\n\n    return Delta, ratio\n\n\ndef check_termination(dF, F, dx_norm, x_norm, ratio, ftol, xtol):\n    \"\"\"Check termination condition for nonlinear least squares.\"\"\"\n    ftol_satisfied = dF < ftol * F and ratio > 0.25\n    xtol_satisfied = dx_norm < xtol * (xtol + x_norm)\n\n    if ftol_satisfied and xtol_satisfied:\n        return 4\n    elif ftol_satisfied:\n        return 2\n    elif xtol_satisfied:\n        return 3\n    else:\n        return None"
  },
  {
    "path": "torchmin/lstsq/least_squares.py",
    "content": "\"\"\"\nGeneric interface for nonlinear least-squares minimization.\n\"\"\"\nfrom warnings import warn\nimport numbers\nimport torch\n\nfrom .trf import trf\nfrom .common import EPS, in_bounds, make_strictly_feasible\n\n__all__ = ['least_squares']\n\n\nTERMINATION_MESSAGES = {\n    -1: \"Improper input parameters status returned from `leastsq`\",\n    0: \"The maximum number of function evaluations is exceeded.\",\n    1: \"`gtol` termination condition is satisfied.\",\n    2: \"`ftol` termination condition is satisfied.\",\n    3: \"`xtol` termination condition is satisfied.\",\n    4: \"Both `ftol` and `xtol` termination conditions are satisfied.\"\n}\n\n\ndef prepare_bounds(bounds, x0):\n    n = x0.shape[0]\n    def process(b):\n        if isinstance(b, numbers.Number):\n            return x0.new_full((n,), b)\n        elif isinstance(b, torch.Tensor):\n            if b.dim() == 0:\n                return x0.new_full((n,), b)\n            return b\n        else:\n            raise ValueError\n\n    lb, ub = [process(b) for b in bounds]\n\n    return lb, ub\n\n\ndef check_tolerance(ftol, xtol, gtol, method):\n    def check(tol, name):\n        if tol is None:\n            tol = 0\n        elif tol < EPS:\n            warn(\"Setting `{}` below the machine epsilon ({:.2e}) effectively \"\n                 \"disables the corresponding termination condition.\"\n                 .format(name, EPS))\n        return tol\n\n    ftol = check(ftol, \"ftol\")\n    xtol = check(xtol, \"xtol\")\n    gtol = check(gtol, \"gtol\")\n\n    if method == \"lm\" and (ftol < EPS or xtol < EPS or gtol < EPS):\n        raise ValueError(\"All tolerances must be higher than machine epsilon \"\n                         \"({:.2e}) for method 'lm'.\".format(EPS))\n    elif ftol < EPS and xtol < EPS and gtol < EPS:\n        raise ValueError(\"At least one of the tolerances must be higher than \"\n                         \"machine epsilon ({:.2e}).\".format(EPS))\n\n    return ftol, xtol, gtol\n\n\ndef check_x_scale(x_scale, x0):\n    if isinstance(x_scale, str) and x_scale == 'jac':\n        return x_scale\n    try:\n        x_scale = torch.as_tensor(x_scale)\n        valid = x_scale.isfinite().all() and x_scale.gt(0).all()\n    except (ValueError, TypeError):\n        valid = False\n\n    if not valid:\n        raise ValueError(\"`x_scale` must be 'jac' or array_like with \"\n                         \"positive numbers.\")\n\n    if x_scale.dim() == 0:\n        x_scale = x0.new_full(x0.shape, x_scale)\n\n    if x_scale.shape != x0.shape:\n        raise ValueError(\"Inconsistent shapes between `x_scale` and `x0`.\")\n\n    return x_scale\n\n\ndef least_squares(\n        fun, x0, bounds=None, method='trf', ftol=1e-8, xtol=1e-8,\n        gtol=1e-8, x_scale=1.0, tr_solver='lsmr', tr_options=None,\n        max_nfev=None, verbose=0):\n    r\"\"\"Solve a nonlinear least-squares problem with bounds on the variables.\n\n    Given the residual function\n    :math:`f: \\mathcal{R}^n \\rightarrow \\mathcal{R}^m`, `least_squares`\n    finds a local minimum of the residual sum-of-squares (RSS) objective:\n\n    .. math::\n        x^* = \\underset{x}{\\operatorname{arg\\,min\\,}}\n        \\frac{1}{2} ||f(x)||_2^2 \\quad \\text{subject to} \\quad lb \\leq x \\leq ub\n\n    The solution is found using variants of the Gauss-Newton method, a\n    modification of Newton's method tailored to RSS problems.\n\n    Parameters\n    ----------\n    fun : callable\n        Function which computes the vector of residuals, with the signature\n        ``fun(x)``. The argument ``x`` passed to this\n        function is a Tensor of shape (n,) (never a scalar, even for n=1).\n        It must allocate and return a 1-D Tensor of shape (m,) or a scalar.\n    x0 : Tensor or float\n        Initial guess on independent variables, with shape (n,). If\n        float, it will be treated as a 1-D Tensor with one element.\n    bounds : 2-tuple of Tensor, optional\n        Lower and upper bounds on independent variables. Defaults to no bounds.\n        Each Tensor must match the size of `x0` or be a scalar, in the latter\n        case a bound will be the same for all variables. Use ``inf`` with\n        an appropriate sign to disable bounds on all or some variables.\n    method : str, optional\n        Algorithm to perform minimization. Default is 'trf'.\n\n            * 'trf' : Trust Region Reflective algorithm, particularly suitable\n              for large sparse problems with bounds. Generally robust method.\n            * 'dogbox' : COMING SOON. dogleg algorithm with rectangular trust regions,\n              typical use case is small problems with bounds. Not recommended\n              for problems with rank-deficient Jacobian.\n    ftol : float or None, optional\n        Tolerance for termination by the change of the cost function. The\n        optimization process is stopped when ``dF < ftol * F``,\n        and there was an adequate agreement between a local quadratic model and\n        the true model in the last step. If None, the termination by this\n        condition is disabled. Default is 1e-8.\n    xtol : float or None, optional\n        Tolerance for termination by the change of the independent variables.\n        Termination occurs when ``norm(dx) < xtol * (xtol + norm(x))``.\n        If None, the termination by this condition is disabled. Default is 1e-8.\n    gtol : float or None, optional\n        Tolerance for termination by the norm of the gradient. Default is 1e-8.\n        The exact condition depends on `method` used:\n\n            * For 'trf' : ``norm(g_scaled, ord=inf) < gtol``, where\n              ``g_scaled`` is the value of the gradient scaled to account for\n              the presence of the bounds [STIR]_.\n            * For 'dogbox' : ``norm(g_free, ord=inf) < gtol``, where\n              ``g_free`` is the gradient with respect to the variables which\n              are not in the optimal state on the boundary.\n    x_scale : Tensor or 'jac', optional\n        Characteristic scale of each variable. Setting `x_scale` is equivalent\n        to reformulating the problem in scaled variables ``xs = x / x_scale``.\n        An alternative view is that the size of a trust region along jth\n        dimension is proportional to ``x_scale[j]``. Improved convergence may\n        be achieved by setting `x_scale` such that a step of a given size\n        along any of the scaled variables has a similar effect on the cost\n        function. If set to 'jac', the scale is iteratively updated using the\n        inverse norms of the columns of the Jacobian matrix (as described in\n        [JJMore]_).\n    max_nfev : None or int, optional\n        Maximum number of function evaluations before the termination.\n        Defaults to 100 * n.\n    tr_solver : str, optional\n        Method for solving trust-region subproblems.\n\n            * 'exact' is suitable for not very large problems with dense\n              Jacobian matrices. The computational complexity per iteration is\n              comparable to a singular value decomposition of the Jacobian\n              matrix.\n            * 'lsmr' is suitable for problems with sparse and large Jacobian\n              matrices. It uses an iterative procedure for finding a solution\n              of a linear least-squares problem and only requires matrix-vector\n              product evaluations.\n    tr_options : dict, optional\n        Keyword options passed to trust-region solver.\n\n            * ``tr_solver='exact'``: `tr_options` are ignored.\n            * ``tr_solver='lsmr'``: options for `scipy.sparse.linalg.lsmr`.\n              Additionally,  ``method='trf'`` supports  'regularize' option\n              (bool, default is True), which adds a regularization term to the\n              normal equation, which improves convergence if the Jacobian is\n              rank-deficient [Byrd]_ (eq. 3.4).\n    verbose : int, optional\n        Level of algorithm's verbosity.\n\n            * 0 : work silently (default).\n            * 1 : display a termination report.\n            * 2 : display progress during iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    References\n    ----------\n    .. [STIR] M. A. Branch, T. F. Coleman, and Y. Li, \"A Subspace, Interior,\n              and Conjugate Gradient Method for Large-Scale Bound-Constrained\n              Minimization Problems,\" SIAM Journal on Scientific Computing,\n              Vol. 21, Number 1, pp 1-23, 1999.\n    .. [Byrd] R. H. Byrd, R. B. Schnabel and G. A. Shultz, \"Approximate\n              solution of the trust region problem by minimization over\n              two-dimensional subspaces\", Math. Programming, 40, pp. 247-263,\n              1988.\n    .. [JJMore] J. J. More, \"The Levenberg-Marquardt Algorithm: Implementation\n                and Theory,\" Numerical Analysis, ed. G. A. Watson, Lecture\n                Notes in Mathematics 630, Springer Verlag, pp. 105-116, 1977.\n\n    \"\"\"\n    if tr_options is None:\n        tr_options = {}\n\n    if method not in ['trf', 'dogbox']:\n        raise ValueError(\"`method` must be 'trf' or 'dogbox'.\")\n\n    if tr_solver not in ['exact', 'lsmr', 'cgls']:\n        raise ValueError(\"`tr_solver` must be one of {'exact', 'lsmr', 'cgls'}.\")\n\n    if verbose not in [0, 1, 2]:\n        raise ValueError(\"`verbose` must be in [0, 1, 2].\")\n\n    if bounds is None:\n        bounds = (-float('inf'), float('inf'))\n    elif not (isinstance(bounds, (tuple, list)) and len(bounds) == 2):\n        raise ValueError(\"`bounds` must be a tuple/list with 2 elements.\")\n\n    if max_nfev is not None and max_nfev <= 0:\n        raise ValueError(\"`max_nfev` must be None or positive integer.\")\n\n    # initial point\n    x0 = torch.atleast_1d(x0)\n    if torch.is_complex(x0):\n        raise ValueError(\"`x0` must be real.\")\n    elif x0.dim() > 1:\n        raise ValueError(\"`x0` must have at most 1 dimension.\")\n\n    # bounds\n    lb, ub = prepare_bounds(bounds, x0)\n    if lb.shape != x0.shape or ub.shape != x0.shape:\n        raise ValueError(\"Inconsistent shapes between bounds and `x0`.\")\n    elif torch.any(lb >= ub):\n        raise ValueError(\"Each lower bound must be strictly less than each \"\n                         \"upper bound.\")\n    elif not in_bounds(x0, lb, ub):\n        raise ValueError(\"`x0` is infeasible.\")\n\n    # x_scale\n    x_scale = check_x_scale(x_scale, x0)\n\n    # tolerance\n    ftol, xtol, gtol = check_tolerance(ftol, xtol, gtol, method)\n\n    if method == 'trf':\n        x0 = make_strictly_feasible(x0, lb, ub)\n\n    def fun_wrapped(x):\n        return torch.atleast_1d(fun(x))\n\n    # check function\n    f0 = fun_wrapped(x0)\n    if f0.dim() != 1:\n        raise ValueError(\"`fun` must return at most 1-d array_like. \"\n                         \"f0.shape: {0}\".format(f0.shape))\n    elif not f0.isfinite().all():\n        raise ValueError(\"Residuals are not finite in the initial point.\")\n\n    initial_cost = 0.5 * f0.dot(f0)\n\n    if isinstance(x_scale, str) and x_scale == 'jac':\n        raise ValueError(\"x_scale='jac' can't be used when `jac` \"\n                         \"returns LinearOperator.\")\n\n    if method == 'trf':\n        result = trf(fun_wrapped, x0, f0, lb, ub, ftol, xtol, gtol,\n                     max_nfev, x_scale, tr_solver, tr_options.copy(), verbose)\n    elif method == 'dogbox':\n        raise NotImplementedError(\"'dogbox' method not yet implemented\")\n        # if tr_solver == 'lsmr' and 'regularize' in tr_options:\n        #     warn(\"The keyword 'regularize' in `tr_options` is not relevant \"\n        #          \"for 'dogbox' method.\")\n        #     tr_options = tr_options.copy()\n        #     del tr_options['regularize']\n        # result = dogbox(fun_wrapped, x0, f0, lb, ub, ftol, xtol, gtol,\n        #                 max_nfev, x_scale, tr_solver, tr_options, verbose)\n    else:\n        raise ValueError(\"`method` must be 'trf' or 'dogbox'.\")\n\n    result.message = TERMINATION_MESSAGES[result.status]\n    result.success = result.status > 0\n\n    if verbose >= 1:\n        print(result.message)\n        print(\"Function evaluations {0}, initial cost {1:.4e}, final cost \"\n              \"{2:.4e}, first-order optimality {3:.2e}.\"\n              .format(result.nfev, initial_cost, result.cost,\n                      result.optimality))\n\n    return result"
  },
  {
    "path": "torchmin/lstsq/linear_operator.py",
    "content": "import torch\nimport torch.autograd as autograd\nfrom torch._vmap_internals import _vmap\n\n\ndef jacobian_dense(fun, x, vectorize=True):\n    x = x.detach().requires_grad_(True)\n    return autograd.functional.jacobian(fun, x, vectorize=vectorize)\n\n\ndef jacobian_linop(fun, x, return_f=False):\n    x = x.detach().requires_grad_(True)\n    with torch.enable_grad():\n        f = fun(x)\n\n    # vector-jacobian product\n    def vjp(v):\n        v = v.view_as(f)\n        vjp, = autograd.grad(f, x, v, retain_graph=True)\n        return vjp.view(-1)\n\n    # jacobian-vector product\n    gf = torch.zeros_like(f, requires_grad=True)\n    with torch.enable_grad():\n        gx, = autograd.grad(f, x, gf, create_graph=True)\n    def jvp(v):\n        v = v.view_as(x)\n        jvp, = autograd.grad(gx, gf, v, retain_graph=True)\n        return jvp.view(-1)\n\n    jac = TorchLinearOperator((f.numel(), x.numel()), matvec=jvp, rmatvec=vjp)\n\n    if return_f:\n        return jac, f.detach()\n    return jac\n\n\nclass TorchLinearOperator(object):\n    \"\"\"Linear operator defined in terms of user-specified operations.\"\"\"\n    def __init__(self, shape, matvec, rmatvec):\n        self.shape = shape\n        self._matvec = matvec\n        self._rmatvec = rmatvec\n\n    def matvec(self, x):\n        return self._matvec(x)\n\n    def rmatvec(self, x):\n        return self._rmatvec(x)\n\n    def matmat(self, X):\n        try:\n            return _vmap(self.matvec)(X.T).T\n        except:\n            return torch.hstack([self.matvec(col).view(-1,1) for col in X.T])\n\n    def transpose(self):\n        new_shape = (self.shape[1], self.shape[0])\n        return type(self)(new_shape, self._rmatvec, self._matvec)\n\n    mv = matvec\n    rmv = rmatvec\n    matmul = matmat\n    t = transpose\n    T = property(transpose)\n\n\ndef aslinearoperator(A):\n    if isinstance(A, TorchLinearOperator):\n        return A\n    elif isinstance(A, torch.Tensor):\n        assert A.dim() == 2\n        return TorchLinearOperator(A.shape, matvec=A.mv, rmatvec=A.T.mv)\n    else:\n        raise ValueError('Input must be either a Tensor or TorchLinearOperator')"
  },
  {
    "path": "torchmin/lstsq/lsmr.py",
    "content": "\"\"\"\nCode modified from scipy.sparse.linalg.lsmr\n\nCopyright (C) 2010 David Fong and Michael Saunders\n\"\"\"\nimport torch\n\nfrom .linear_operator import aslinearoperator\n\n\ndef _sym_ortho(a, b, out):\n    torch.hypot(a, b, out=out[2])\n    torch.div(a, out[2], out=out[0])\n    torch.div(b, out[2], out=out[1])\n    return out\n\n\n@torch.no_grad()\ndef lsmr(A, b, damp=0., atol=1e-6, btol=1e-6, conlim=1e8, maxiter=None,\n         x0=None, check_nonzero=True):\n    \"\"\"Iterative solver for least-squares problems.\n\n    lsmr solves the system of linear equations ``Ax = b``. If the system\n    is inconsistent, it solves the least-squares problem ``min ||b - Ax||_2``.\n    ``A`` is a rectangular matrix of dimension m-by-n, where all cases are\n    allowed: m = n, m > n, or m < n. ``b`` is a vector of length m.\n    The matrix A may be dense or sparse (usually sparse).\n\n    Parameters\n    ----------\n    A : {matrix, sparse matrix, ndarray, LinearOperator}\n        Matrix A in the linear system.\n        Alternatively, ``A`` can be a linear operator which can\n        produce ``Ax`` and ``A^H x`` using, e.g.,\n        ``scipy.sparse.linalg.LinearOperator``.\n    b : array_like, shape (m,)\n        Vector ``b`` in the linear system.\n    damp : float\n        Damping factor for regularized least-squares. `lsmr` solves\n        the regularized least-squares problem::\n         min ||(b) - (  A   )x||\n             ||(0)   (damp*I) ||_2\n        where damp is a scalar.  If damp is None or 0, the system\n        is solved without regularization.\n    atol, btol : float, optional\n        Stopping tolerances. `lsmr` continues iterations until a\n        certain backward error estimate is smaller than some quantity\n        depending on atol and btol.  Let ``r = b - Ax`` be the\n        residual vector for the current approximate solution ``x``.\n        If ``Ax = b`` seems to be consistent, ``lsmr`` terminates\n        when ``norm(r) <= atol * norm(A) * norm(x) + btol * norm(b)``.\n        Otherwise, lsmr terminates when ``norm(A^H r) <=\n        atol * norm(A) * norm(r)``.  If both tolerances are 1.0e-6 (say),\n        the final ``norm(r)`` should be accurate to about 6\n        digits. (The final ``x`` will usually have fewer correct digits,\n        depending on ``cond(A)`` and the size of LAMBDA.)  If `atol`\n        or `btol` is None, a default value of 1.0e-6 will be used.\n        Ideally, they should be estimates of the relative error in the\n        entries of ``A`` and ``b`` respectively.  For example, if the entries\n        of ``A`` have 7 correct digits, set ``atol = 1e-7``. This prevents\n        the algorithm from doing unnecessary work beyond the\n        uncertainty of the input data.\n    conlim : float, optional\n        `lsmr` terminates if an estimate of ``cond(A)`` exceeds\n        `conlim`.  For compatible systems ``Ax = b``, conlim could be\n        as large as 1.0e+12 (say).  For least-squares problems,\n        `conlim` should be less than 1.0e+8. If `conlim` is None, the\n        default value is 1e+8.  Maximum precision can be obtained by\n        setting ``atol = btol = conlim = 0``, but the number of\n        iterations may then be excessive.\n    maxiter : int, optional\n        `lsmr` terminates if the number of iterations reaches\n        `maxiter`.  The default is ``maxiter = min(m, n)``.  For\n        ill-conditioned systems, a larger value of `maxiter` may be\n        needed.\n    x0 : array_like, shape (n,), optional\n        Initial guess of ``x``, if None zeros are used.\n\n    Returns\n    -------\n    x : ndarray of float\n        Least-square solution returned.\n    itn : int\n        Number of iterations used.\n\n    \"\"\"\n    A = aslinearoperator(A)\n    b = torch.atleast_1d(b)\n    if b.dim() > 1:\n        b = b.squeeze()\n    eps = torch.finfo(b.dtype).eps\n    damp = torch.as_tensor(damp, dtype=b.dtype, device=b.device)\n    ctol = 1 / conlim if conlim > 0 else 0.\n    m, n = A.shape\n    if maxiter is None:\n        maxiter = min(m, n)\n\n    u = b.clone()\n    normb = b.norm()\n    if x0 is None:\n        x = b.new_zeros(n)\n        beta = normb.clone()\n    else:\n        x = torch.atleast_1d(x0).clone()\n        u.sub_(A.matvec(x))\n        beta = u.norm()\n\n    if beta > 0:\n        u.div_(beta)\n        v = A.rmatvec(u)\n        alpha = v.norm()\n    else:\n        v = b.new_zeros(n)\n        alpha = b.new_tensor(0)\n\n    v = torch.where(alpha > 0, v / alpha, v)\n\n    # Initialize variables for 1st iteration.\n\n    zetabar = alpha * beta\n    alphabar = alpha.clone()\n    rho = b.new_tensor(1)\n    rhobar = b.new_tensor(1)\n    cbar = b.new_tensor(1)\n    sbar = b.new_tensor(0)\n\n    h = v.clone()\n    hbar = b.new_zeros(n)\n\n    # Initialize variables for estimation of ||r||.\n\n    betadd = beta.clone()\n    betad = b.new_tensor(0)\n    rhodold = b.new_tensor(1)\n    tautildeold = b.new_tensor(0)\n    thetatilde = b.new_tensor(0)\n    zeta = b.new_tensor(0)\n    d = b.new_tensor(0)\n\n    # Initialize variables for estimation of ||A|| and cond(A)\n\n    normA2 = alpha.square()\n    maxrbar = b.new_tensor(0)\n    minrbar = b.new_tensor(0.99 * torch.finfo(b.dtype).max)\n    normA = normA2.sqrt()\n    condA = b.new_tensor(1)\n    normx = b.new_tensor(0)\n    normar = b.new_tensor(0)\n    normr = b.new_tensor(0)\n\n    # extra buffers (added by Reuben)\n    c = b.new_tensor(0)\n    s = b.new_tensor(0)\n    chat = b.new_tensor(0)\n    shat = b.new_tensor(0)\n    alphahat = b.new_tensor(0)\n    ctildeold = b.new_tensor(0)\n    stildeold = b.new_tensor(0)\n    rhotildeold = b.new_tensor(0)\n    rhoold = b.new_tensor(0)\n    rhobarold = b.new_tensor(0)\n    zetaold = b.new_tensor(0)\n    thetatildeold = b.new_tensor(0)\n    betaacute = b.new_tensor(0)\n    betahat = b.new_tensor(0)\n    betacheck = b.new_tensor(0)\n    taud = b.new_tensor(0)\n\n\n    # Main iteration loop.\n    for itn in range(1, maxiter+1):\n\n        # Perform the next step of the bidiagonalization to obtain the\n        # next  beta, u, alpha, v.  These satisfy the relations\n        #         beta*u  =  a*v   -  alpha*u,\n        #        alpha*v  =  A'*u  -  beta*v.\n\n        u.mul_(-alpha).add_(A.matvec(v))\n        torch.norm(u, out=beta)\n\n        if (not check_nonzero) or beta > 0:\n            # check_nonzero option provides a means to avoid the GPU-CPU\n            # synchronization of a `beta > 0` check. For most cases\n            # beta == 0 is unlikely, but use this option with caution.\n            u.div_(beta)\n            v.mul_(-beta).add_(A.rmatvec(u))\n            torch.norm(v, out=alpha)\n            v = torch.where(alpha > 0, v / alpha, v)\n\n        # At this point, beta = beta_{k+1}, alpha = alpha_{k+1}.\n\n        _sym_ortho(alphabar, damp, out=(chat, shat, alphahat))\n\n        # Use a plane rotation (Q_i) to turn B_i to R_i\n\n        rhoold.copy_(rho, non_blocking=True)\n        _sym_ortho(alphahat, beta, out=(c, s, rho))\n        thetanew = torch.mul(s, alpha)\n        torch.mul(c, alpha, out=alphabar)\n\n        # Use a plane rotation (Qbar_i) to turn R_i^T to R_i^bar\n\n        rhobarold.copy_(rhobar, non_blocking=True)\n        zetaold.copy_(zeta, non_blocking=True)\n        thetabar = sbar * rho\n        rhotemp = cbar * rho\n        _sym_ortho(cbar * rho, thetanew, out=(cbar, sbar, rhobar))\n        torch.mul(cbar, zetabar, out=zeta)\n        zetabar.mul_(-sbar)\n\n        # Update h, h_hat, x.\n\n        hbar.mul_(-thetabar * rho).div_(rhoold * rhobarold)\n        hbar.add_(h)\n        x.addcdiv_(zeta * hbar, rho * rhobar)\n        h.mul_(-thetanew).div_(rho)\n        h.add_(v)\n\n        # Estimate of ||r||.\n\n        # Apply rotation Qhat_{k,2k+1}.\n        torch.mul(chat, betadd, out=betaacute)\n        torch.mul(-shat, betadd, out=betacheck)\n\n        # Apply rotation Q_{k,k+1}.\n        torch.mul(c, betaacute, out=betahat)\n        torch.mul(-s, betaacute, out=betadd)\n\n        # Apply rotation Qtilde_{k-1}.\n        # betad = betad_{k-1} here.\n\n        thetatildeold.copy_(thetatilde, non_blocking=True)\n        _sym_ortho(rhodold, thetabar, out=(ctildeold, stildeold, rhotildeold))\n        torch.mul(stildeold, rhobar, out=thetatilde)\n        torch.mul(ctildeold, rhobar, out=rhodold)\n        betad.mul_(-stildeold).addcmul_(ctildeold, betahat)\n\n        # betad   = betad_k here.\n        # rhodold = rhod_k  here.\n\n        tautildeold.mul_(-thetatildeold).add_(zetaold).div_(rhotildeold)\n        torch.div(zeta - thetatilde * tautildeold, rhodold, out=taud)\n        d.addcmul_(betacheck, betacheck)\n        torch.sqrt(d + (betad - taud).square() + betadd.square(), out=normr)\n\n        # Estimate ||A||.\n        normA2.addcmul_(beta, beta)\n        torch.sqrt(normA2, out=normA)\n        normA2.addcmul_(alpha, alpha)\n\n        # Estimate cond(A).\n        torch.max(maxrbar, rhobarold, out=maxrbar)\n        if itn > 1:\n            torch.min(minrbar, rhobarold, out=minrbar)\n\n\n        # ------- Test for convergence --------\n\n        if itn % 10 == 0:\n\n            # Compute norms for convergence testing.\n            torch.abs(zetabar, out=normar)\n            torch.norm(x, out=normx)\n            torch.div(torch.max(maxrbar, rhotemp), torch.min(minrbar, rhotemp),\n                      out=condA)\n\n            # Now use these norms to estimate certain other quantities,\n            # some of which will be small near a solution.\n            test1 = normr / normb\n            test2 = normar / (normA * normr + eps)\n            test3 = 1 / (condA + eps)\n            t1 = test1 / (1 + normA * normx / normb)\n            rtol = btol + atol * normA * normx / normb\n\n            # The first 3 tests guard against extremely small values of\n            # atol, btol or ctol.  (The user may have set any or all of\n            # the parameters atol, btol, conlim  to 0.)\n            # The effect is equivalent to the normAl tests using\n            # atol = eps,  btol = eps,  conlim = 1/eps.\n\n            # The second 3 tests allow for tolerances set by the user.\n\n            stop = ((1 + test3 <= 1) | (1 + test2 <= 1) | (1 + t1 <= 1)\n                    | (test3 <= ctol) | (test2 <= atol) | (test1 <= rtol))\n\n            if stop:\n                break\n\n    return x, itn"
  },
  {
    "path": "torchmin/lstsq/trf.py",
    "content": "\"\"\"Trust Region Reflective algorithm for least-squares optimization.\n\"\"\"\nimport torch\nimport numpy as np\nfrom scipy.optimize import OptimizeResult\nfrom scipy.optimize._lsq.common import (print_header_nonlinear,\n                                        print_iteration_nonlinear)\n\nfrom .cg import cgls\nfrom .lsmr import lsmr\nfrom .linear_operator import jacobian_linop, jacobian_dense\nfrom .common import (right_multiplied_operator, build_quadratic_1d,\n                     minimize_quadratic_1d, evaluate_quadratic,\n                     solve_trust_region_2d, check_termination,\n                     update_tr_radius, solve_lsq_trust_region)\n\n\ndef trf(fun, x0, f0, lb, ub, ftol, xtol, gtol, max_nfev, x_scale,\n        tr_solver, tr_options, verbose):\n    # For efficiency, it makes sense to run the simplified version of the\n    # algorithm when no bounds are imposed. We decided to write the two\n    # separate functions. It violates the DRY principle, but the individual\n    # functions are kept the most readable.\n    if lb.isneginf().all() and ub.isposinf().all():\n        return trf_no_bounds(\n            fun, x0, f0, ftol, xtol, gtol, max_nfev, x_scale,\n            tr_solver, tr_options, verbose)\n    else:\n        raise NotImplementedError('trf with bounds not currently supported.')\n\n\ndef trf_no_bounds(fun, x0, f0=None, ftol=1e-8, xtol=1e-8, gtol=1e-8,\n                  max_nfev=None, x_scale=1.0, tr_solver='lsmr',\n                  tr_options=None, verbose=0):\n    if max_nfev is None:\n        max_nfev = x0.numel() * 100\n    if tr_options is None:\n        tr_options = {}\n    assert tr_solver in ['exact', 'lsmr', 'cgls']\n    if tr_solver == 'exact':\n        jacobian = jacobian_dense\n    else:\n        jacobian = jacobian_linop\n\n    x = x0.clone()\n    if f0 is None:\n        f = fun(x)\n    else:\n        f = f0\n    f_true = f.clone()\n    J = jacobian(fun, x)\n    nfev = njev = 1\n    m, n = J.shape\n\n    cost = 0.5 * f.dot(f)\n    g = J.T.mv(f)\n\n    scale = x_scale\n    Delta = (x0 / scale).norm()\n    if Delta == 0:\n        Delta.fill_(1.)\n\n    if tr_solver != 'exact':\n        damp = tr_options.pop('damp', 1e-4)\n        regularize = tr_options.pop('regularize', False)\n        reg_term = 0.\n\n    alpha = x0.new_tensor(0.)  # \"Levenberg-Marquardt\" parameter\n    termination_status = None\n    iteration = 0\n    step_norm = None\n    actual_reduction = None\n\n    if verbose == 2:\n        print_header_nonlinear()\n\n    while True:\n        g_norm = g.norm(np.inf)\n        if g_norm < gtol:\n            termination_status = 1\n\n        if verbose == 2:\n            print_iteration_nonlinear(iteration, nfev, cost, actual_reduction,\n                                      step_norm, g_norm)\n\n        if termination_status is not None or nfev == max_nfev:\n            break\n\n        d = scale\n        g_h = d * g\n\n        if tr_solver == 'exact':\n            J_h = J * d\n            U, s, V = torch.linalg.svd(J_h, full_matrices=False)\n            V = V.T\n            uf = U.T.mv(f)\n        else:\n            J_h = right_multiplied_operator(J, d)\n\n            if regularize:\n                a, b = build_quadratic_1d(J_h, g_h, -g_h)\n                to_tr = Delta / g_h.norm()\n                ag_value = minimize_quadratic_1d(a, b, 0, to_tr)[1]\n                reg_term = -ag_value / Delta**2\n\n            damp_full = (damp**2 + reg_term)**0.5\n            if tr_solver == 'lsmr':\n                gn_h = lsmr(J_h, f, damp=damp_full, **tr_options)[0]\n            elif tr_solver == 'cgls':\n                gn_h = cgls(J_h, f, alpha=damp_full, max_iter=min(m,n), **tr_options)\n            else:\n                raise RuntimeError\n            S = torch.vstack((g_h, gn_h)).T  # [n,2]\n            # Dispatch qr to CPU so long as pytorch/pytorch#22573 is not fixed\n            S = torch.linalg.qr(S.cpu(), mode='reduced')[0].to(S.device)  # [n,2]\n            JS = J_h.matmul(S)  # [m,2]\n            B_S = JS.T.matmul(JS)  # [2,2]\n            g_S = S.T.mv(g_h)  # [2]\n\n        actual_reduction = -1\n        while actual_reduction <= 0 and nfev < max_nfev:\n            if tr_solver == 'exact':\n                step_h, alpha, n_iter = solve_lsq_trust_region(\n                    n, m, uf, s, V, Delta, initial_alpha=alpha)\n            else:\n                p_S, _ = solve_trust_region_2d(B_S, g_S, Delta)\n                step_h = S.matmul(p_S)\n\n            predicted_reduction = -evaluate_quadratic(J_h, g_h, step_h)\n            step = d * step_h\n            x_new = x + step\n            f_new = fun(x_new)\n            nfev += 1\n\n            step_h_norm = step_h.norm()\n\n            if not f_new.isfinite().all():\n                Delta = 0.25 * step_h_norm\n                continue\n\n            # Usual trust-region step quality estimation.\n            cost_new = 0.5 * f_new.dot(f_new)\n            actual_reduction = cost - cost_new\n\n            Delta_new, ratio = update_tr_radius(\n                Delta, actual_reduction, predicted_reduction,\n                step_h_norm, step_h_norm > 0.95 * Delta)\n\n            step_norm = step.norm()\n            termination_status = check_termination(\n                actual_reduction, cost, step_norm, x.norm(), ratio, ftol, xtol)\n            if termination_status is not None:\n                break\n\n            alpha *= Delta / Delta_new\n            Delta = Delta_new\n\n        if actual_reduction > 0:\n            x, f, cost = x_new, f_new, cost_new\n            f_true.copy_(f)\n            J = jacobian(fun, x)\n            g = J.T.mv(f)\n            njev += 1\n        else:\n            step_norm = 0\n            actual_reduction = 0\n\n        iteration += 1\n\n    if termination_status is None:\n        termination_status = 0\n\n    active_mask = torch.zeros_like(x)\n    return OptimizeResult(\n        x=x, cost=cost, fun=f_true, jac=J, grad=g, optimality=g_norm,\n        active_mask=active_mask, nfev=nfev, njev=njev,\n        status=termination_status)\n"
  },
  {
    "path": "torchmin/minimize.py",
    "content": "import torch\n\nfrom .bfgs import _minimize_bfgs, _minimize_lbfgs\nfrom .cg import _minimize_cg\nfrom .newton import _minimize_newton_cg, _minimize_newton_exact\nfrom .trustregion import (_minimize_trust_exact, _minimize_dogleg,\n                          _minimize_trust_ncg, _minimize_trust_krylov)\n\n_tolerance_keys = {\n    'l-bfgs': 'gtol',\n    'bfgs': 'gtol',\n    'cg': 'gtol',\n    'newton-cg': 'xtol',\n    'newton-exact': 'xtol',\n    'dogleg': 'gtol',\n    'trust-ncg': 'gtol',\n    'trust-exact': 'gtol',\n    'trust-krylov': 'gtol'\n}\n\n\ndef minimize(\n        fun, x0, method, max_iter=None, tol=None, options=None, callback=None,\n        disp=0, return_all=False):\n    \"\"\"Minimize a scalar function of one or more variables.\n\n    .. note::\n        This is a general-purpose minimizer that calls one of the available\n        routines based on a supplied `method` argument.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    method : str\n        The minimization routine to use. Should be one of\n\n            - 'bfgs'\n            - 'l-bfgs'\n            - 'cg'\n            - 'newton-cg'\n            - 'newton-exact'\n            - 'dogleg'\n            - 'trust-ncg'\n            - 'trust-exact'\n            - 'trust-krylov'\n\n        At the moment, method must be specified; there is no default.\n    max_iter : int, optional\n        Maximum number of iterations to perform. If unspecified, this will\n        be set to the default of the selected method.\n    tol : float\n        Tolerance for termination. For detailed control, use solver-specific\n        options.\n    options : dict, optional\n        A dictionary of keyword arguments to pass to the selected minimization\n        routine.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool, optional\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    \"\"\"\n    x0 = torch.as_tensor(x0)\n    method = method.lower()\n    assert method in ['bfgs', 'l-bfgs', 'cg', 'newton-cg', 'newton-exact',\n                      'dogleg', 'trust-ncg', 'trust-exact', 'trust-krylov']\n    if options is None:\n        options = {}\n    if tol is not None:\n        options.setdefault(_tolerance_keys[method], tol)\n    options.setdefault('max_iter', max_iter)\n    options.setdefault('callback', callback)\n    options.setdefault('disp', disp)\n    options.setdefault('return_all', return_all)\n\n    if method == 'bfgs':\n        return _minimize_bfgs(fun, x0, **options)\n    elif method == 'l-bfgs':\n        return _minimize_lbfgs(fun, x0, **options)\n    elif method == 'cg':\n        return _minimize_cg(fun, x0, **options)\n    elif method == 'newton-cg':\n        return _minimize_newton_cg(fun, x0, **options)\n    elif method == 'newton-exact':\n        return _minimize_newton_exact(fun, x0, **options)\n    elif method == 'dogleg':\n        return _minimize_dogleg(fun, x0, **options)\n    elif method == 'trust-ncg':\n        return _minimize_trust_ncg(fun, x0, **options)\n    elif method == 'trust-exact':\n        return _minimize_trust_exact(fun, x0, **options)\n    elif method == 'trust-krylov':\n        return _minimize_trust_krylov(fun, x0, **options)\n    else:\n        raise RuntimeError('invalid method \"{}\" encountered.'.format(method))"
  },
  {
    "path": "torchmin/minimize_constr.py",
    "content": "import numbers\nimport numpy as np\nimport torch\nfrom scipy.optimize import Bounds\n\nfrom .constrained.lbfgsb import _minimize_lbfgsb\nfrom .constrained.frankwolfe import _minimize_frankwolfe\nfrom .constrained.trust_constr import _minimize_trust_constr\n\n\n_tolerance_keys = {\n    'l-bfgs-b': 'gtol',\n    'frank-wolfe': 'gtol',\n    'trust-constr': 'tol',\n}\n\n\ndef _maybe_to_number(val):\n    if isinstance(val, np.ndarray) and val.size == 1:\n        return val.item()\n    elif isinstance(val, torch.Tensor) and val.numel() == 1:\n        return val.item()\n    else:\n        return val\n\n\ndef _check_bound(val, x0, numpy=False):\n    n = x0.numel()\n    if isinstance(val, numbers.Number):\n        if numpy:\n            return np.full(n, val, dtype=float)  # TODO: correct dtype\n        else:\n            return x0.new_full((n,), val)\n\n    if isinstance(val, (list, tuple)):\n        if numpy:\n            val = np.array(val, dtype=float)  # TODO: correct dtype\n        else:\n            val = x0.new_tensor(val)\n\n    if isinstance(val, torch.Tensor):\n        assert val.numel() == n, f'Bound tensor has incorrect size'\n        val = val.flatten()\n        if numpy:\n            val = val.detach().cpu().numpy()\n        return val\n    elif isinstance(val, np.ndarray):\n        assert val.size == n, f'Bound array has incorrect size'\n        val = val.flatten()\n        if not numpy:\n            val = x0.new_tensor(val)\n        return val\n    else:\n        raise ValueError(f'Bound has invalid type: {type(val)}')\n\n\ndef _check_bounds(bounds, x0, method):\n    if isinstance(bounds, Bounds):\n        if method == 'trust-constr':\n            return bounds\n        else:\n            bounds = (bounds.lb, bounds.ub)\n            bounds = tuple(map(_maybe_to_number, bounds))\n\n    assert isinstance(bounds, (list, tuple)), \\\n        f'Argument `bounds` must be a list or tuple but got {type(bounds)}'\n    assert len(bounds) == 2, \\\n        f'Argument `bounds` must have length 2: (min, max)'\n    lb, ub = bounds\n\n    lb = float('-inf') if lb is None else lb\n    ub = float('inf') if ub is None else ub\n\n    numpy = (method == 'trust-constr')\n    lb = _check_bound(lb, x0, numpy=numpy)\n    ub = _check_bound(ub, x0, numpy=numpy)\n\n    return lb, ub\n\n\ndef minimize_constr(\n        f,\n        x0,\n        method=None,\n        constr=None,\n        bounds=None,\n        max_iter=None,\n        tol=None,\n        options=None,\n        callback=None,\n        disp=0,\n        ):\n    \"\"\"Minimize a scalar function of one or more variables subject to\n    bounds and/or constraints.\n\n    .. note::\n        Method ``'trust-constr'`` is currently a wrapper for SciPy's \n        `trust-constr <https://docs.scipy.org/doc/scipy/reference/optimize.minimize-trustconstr.html>`_ \n        solver.\n\n    Parameters\n    ----------\n    f : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    method : str, optional\n        The minimization routine to use. Should be one of the following:\n\n            - 'l-bfgs-b'\n            - 'frank-wolfe'\n            - 'trust-constr'\n\n        If no method is provided, a default method will be selected based\n        on the criteria of the problem.\n    constr : dict or string, optional\n        Constraint specifications. Should either be a string (Frank-Wolfe\n        method) or a dictionary (trust-constr method) with the following fields:\n\n            * fun (callable) - Constraint function\n            * lb (Tensor or float, optional) - Constraint lower bounds\n            * ub (Tensor or float, optional) - Constraint upper bounds\n\n        One of either `lb` or `ub` must be provided. When `lb` == `ub` it is\n        interpreted as an equality constraint.\n    bounds : sequence or `Bounds`, optional\n        Bounds on variables. There are two ways to specify the bounds:\n\n            1. Sequence of ``(min, max)`` pairs for each element in `x`. None\n               is used to specify no bound.\n            2. Instance of :class:`scipy.optimize.Bounds` class.\n\n        Bounds of `-inf`/`inf` are interpreted as no bound. When `lb` == `ub`\n        it is interpreted as an equality constraint.\n    max_iter : int, optional\n        Maximum number of iterations to perform. If unspecified, this will\n        be set to the default of the selected method.\n    tol : float, optional\n        Tolerance for termination. For detailed control, use solver-specific\n        options.\n    options : dict, optional\n        A dictionary of keyword arguments to pass to the selected minimization\n        routine.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int\n        Level of algorithm's verbosity:\n\n            * 0 : work silently (default).\n            * 1 : display a termination report.\n            * 2 : display progress during iterations.\n            * 3 : display progress during iterations (more complete report).\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    \"\"\"\n\n    if method is None:\n        if constr is not None:\n            _frank_wolfe_constraints = {\n                'tracenorm', 'trace-norm', 'birkhoff', 'birkhoff-polytope'}\n            if (\n                isinstance(constr, str)\n                and constr.lower() in _frank_wolfe_constraints\n                ):\n                method = 'frank-wolfe'\n            else:\n                method = 'trust-constr'\n        else:\n            method = 'l-bfgs-b'\n\n    assert isinstance(method, str)\n    method = method.lower()\n\n    if bounds is not None:\n        bounds = _check_bounds(bounds, x0, method)\n\n        # TODO: update `_minimize_trust_constr()` accepted bounds format\n        # and remove this\n        if method == 'trust-constr':\n            if isinstance(bounds, Bounds):\n                bounds = dict(\n                    lb=_maybe_to_number(bounds.lb),\n                    ub=_maybe_to_number(bounds.ub),\n                    keep_feasible=bounds.keep_feasible,\n                )\n            else:\n                bounds = dict(lb=bounds[0], ub=bounds[1])\n\n    if options is None:\n        options = {}\n    else:\n        assert isinstance(options, dict)\n        options = options.copy()\n    options.setdefault('max_iter', max_iter)\n    options.setdefault('callback', callback)\n    options.setdefault('disp', disp)\n    # options.setdefault('return_all', return_all)\n    if tol is not None:\n        options.setdefault(_tolerance_keys[method], tol)\n\n    if method == 'l-bfgs-b':\n        assert constr is None\n        return _minimize_lbfgsb(f, x0, bounds=bounds, **options)\n    elif method == 'frank-wolfe':\n        assert bounds is None\n        return _minimize_frankwolfe(f, x0, constr=constr, **options)\n    elif method == 'trust-constr':\n        return _minimize_trust_constr(\n            f, x0, constr=constr, bounds=bounds, **options)\n    else:\n        raise RuntimeError(f'Invalid method: \"{method}\".')\n"
  },
  {
    "path": "torchmin/newton.py",
    "content": "from scipy.optimize import OptimizeResult\nfrom scipy.sparse.linalg import eigsh\nfrom torch import Tensor\nimport torch\n\nfrom ._optimize import _status_message\nfrom .function import ScalarFunction\nfrom .line_search import strong_wolfe\n\n\n_status_message['cg_warn'] = \"Warning: CG iterations didn't converge. The \" \\\n                             \"Hessian is not positive definite.\"\n\n\ndef _cg_iters(grad, hess, max_iter, normp=1):\n    \"\"\"A CG solver specialized for the NewtonCG sub-problem.\n\n    Derived from Algorithm 7.1 of \"Numerical Optimization (2nd Ed.)\"\n    (Nocedal & Wright, 2006; pp. 169)\n    \"\"\"\n    # Get the most efficient dot product method for this problem\n    if grad.dim() == 1:\n        # standard dot product\n        dot = torch.dot\n    elif grad.dim() == 2:\n        # batched dot product\n        dot = lambda u,v: torch.bmm(u.unsqueeze(1), v.unsqueeze(2)).view(-1,1)\n    else:\n        # generalized dot product that supports batch inputs\n        dot = lambda u,v: u.mul(v).sum(-1, keepdim=True)\n\n    g_norm = grad.norm(p=normp)\n    tol = g_norm * g_norm.sqrt().clamp(0, 0.5)\n    eps = torch.finfo(grad.dtype).eps\n    n_iter = 0  # TODO: remove?\n    maxiter_reached = False\n\n    # initialize state and iterate\n    x = torch.zeros_like(grad)\n    r = grad.clone()\n    p = grad.neg()\n    rs = dot(r, r)\n    for n_iter in range(max_iter):\n        if r.norm(p=normp) < tol:\n            break\n        Bp = hess.mv(p)\n        curv = dot(p, Bp)\n        curv_sum = curv.sum()\n        if curv_sum < 0:\n            # hessian is not positive-definite\n            if n_iter == 0:\n                # if first step, fall back to steepest descent direction\n                # (scaled by Rayleigh quotient)\n                x = grad.mul(rs / curv)\n                #x = grad.neg()\n            break\n        elif curv_sum <= 3 * eps:\n            break\n        alpha = rs / curv\n        x.addcmul_(alpha, p)\n        r.addcmul_(alpha, Bp)\n        rs_new = dot(r, r)\n        p.mul_(rs_new / rs).sub_(r)\n        rs = rs_new\n    else:\n        # curvature keeps increasing; bail\n        maxiter_reached = True\n\n    return x, n_iter, maxiter_reached\n\n\n@torch.no_grad()\ndef _minimize_newton_cg(\n        fun, x0, lr=1., max_iter=None, cg_max_iter=None,\n        twice_diffable=True, line_search='strong-wolfe', xtol=1e-5,\n        normp=1, callback=None, disp=0, return_all=False):\n    \"\"\"Minimize a scalar function of one or more variables using the\n    Newton-Raphson method, with Conjugate Gradient for the linear inverse\n    sub-problem.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    lr : float\n        Step size for parameter updates. If using line search, this will be\n        used as the initial step size for the search.\n    max_iter : int, optional\n        Maximum number of iterations to perform. Defaults to\n        ``200 * x0.numel()``.\n    cg_max_iter : int, optional\n        Maximum number of iterations for CG subproblem. Recommended to\n        leave this at the default of ``20 * x0.numel()``.\n    twice_diffable : bool\n        Whether to assume the function is twice continuously differentiable.\n        If True, hessian-vector products will be much faster.\n    line_search : str\n        Line search specifier. Currently the available options are\n        {'none', 'strong_wolfe'}.\n    xtol : float\n        Average relative error in solution `xopt` acceptable for\n        convergence.\n    normp : Number or str\n        The norm type to use for termination conditions. Can be any value\n        supported by :func:`torch.norm`.\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n    \"\"\"\n    lr = float(lr)\n    disp = int(disp)\n    xtol = x0.numel() * xtol\n    if max_iter is None:\n        max_iter = x0.numel() * 200\n    if cg_max_iter is None:\n        cg_max_iter = x0.numel() * 20\n\n    # construct scalar objective function\n    sf = ScalarFunction(fun, x0.shape, hessp=True, twice_diffable=twice_diffable)\n    closure = sf.closure\n    if line_search == 'strong-wolfe':\n        dir_evaluate = sf.dir_evaluate\n\n    # initial settings\n    x = x0.detach().clone(memory_format=torch.contiguous_format)\n    f, g, hessp, _ = closure(x)\n    if disp > 1:\n        print('initial fval: %0.4f' % f)\n    if return_all:\n        allvecs = [x]\n    ncg = 0   # number of cg iterations\n    n_iter = 0\n\n    # begin optimization loop\n    for n_iter in range(1, max_iter + 1):\n\n        # ============================================================\n        #  Compute a search direction pk by applying the CG method to\n        #       H_f(xk) p = - J_f(xk) starting from 0.\n        # ============================================================\n\n        # Compute search direction with conjugate gradient (GG)\n        d, cg_iters, cg_fail = _cg_iters(g, hessp, cg_max_iter, normp)\n        ncg += cg_iters\n\n        if cg_fail:\n            warnflag = 3\n            msg = _status_message['cg_warn']\n            break\n\n        # =====================================================\n        #  Perform variable update (with optional line search)\n        # =====================================================\n\n        if line_search == 'none':\n            update = d.mul(lr)\n            x = x + update\n        elif line_search == 'strong-wolfe':\n            # strong-wolfe line search\n            _, _, t, ls_nevals = strong_wolfe(dir_evaluate, x, lr, d, f, g)\n            update = d.mul(t)\n            x = x + update\n        else:\n            raise ValueError('invalid line_search option {}.'.format(line_search))\n\n        # re-evaluate function\n        f, g, hessp, _ = closure(x)\n\n        if disp > 1:\n            print('iter %3d - fval: %0.4f' % (n_iter, f))\n        if callback is not None:\n            if callback(x):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n        if return_all:\n            allvecs.append(x)\n\n        # ==========================\n        #  check for convergence\n        # ==========================\n\n        if update.norm(p=normp) <= xtol:\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n        if not f.isfinite():\n            warnflag = 3\n            msg = _status_message['nan']\n            break\n\n    else:\n        # if we get to the end, the maximum num. iterations was reached\n        warnflag = 1\n        msg = _status_message['maxiter']\n\n    if disp:\n        print(msg)\n        print(\"         Current function value: %f\" % f)\n        print(\"         Iterations: %d\" % n_iter)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n        print(\"         CG iterations: %d\" % ncg)\n    result = OptimizeResult(fun=f, x=x.view_as(x0), grad=g.view_as(x0),\n                            status=warnflag, success=(warnflag==0),\n                            message=msg, nit=n_iter, nfev=sf.nfev, ncg=ncg)\n    if return_all:\n        result['allvecs'] = allvecs\n    return result\n\n\n\n@torch.no_grad()\ndef _minimize_newton_exact(\n        fun, x0, lr=1., max_iter=None, line_search='strong-wolfe', xtol=1e-5,\n        normp=1, tikhonov=0., handle_npd='grad', callback=None, disp=0,\n        return_all=False):\n    \"\"\"Minimize a scalar function of one or more variables using the\n    Newton-Raphson method.\n\n    This variant uses an \"exact\" Newton routine based on Cholesky factorization\n    of the explicit Hessian matrix.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    lr : float\n        Step size for parameter updates. If using line search, this will be\n        used as the initial step size for the search.\n    max_iter : int, optional\n        Maximum number of iterations to perform. Defaults to\n        ``200 * x0.numel()``.\n    line_search : str\n        Line search specifier. Currently the available options are\n        {'none', 'strong_wolfe'}.\n    xtol : float\n        Average relative error in solution `xopt` acceptable for\n        convergence.\n    normp : Number or str\n        The norm type to use for termination conditions. Can be any value\n        supported by :func:`torch.norm`.\n    tikhonov : float\n        Optional diagonal regularization (Tikhonov) parameter for the Hessian.\n    handle_npd : str\n        Mode for handling non-positive definite hessian matrices. Can be one\n        of the following:\n\n            * 'grad' : use steepest descent direction (gradient)\n            * 'lu' : solve the inverse hessian with LU factorization\n            * 'eig' : use symmetric eigendecomposition to determine a\n              diagonal regularization parameter\n    callback : callable, optional\n        Function to call after each iteration with the current parameter\n        state, e.g. ``callback(x)``.\n    disp : int or bool\n        Display (verbosity) level. Set to >0 to print status messages.\n    return_all : bool\n        Set to True to return a list of the best solution at each of the\n        iterations.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n    \"\"\"\n    lr = float(lr)\n    disp = int(disp)\n    xtol = x0.numel() * xtol\n    if max_iter is None:\n        max_iter = x0.numel() * 200\n\n    # Construct scalar objective function\n    sf = ScalarFunction(fun, x0.shape, hess=True)\n    closure = sf.closure\n    if line_search == 'strong-wolfe':\n        dir_evaluate = sf.dir_evaluate\n\n    # initial settings\n    x = x0.detach().view(-1).clone(memory_format=torch.contiguous_format)\n    f, g, _, hess = closure(x)\n    if tikhonov > 0:\n        hess.diagonal().add_(tikhonov)\n    if disp > 1:\n        print('initial fval: %0.4f' % f)\n    if return_all:\n        allvecs = [x]\n    nfail = 0\n    n_iter = 0\n\n    # begin optimization loop\n    for n_iter in range(1, max_iter + 1):\n\n        # ==================================================\n        #  Compute a search direction d by solving\n        #          H_f(x) d = - J_f(x)\n        #  with the true Hessian and Cholesky factorization\n        # ===================================================\n\n        # Compute search direction with Cholesky solve\n        L, info = torch.linalg.cholesky_ex(hess)\n\n        if info == 0:\n            d = torch.cholesky_solve(g.neg().unsqueeze(1), L).squeeze(1)\n        else:\n            nfail += 1\n            if handle_npd == 'lu':\n                d = torch.linalg.solve(hess, g.neg())\n            elif handle_npd in ['grad', 'cauchy']:\n                d = g.neg()\n                if handle_npd == 'cauchy':\n                    # cauchy point for a trust radius of delta=1.\n                    # equivalent to 'grad' with a scaled lr\n                    gnorm = g.norm(p=2)\n                    scale = 1 / gnorm\n                    gHg = g.dot(hess.mv(g))\n                    if gHg > 0:\n                        scale *= torch.clamp_(gnorm.pow(3) / gHg, max=1)\n                    d *= scale\n            elif handle_npd == 'eig':\n                # this setting is experimental! use with caution\n                # TODO: why use the factor 1.5 here? Seems to work best\n                eig0 = eigsh(hess.cpu().numpy(), k=1, which=\"SA\", tol=1e-4)[0].item()\n                tau = max(1e-3 - 1.5 * eig0, 0)\n                hess.diagonal().add_(tau)\n                L = torch.linalg.cholesky(hess)\n                d = torch.cholesky_solve(g.neg().unsqueeze(1), L).squeeze(1)\n            else:\n                raise RuntimeError('invalid handle_npd encountered.')\n\n\n        # =====================================================\n        #  Perform variable update (with optional line search)\n        # =====================================================\n\n        if line_search == 'none':\n            update = d.mul(lr)\n            x = x + update\n        elif line_search == 'strong-wolfe':\n            # strong-wolfe line search\n            _, _, t, ls_nevals = strong_wolfe(dir_evaluate, x, lr, d, f, g)\n            update = d.mul(t)\n            x = x + update\n        else:\n            raise ValueError('invalid line_search option {}.'.format(line_search))\n\n        # ===================================\n        #  Re-evaluate func/Jacobian/Hessian\n        # ===================================\n\n        f, g, _, hess = closure(x)\n        if tikhonov > 0:\n            hess.diagonal().add_(tikhonov)\n\n        if disp > 1:\n            print('iter %3d - fval: %0.4f - info: %d' % (n_iter, f, info))\n        if callback is not None:\n            if callback(x):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n        if return_all:\n            allvecs.append(x)\n\n        # ==========================\n        #  check for convergence\n        # ==========================\n\n        if update.norm(p=normp) <= xtol:\n            warnflag = 0\n            msg = _status_message['success']\n            break\n\n        if not f.isfinite():\n            warnflag = 3\n            msg = _status_message['nan']\n            break\n\n    else:\n        # if we get to the end, the maximum num. iterations was reached\n        warnflag = 1\n        msg = _status_message['maxiter']\n\n    if disp:\n        print(msg)\n        print(\"         Current function value: %f\" % f)\n        print(\"         Iterations: %d\" % n_iter)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n    result = OptimizeResult(fun=f, x=x.view_as(x0), grad=g.view_as(x0),\n                            hess=hess.view(2 * x0.shape),\n                            status=warnflag, success=(warnflag==0),\n                            message=msg, nit=n_iter, nfev=sf.nfev, nfail=nfail)\n    if return_all:\n        result['allvecs'] = allvecs\n    return result\n"
  },
  {
    "path": "torchmin/optim/__init__.py",
    "content": "from .minimizer import Minimizer\nfrom .scipy_minimizer import ScipyMinimizer"
  },
  {
    "path": "torchmin/optim/minimizer.py",
    "content": "from functools import reduce\nimport torch\nfrom torch.optim import Optimizer\n\n\nclass LinearOperator:\n    \"\"\"A generic linear operator to use with Minimizer\"\"\"\n    def __init__(self, matvec, shape, dtype=torch.float, device=None):\n        self.rmv = matvec\n        self.mv = matvec\n        self.shape = shape\n        self.dtype = dtype\n        self.device = device\n\n\nclass Minimizer(Optimizer):\n    \"\"\"A general-purpose PyTorch optimizer for unconstrained function\n    minimization.\n\n    .. warning::\n        This optimizer doesn't support per-parameter options and parameter\n        groups (there can be only one).\n\n    .. warning::\n        Right now all parameters have to be on a single device. This will be\n        improved in the future.\n\n    Parameters\n    ----------\n    params : iterable\n        An iterable of :class:`torch.Tensor` s. Specifies what Tensors\n        should be optimized.\n    method : str\n        Minimization method (algorithm) to use. Must be one of the methods\n        offered in :func:`torchmin.minimize()`. Defaults to 'bfgs'.\n    **minimize_kwargs : dict\n        Additional keyword arguments that will be passed to\n        :func:`torchmin.minimize()`.\n\n    \"\"\"\n    def __init__(self,\n                 params,\n                 method='bfgs',\n                 **minimize_kwargs):\n        assert isinstance(method, str)\n        method_ = method.lower()\n\n        self._hessp = self._hess = False\n        if method_ in ['bfgs', 'l-bfgs', 'cg']:\n            pass\n        elif method_ in ['newton-cg', 'trust-ncg', 'trust-krylov']:\n            self._hessp = True\n        elif method_ in ['newton-exact', 'dogleg', 'trust-exact']:\n            self._hess = True\n        else:\n            raise ValueError('Unknown method {}'.format(method))\n\n        defaults = dict(method=method_, **minimize_kwargs)\n        super().__init__(params, defaults)\n\n        if len(self.param_groups) != 1:\n            raise ValueError(\"Minimizer doesn't support per-parameter options\")\n\n        self._nfev = [0]\n        self._params = self.param_groups[0]['params']\n        self._numel_cache = None\n        self._closure = None\n        self._result = None\n\n    @property\n    def nfev(self):\n        return self._nfev[0]\n\n    def _numel(self):\n        if self._numel_cache is None:\n            self._numel_cache = reduce(lambda total, p: total + p.numel(), self._params, 0)\n        return self._numel_cache\n\n    def _gather_flat_param(self):\n        params = []\n        for p in self._params:\n            if p.data.is_sparse:\n                p = p.data.to_dense().view(-1)\n            else:\n                p = p.data.view(-1)\n            params.append(p)\n        return torch.cat(params)\n\n    def _gather_flat_grad(self):\n        grads = []\n        for p in self._params:\n            if p.grad is None:\n                g = p.new_zeros(p.numel())\n            elif p.grad.is_sparse:\n                g = p.grad.to_dense().view(-1)\n            else:\n                g = p.grad.view(-1)\n            grads.append(g)\n        return torch.cat(grads)\n\n    def _set_flat_param(self, value):\n        offset = 0\n        for p in self._params:\n            numel = p.numel()\n            p.copy_(value[offset:offset+numel].view_as(p))\n            offset += numel\n        assert offset == self._numel()\n\n    def closure(self, x):\n        from torchmin.function import sf_value\n\n        assert self._closure is not None\n        self._set_flat_param(x)\n        with torch.enable_grad():\n            f = self._closure()\n            f.backward(create_graph=self._hessp or self._hess)\n            grad = self._gather_flat_grad()\n\n        grad_out = grad.detach().clone()\n        hessp = None\n        hess = None\n        if self._hessp or self._hess:\n            grad_accum = grad.detach().clone()\n            def hvp(v):\n                assert v.shape == grad.shape\n                grad.backward(gradient=v, retain_graph=True)\n                output = self._gather_flat_grad().detach() - grad_accum\n                grad_accum.add_(output)\n                return output\n\n            numel = self._numel()\n            if self._hessp:\n                hessp = LinearOperator(hvp, shape=(numel, numel),\n                                       dtype=grad.dtype, device=grad.device)\n            if self._hess:\n                eye = torch.eye(numel, dtype=grad.dtype, device=grad.device)\n                hess = torch.zeros(numel, numel, dtype=grad.dtype, device=grad.device)\n                for i in range(numel):\n                    hess[i] = hvp(eye[i])\n\n        return sf_value(f=f.detach(), grad=grad_out.detach(), hessp=hessp, hess=hess)\n\n    def dir_evaluate(self, x, t, d):\n        from torchmin.function import de_value\n\n        self._set_flat_param(x + d.mul(t))\n        with torch.enable_grad():\n            f = self._closure()\n        f.backward()\n        grad = self._gather_flat_grad()\n        self._set_flat_param(x)\n\n        return de_value(f=float(f), grad=grad)\n\n    @torch.no_grad()\n    def step(self, closure):\n        \"\"\"Perform an optimization step.\n\n        The function \"closure\" should have a slightly different\n        form vs. the PyTorch standard: namely, it should not include any\n        `backward()` calls. Backward steps will be performed internally\n        by the optimizer.\n\n        >>> def closure():\n        >>>    optimizer.zero_grad()\n        >>>    output = model(input)\n        >>>    loss = loss_fn(output, target)\n        >>>    # loss.backward() <-- skip this step!\n        >>>    return loss\n\n        Parameters\n        ----------\n        closure : callable\n            A function that re-evaluates the model and returns the loss.\n\n        \"\"\"\n        from torchmin.minimize import minimize\n\n        # sanity check\n        assert len(self.param_groups) == 1\n\n        # overwrite closure\n        closure_ = closure\n        def closure():\n            self._nfev[0] += 1\n            return closure_()\n        self._closure = closure\n\n        # get initial value\n        x0 = self._gather_flat_param()\n\n        # perform parameter update\n        kwargs = {k:v for k,v in self.param_groups[0].items() if k != 'params'}\n        self._result = minimize(self, x0, **kwargs)\n\n        # set final value\n        self._set_flat_param(self._result.x)\n\n        return self._result.fun"
  },
  {
    "path": "torchmin/optim/scipy_minimizer.py",
    "content": "import numbers\nimport numpy as np\nimport torch\nfrom functools import reduce\nfrom torch.optim import Optimizer\nfrom scipy import optimize\nfrom torch._vmap_internals import _vmap\nfrom torch.autograd.functional import (_construct_standard_basis_for,\n                                       _grad_postprocess, _tuple_postprocess,\n                                       _as_tuple)\n\n\ndef _build_bounds(bounds, params, numel_total):\n    if len(bounds) != len(params):\n        raise ValueError('bounds must be an iterable with same length as params')\n\n    lb = np.full(numel_total, -np.inf)\n    ub = np.full(numel_total, np.inf)\n    keep_feasible = np.zeros(numel_total, dtype=np.bool)\n\n    def process_bound(x, numel):\n        if isinstance(x, torch.Tensor):\n            assert x.numel() == numel\n            return x.view(-1).detach().cpu().numpy()\n        elif isinstance(x, np.ndarray):\n            assert x.size == numel\n            return x.flatten()\n        elif isinstance(x, (bool, numbers.Number)):\n            return x\n        else:\n            raise ValueError('invalid bound value.')\n\n    offset = 0\n    for bound, p in zip(bounds, params):\n        numel = p.numel()\n        if bound is None:\n            offset += numel\n            continue\n        if not isinstance(bound, (list, tuple)) and len(bound) in [2,3]:\n            raise ValueError('elements of \"bounds\" must each be a '\n                             'list/tuple of length 2 or 3')\n        if bound[0] is None and bound[1] is None:\n            raise ValueError('either lower or upper bound must be defined.')\n        if bound[0] is not None:\n            lb[offset:offset + numel] = process_bound(bound[0], numel)\n        if bound[1] is not None:\n            ub[offset:offset + numel] = process_bound(bound[1], numel)\n        if len(bound) == 3:\n            keep_feasible[offset:offset + numel] = process_bound(bound[2], numel)\n        offset += numel\n\n    return optimize.Bounds(lb, ub, keep_feasible)\n\n\ndef _jacobian(inputs, outputs):\n    \"\"\"A modified variant of torch.autograd.functional.jacobian for\n    pre-computed outputs\n\n    This is only used for nonlinear parameter constraints (if provided)\n    \"\"\"\n    is_inputs_tuple, inputs = _as_tuple(inputs, \"inputs\", \"jacobian\")\n    is_outputs_tuple, outputs = _as_tuple(outputs, \"outputs\", \"jacobian\")\n\n    output_numels = tuple(output.numel() for output in outputs)\n    grad_outputs = _construct_standard_basis_for(outputs, output_numels)\n    with torch.enable_grad():\n        flat_outputs = tuple(output.reshape(-1) for output in outputs)\n\n    def vjp(grad_output):\n        vj = list(torch.autograd.grad(flat_outputs, inputs, grad_output, allow_unused=True))\n        for el_idx, vj_el in enumerate(vj):\n            if vj_el is not None:\n                continue\n            vj[el_idx] = torch.zeros_like(inputs[el_idx])\n        return tuple(vj)\n\n    jacobians_of_flat_output = _vmap(vjp)(grad_outputs)\n\n    jacobian_input_output = []\n    for jac, input_i in zip(jacobians_of_flat_output, inputs):\n        jacobian_input_i_output = []\n        for jac, output_j in zip(jac.split(output_numels, dim=0), outputs):\n            jacobian_input_i_output_j = jac.view(output_j.shape + input_i.shape)\n            jacobian_input_i_output.append(jacobian_input_i_output_j)\n        jacobian_input_output.append(jacobian_input_i_output)\n\n    jacobian_output_input = tuple(zip(*jacobian_input_output))\n\n    jacobian_output_input = _grad_postprocess(jacobian_output_input, create_graph=False)\n    return _tuple_postprocess(jacobian_output_input, (is_outputs_tuple, is_inputs_tuple))\n\n\nclass ScipyMinimizer(Optimizer):\n    \"\"\"A PyTorch optimizer for constrained & unconstrained function\n    minimization.\n\n    .. note::\n        This optimizer is a wrapper for :func:`scipy.optimize.minimize`.\n        It uses autograd behind the scenes to build jacobian & hessian\n        callables before invoking scipy. Inputs and objectivs should use\n        PyTorch tensors like other routines. CUDA is supported; however,\n        data will be transferred back-and-forth between GPU/CPU.\n\n    .. warning::\n        This optimizer doesn't support per-parameter options and parameter\n        groups (there can be only one).\n\n    .. warning::\n        Right now all parameters have to be on a single device. This will be\n        improved in the future.\n\n    Parameters\n    ----------\n    params : iterable\n        An iterable of :class:`torch.Tensor` s. Specifies what Tensors\n        should be optimized.\n    method : str\n        One of the various optimization methods offered in scipy minimize.\n        Defaults to 'bfgs'.\n    bounds : iterable, optional\n        An iterable of :class:`torch.Tensor` s or :class:`float` s with same\n        length as `params`. Specifies boundaries for each parameter.\n    constraints : dict, optional\n        TODO\n    tol : float, optional\n        TODO\n    options : dict, optional\n        TODO\n\n    \"\"\"\n    def __init__(self,\n                 params,\n                 method='bfgs',\n                 bounds=None,\n                 constraints=(),  # experimental feature! use with caution\n                 tol=None,\n                 options=None):\n        assert isinstance(method, str)\n        method = method.lower()\n        defaults = dict(\n            method=method,\n            bounds=bounds,\n            constraints=constraints,\n            tol=tol,\n            options=options)\n        super().__init__(params, defaults)\n\n        if len(self.param_groups) != 1:\n            raise ValueError(\"Minimize doesn't support per-parameter options \"\n                             \"(parameter groups)\")\n        if constraints != () and method != 'trust-constr':\n            raise NotImplementedError(\"Constraints only currently supported for \"\n                                      \"method='trust-constr'.\")\n\n        self._params = self.param_groups[0]['params']\n        self._param_bounds = self.param_groups[0]['bounds']\n        self._numel_cache = None\n        self._bounds_cache = None\n        self._result = None\n\n    def _numel(self):\n        if self._numel_cache is None:\n            self._numel_cache = reduce(lambda total, p: total + p.numel(), self._params, 0)\n        return self._numel_cache\n\n    def _bounds(self):\n        if self._param_bounds is None:\n            return None\n        if self._bounds_cache is None:\n            self._bounds_cache = _build_bounds(self._param_bounds, self._params,\n                                               self._numel())\n        return self._bounds_cache\n\n    def _gather_flat_param(self):\n        views = []\n        for p in self._params:\n            if p.data.is_sparse:\n                view = p.data.to_dense().view(-1)\n            else:\n                view = p.data.view(-1)\n            views.append(view)\n        return torch.cat(views, 0)\n\n    def _gather_flat_grad(self):\n        views = []\n        for p in self._params:\n            if p.grad is None:\n                view = p.new_zeros(p.numel())\n            elif p.grad.is_sparse:\n                view = p.grad.to_dense().view(-1)\n            else:\n                view = p.grad.view(-1)\n            views.append(view)\n        return torch.cat(views, 0)\n\n    def _set_flat_param(self, value):\n        offset = 0\n        for p in self._params:\n            numel = p.numel()\n            # view as to avoid deprecated pointwise semantics\n            p.copy_(value[offset:offset + numel].view_as(p))\n            offset += numel\n        assert offset == self._numel()\n\n    def _build_constraints(self, constraints):\n        assert isinstance(constraints, dict)\n        assert 'fun' in constraints\n        assert 'lb' in constraints or 'ub' in constraints\n\n        to_tensor = lambda x: self._params[0].new_tensor(x)\n        to_array = lambda x: x.cpu().numpy()\n        fun_ = constraints['fun']\n        lb = constraints.get('lb', -np.inf)\n        ub = constraints.get('ub', np.inf)\n        strict = constraints.get('keep_feasible', False)\n        lb = to_array(lb) if torch.is_tensor(lb) else lb\n        ub = to_array(ub) if torch.is_tensor(ub) else ub\n        strict = to_array(strict) if torch.is_tensor(strict) else strict\n\n        def fun(x):\n            self._set_flat_param(to_tensor(x))\n            return to_array(fun_())\n\n        def jac(x):\n            self._set_flat_param(to_tensor(x))\n            with torch.enable_grad():\n                output = fun_()\n\n            # this is now a tuple of tensors, one per parameter, each with\n            # shape (num_outputs, *param_shape).\n            J_seq = _jacobian(inputs=tuple(self._params), outputs=output)\n\n            # flatten and stack the tensors along dim 1 to get our full matrix\n            J = torch.cat([elt.view(output.numel(), -1) for elt in J_seq], 1)\n\n            return to_array(J)\n\n        return optimize.NonlinearConstraint(fun, lb, ub, jac=jac, keep_feasible=strict)\n\n    @torch.no_grad()\n    def step(self, closure):\n        \"\"\"Perform an optimization step.\n\n        Parameters\n        ----------\n        closure : callable\n            A function that re-evaluates the model and returns the loss.\n            See the `closure instructions\n            <https://pytorch.org/docs/stable/optim.html#optimizer-step-closure>`_\n            from PyTorch Optimizer docs for areference on how to construct\n            this callable.\n        \"\"\"\n        # sanity check\n        assert len(self.param_groups) == 1\n\n        # functions to convert numpy -> torch and torch -> numpy\n        to_tensor = lambda x: self._params[0].new_tensor(x)\n        to_array = lambda x: x.cpu().numpy()\n\n        # optimizer settings\n        group = self.param_groups[0]\n        method = group['method']\n        bounds = self._bounds()\n        constraints = group['constraints']\n        tol = group['tol']\n        options = group['options']\n\n        # build constraints (if provided)\n        if constraints != ():\n            constraints = self._build_constraints(constraints)\n\n        # build objective\n        def fun(x):\n            x = to_tensor(x)\n            self._set_flat_param(x)\n            with torch.enable_grad():\n                loss = closure()\n            grad = self._gather_flat_grad()\n            return float(loss), to_array(grad)\n\n        # initial value (numpy array)\n        x0 = to_array(self._gather_flat_param())\n\n        # optimize\n        self._result = optimize.minimize(\n            fun, x0, method=method, jac=True, bounds=bounds,\n            constraints=constraints, tol=tol, options=options\n        )\n\n        # set final param\n        self._set_flat_param(to_tensor(self._result.x))\n\n        return to_tensor(self._result.fun)\n"
  },
  {
    "path": "torchmin/trustregion/__init__.py",
    "content": "from .ncg import _minimize_trust_ncg\nfrom .exact import _minimize_trust_exact\nfrom .dogleg import _minimize_dogleg\nfrom .krylov import _minimize_trust_krylov"
  },
  {
    "path": "torchmin/trustregion/base.py",
    "content": "\"\"\"\nTrust-region optimization.\n\nCode ported from SciPy to PyTorch\n\nCopyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers.\nAll rights reserved.\n\"\"\"\nfrom abc import ABC, abstractmethod\nimport torch\nfrom torch.linalg import norm\nfrom scipy.optimize import OptimizeResult\n\nfrom .._optimize import _status_message\nfrom ..function import ScalarFunction\nfrom ..optim.minimizer import Minimizer\n\nstatus_messages = (\n    _status_message['success'],\n    _status_message['maxiter'],\n    'A bad approximation caused failure to predict improvement.',\n    'A linalg error occurred, such as a non-psd Hessian.',\n)\n\n\nclass BaseQuadraticSubproblem(ABC):\n    \"\"\"\n    Base/abstract class defining the quadratic model for trust-region\n    minimization. Child classes must implement the ``solve`` method and\n    ``hess_prod`` property.\n    \"\"\"\n    def __init__(self, x, closure):\n        # evaluate closure\n        f, g, hessp, hess = closure(x)\n\n        self._x = x\n        self._f = f\n        self._g = g\n        self._h = hessp if self.hess_prod else hess\n        self._g_mag = None\n        self._cauchy_point = None\n        self._newton_point = None\n\n        # buffer for boundaries computation\n        self._tab = x.new_empty(2)\n\n    def __call__(self, p):\n        return self.fun + self.jac.dot(p) + 0.5 * p.dot(self.hessp(p))\n\n    @property\n    def fun(self):\n        \"\"\"Value of objective function at current iteration.\"\"\"\n        return self._f\n\n    @property\n    def jac(self):\n        \"\"\"Value of Jacobian of objective function at current iteration.\"\"\"\n        return self._g\n\n    @property\n    def hess(self):\n        \"\"\"Value of Hessian of objective function at current iteration.\"\"\"\n        if self.hess_prod:\n            raise Exception('class {} does not have '\n                            'method `hess`'.format(type(self)))\n        return self._h\n\n    def hessp(self, p):\n        \"\"\"Value of Hessian-vector product at current iteration for a\n        particular vector ``p``.\n\n        Note: ``self._h`` is either a Tensor or a LinearOperator. In either\n        case, it has a method ``mv()``.\n        \"\"\"\n        return self._h.mv(p)\n\n    @property\n    def jac_mag(self):\n        \"\"\"Magnitude of jacobian of objective function at current iteration.\"\"\"\n        if self._g_mag is None:\n            self._g_mag = norm(self.jac)\n        return self._g_mag\n\n    def get_boundaries_intersections(self, z, d, trust_radius):\n        \"\"\"\n        Solve the scalar quadratic equation ||z + t d|| == trust_radius.\n        This is like a line-sphere intersection.\n        Return the two values of t, sorted from low to high.\n        \"\"\"\n        a = d.dot(d)\n        b = 2 * z.dot(d)\n        c = z.dot(z) - trust_radius**2\n        sqrt_discriminant = torch.sqrt(b*b - 4*a*c)\n\n        # The following calculation is mathematically equivalent to:\n        #   ta = (-b - sqrt_discriminant) / (2*a)\n        #   tb = (-b + sqrt_discriminant) / (2*a)\n        # but produces smaller round off errors.\n        aux = b + torch.copysign(sqrt_discriminant, b)\n        self._tab[0] = -aux / (2*a)\n        self._tab[1] = -2*c / aux\n        return self._tab.sort()[0]\n\n    @abstractmethod\n    def solve(self, trust_radius):\n        pass\n\n    @property\n    @abstractmethod\n    def hess_prod(self):\n        \"\"\"A property that must be set by every sub-class indicating whether\n        to use full hessian matrix or hessian-vector products.\"\"\"\n        pass\n\n\n@torch.no_grad()\ndef _minimize_trust_region(fun, x0, subproblem=None, initial_trust_radius=1.,\n                           max_trust_radius=1000., eta=0.15, gtol=1e-4,\n                           max_iter=None, disp=False, return_all=False,\n                           callback=None):\n    \"\"\"\n    Minimization of scalar function of one or more variables using a\n    trust-region algorithm.\n\n    Options for the trust-region algorithm are:\n        initial_trust_radius : float\n            Initial trust radius.\n        max_trust_radius : float\n            Never propose steps that are longer than this value.\n        eta : float\n            Trust region related acceptance stringency for proposed steps.\n        gtol : float\n            Gradient norm must be less than `gtol`\n            before successful termination.\n        max_iter : int\n            Maximum number of iterations to perform.\n        disp : bool\n            If True, print convergence message.\n\n    This function is called by :func:`torchmin.minimize`.\n    It is not supposed to be called directly.\n    \"\"\"\n    if subproblem is None:\n        raise ValueError('A subproblem solving strategy is required for '\n                         'trust-region methods')\n    if not (0 <= eta < 0.25):\n        raise Exception('invalid acceptance stringency')\n    if max_trust_radius <= 0:\n        raise Exception('the max trust radius must be positive')\n    if initial_trust_radius <= 0:\n        raise ValueError('the initial trust radius must be positive')\n    if initial_trust_radius >= max_trust_radius:\n        raise ValueError('the initial trust radius must be less than the '\n                         'max trust radius')\n\n    # Input check/pre-process\n    disp = int(disp)\n    if max_iter is None:\n        max_iter = x0.numel() * 200\n\n    # Construct scalar objective function\n    hessp = subproblem.hess_prod\n    sf = ScalarFunction(fun, x0.shape, hessp=hessp, hess=not hessp)\n    closure = sf.closure\n\n    # init the search status\n    warnflag = 1  # maximum iterations flag\n    k = 0\n\n    # initialize the search\n    trust_radius = torch.as_tensor(initial_trust_radius,\n                                   dtype=x0.dtype, device=x0.device)\n    x = x0.detach().flatten()\n    if return_all:\n        allvecs = [x]\n\n    # initial subproblem\n    m = subproblem(x, closure)\n\n    # search for the function min\n    # do not even start if the gradient is small enough\n    while k < max_iter:\n\n        # Solve the sub-problem.\n        # This gives us the proposed step relative to the current position\n        # and it tells us whether the proposed step\n        # has reached the trust region boundary or not.\n        try:\n            p, hits_boundary = m.solve(trust_radius)\n        except RuntimeError as exc:\n            # TODO: catch general linalg error like np.linalg.linalg.LinAlgError\n            if 'singular' in exc.args[0]:\n                warnflag = 3\n                break\n            else:\n                raise\n\n        # calculate the predicted value at the proposed point\n        predicted_value = m(p)\n\n        # define the local approximation at the proposed point\n        x_proposed = x + p\n        m_proposed = subproblem(x_proposed, closure)\n\n        # evaluate the ratio defined in equation (4.4)\n        actual_reduction = m.fun - m_proposed.fun\n        predicted_reduction = m.fun - predicted_value\n        if predicted_reduction <= 0:\n            warnflag = 2\n            break\n        rho = actual_reduction / predicted_reduction\n\n        # update the trust radius according to the actual/predicted ratio\n        if rho < 0.25:\n            trust_radius = trust_radius.mul(0.25)\n        elif rho > 0.75 and hits_boundary:\n            trust_radius = torch.clamp(2*trust_radius, max=max_trust_radius)\n\n        # if the ratio is high enough then accept the proposed step\n        if rho > eta:\n            x = x_proposed\n            m = m_proposed\n        elif isinstance(sf, Minimizer):\n            # if we are using a Minimizer as our ScalarFunction then we\n            # need to re-compute the previous state because it was\n            # overwritten during the call `subproblem(x_proposed, closure)`\n            m = subproblem(x, closure)\n\n        # append the best guess, call back, increment the iteration count\n        if return_all:\n            allvecs.append(x.clone())\n        if callback is not None:\n            if callback(x.clone()):\n                warnflag = 5\n                msg = _status_message['callback_stop']\n                break\n        k += 1\n\n        # verbosity check\n        if disp > 1:\n            print('iter %d - fval: %0.4f' % (k, m.fun))\n\n        # check if the gradient is small enough to stop\n        if m.jac_mag < gtol:\n            warnflag = 0\n            break\n\n    # print some stuff if requested\n    if disp:\n        msg = status_messages[warnflag]\n        if warnflag != 0:\n            msg = 'Warning: ' + msg\n        print(msg)\n        print(\"         Current function value: %f\" % m.fun)\n        print(\"         Iterations: %d\" % k)\n        print(\"         Function evaluations: %d\" % sf.nfev)\n        # print(\"         Gradient evaluations: %d\" % sf.ngev)\n        # print(\"         Hessian evaluations: %d\" % (sf.nhev + nhessp[0]))\n\n    result = OptimizeResult(x=x.view_as(x0), fun=m.fun, grad=m.jac.view_as(x0),\n                            success=(warnflag == 0), status=warnflag,\n                            nfev=sf.nfev, nit=k, message=status_messages[warnflag])\n\n    if not subproblem.hess_prod:\n        result['hess'] = m.hess.view(2 * x0.shape)\n\n    if return_all:\n        result['allvecs'] = allvecs\n\n    return result"
  },
  {
    "path": "torchmin/trustregion/dogleg.py",
    "content": "\"\"\"\nDog-leg trust-region optimization.\n\nCode ported from SciPy to PyTorch\n\nCopyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers.\nAll rights reserved.\n\"\"\"\nimport torch\nfrom torch.linalg import norm\n\nfrom .base import _minimize_trust_region, BaseQuadraticSubproblem\n\n\ndef _minimize_dogleg(\n        fun, x0, **trust_region_options):\n    \"\"\"Minimization of scalar function of one or more variables using\n    the dog-leg trust-region algorithm.\n\n    .. warning::\n        The Hessian is required to be positive definite at all times;\n        otherwise this algorithm will fail.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize\n    x0 : Tensor\n        Initialization point\n    initial_trust_radius : float\n        Initial trust-region radius.\n    max_trust_radius : float\n        Maximum value of the trust-region radius. No steps that are longer\n        than this value will be proposed.\n    eta : float\n        Trust region related acceptance stringency for proposed steps.\n    gtol : float\n        Gradient norm must be less than `gtol` before successful\n        termination.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    References\n    ----------\n    .. [1] Jorge Nocedal and Stephen Wright,\n           Numerical Optimization, second edition,\n           Springer-Verlag, 2006, page 73.\n\n    \"\"\"\n    return _minimize_trust_region(fun, x0,\n                                  subproblem=DoglegSubproblem,\n                                  **trust_region_options)\n\n\nclass DoglegSubproblem(BaseQuadraticSubproblem):\n    \"\"\"Quadratic subproblem solved by the dogleg method\"\"\"\n    hess_prod = False\n\n    def cauchy_point(self):\n        \"\"\"\n        The Cauchy point is minimal along the direction of steepest descent.\n        \"\"\"\n        if self._cauchy_point is None:\n            g = self.jac\n            Bg = self.hessp(g)\n            self._cauchy_point = -(g.dot(g) / g.dot(Bg)) * g\n        return self._cauchy_point\n\n    def newton_point(self):\n        \"\"\"\n        The Newton point is a global minimum of the approximate function.\n        \"\"\"\n        if self._newton_point is None:\n            p = -torch.cholesky_solve(self.jac.view(-1,1),\n                                      torch.linalg.cholesky(self.hess))\n            self._newton_point = p.view(-1)\n        return self._newton_point\n\n    def solve(self, trust_radius):\n        \"\"\"Solve quadratic subproblem\"\"\"\n\n        # Compute the Newton point.\n        # This is the optimum for the quadratic model function.\n        # If it is inside the trust radius then return this point.\n        p_best = self.newton_point()\n        if norm(p_best) < trust_radius:\n            hits_boundary = False\n            return p_best, hits_boundary\n\n        # Compute the Cauchy point.\n        # This is the predicted optimum along the direction of steepest descent.\n        p_u = self.cauchy_point()\n\n        # If the Cauchy point is outside the trust region,\n        # then return the point where the path intersects the boundary.\n        p_u_norm = norm(p_u)\n        if p_u_norm >= trust_radius:\n            p_boundary = p_u * (trust_radius / p_u_norm)\n            hits_boundary = True\n            return p_boundary, hits_boundary\n\n        # Compute the intersection of the trust region boundary\n        # and the line segment connecting the Cauchy and Newton points.\n        # This requires solving a quadratic equation.\n        # ||p_u + t*(p_best - p_u)||**2 == trust_radius**2\n        # Solve this for positive time t using the quadratic formula.\n        _, tb = self.get_boundaries_intersections(p_u, p_best - p_u,\n                                                  trust_radius)\n        p_boundary = p_u + tb * (p_best - p_u)\n        hits_boundary = True\n        return p_boundary, hits_boundary"
  },
  {
    "path": "torchmin/trustregion/exact.py",
    "content": "\"\"\"\nNearly exact trust-region optimization subproblem.\n\nCode ported from SciPy to PyTorch\n\nCopyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers.\nAll rights reserved.\n\"\"\"\nfrom typing import Tuple\nfrom torch import Tensor\nimport torch\nfrom torch.linalg import norm\nfrom scipy.linalg import get_lapack_funcs\n\nfrom .base import _minimize_trust_region, BaseQuadraticSubproblem\n\n\ndef _minimize_trust_exact(fun, x0, **trust_region_options):\n    \"\"\"Minimization of scalar function of one or more variables using\n    a nearly exact trust-region algorithm.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    initial_tr_radius : float\n        Initial trust-region radius.\n    max_tr_radius : float\n        Maximum value of the trust-region radius. No steps that are longer\n        than this value will be proposed.\n    eta : float\n        Trust region related acceptance stringency for proposed steps.\n    gtol : float\n        Gradient norm must be less than ``gtol`` before successful\n        termination.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    Notes\n    -----\n    This trust-region solver was based on [1]_, [2]_ and [3]_,\n    which implement similar algorithms. The algorithm is basically\n    that of [1]_ but ideas from [2]_ and [3]_ were also used.\n\n    References\n    ----------\n    .. [1] A.R. Conn, N.I. Gould, and P.L. Toint, \"Trust region methods\",\n           Siam, pp. 169-200, 2000.\n    .. [2] J. Nocedal and  S. Wright, \"Numerical optimization\",\n           Springer Science & Business Media. pp. 83-91, 2006.\n    .. [3] J.J. More and D.C. Sorensen, \"Computing a trust region step\",\n           SIAM Journal on Scientific and Statistical Computing, vol. 4(3),\n           pp. 553-572, 1983.\n\n    \"\"\"\n    return _minimize_trust_region(fun, x0,\n                                  subproblem=IterativeSubproblem,\n                                  **trust_region_options)\n\n\ndef solve_triangular(A, b, *, upper=True, transpose=False, **kwargs):\n    extra_dim = 0 if transpose else 1\n    return torch.linalg.solve_triangular(\n        A,\n        b.unsqueeze(extra_dim),\n        upper=upper,\n        left=not transpose,\n        **kwargs\n    ).squeeze(extra_dim)\n\n\ndef solve_cholesky(A, b, **kwargs):\n    return torch.cholesky_solve(b.unsqueeze(1), A, **kwargs).squeeze(1)\n\n\n@torch.jit.script\ndef estimate_smallest_singular_value(U) -> Tuple[Tensor, Tensor]:\n    \"\"\"Given upper triangular matrix ``U`` estimate the smallest singular\n    value and the correspondent right singular vector in O(n**2) operations.\n\n    A vector `e` with components selected from {+1, -1}\n    is selected so that the solution `w` to the system\n    `U.T w = e` is as large as possible. Implementation\n    based on algorithm 3.5.1, p. 142, from reference [1]_\n    adapted for lower triangular matrix.\n\n    References\n    ----------\n    .. [1] G.H. Golub, C.F. Van Loan. \"Matrix computations\".\n           Forth Edition. JHU press. pp. 140-142.\n    \"\"\"\n\n    U = torch.atleast_2d(U)\n    UT = U.T\n    m, n = U.shape\n    if m != n:\n        raise ValueError(\"A square triangular matrix should be provided.\")\n\n    p = torch.zeros(n, dtype=U.dtype, device=U.device)\n    w = torch.empty(n, dtype=U.dtype, device=U.device)\n\n    for k in range(n):\n        wp = (1-p[k]) / UT[k, k]\n        wm = (-1-p[k]) / UT[k, k]\n        pp = p[k+1:] + UT[k+1:, k] * wp\n        pm = p[k+1:] + UT[k+1:, k] * wm\n\n        if wp.abs() + norm(pp, 1) >= wm.abs() + norm(pm, 1):\n            w[k] = wp\n            p[k+1:] = pp\n        else:\n            w[k] = wm\n            p[k+1:] = pm\n\n    # The system `U v = w` is solved using backward substitution.\n    v = torch.triangular_solve(w.view(-1,1), U)[0].view(-1)\n    v_norm = norm(v)\n\n    s_min = norm(w) / v_norm  # Smallest singular value\n    z_min = v / v_norm        # Associated vector\n\n    return s_min, z_min\n\n\ndef gershgorin_bounds(H):\n    \"\"\"\n    Given a square matrix ``H`` compute upper\n    and lower bounds for its eigenvalues (Gregoshgorin Bounds).\n    \"\"\"\n    H_diag = torch.diag(H)\n    H_diag_abs = H_diag.abs()\n    H_row_sums = H.abs().sum(dim=1)\n    lb = torch.min(H_diag + H_diag_abs - H_row_sums)\n    ub = torch.max(H_diag - H_diag_abs + H_row_sums)\n\n    return lb, ub\n\n\ndef singular_leading_submatrix(A, U, k):\n    \"\"\"\n    Compute term that makes the leading ``k`` by ``k``\n    submatrix from ``A`` singular.\n    \"\"\"\n    u = U[:k-1, k-1]\n\n    # Compute delta\n    delta = u.dot(u) - A[k-1, k-1]\n\n    # Initialize v\n    v = A.new_zeros(A.shape[0])\n    v[k-1] = 1\n\n    # Compute the remaining values of v by solving a triangular system.\n    if k != 1:\n        v[:k-1] = solve_triangular(U[:k-1, :k-1], -u)\n\n    return delta, v\n\n\nclass IterativeSubproblem(BaseQuadraticSubproblem):\n    \"\"\"Quadratic subproblem solved by nearly exact iterative method.\"\"\"\n\n    # UPDATE_COEFF appears in reference [1]_\n    # in formula 7.3.14 (p. 190) named as \"theta\".\n    # As recommended there it value is fixed in 0.01.\n    UPDATE_COEFF = 0.01\n    hess_prod = False\n\n    def __init__(self, x, fun, k_easy=0.1, k_hard=0.2):\n\n        super().__init__(x, fun)\n\n        # When the trust-region shrinks in two consecutive\n        # calculations (``tr_radius < previous_tr_radius``)\n        # the lower bound ``lambda_lb`` may be reused,\n        # facilitating  the convergence. To indicate no\n        # previous value is known at first ``previous_tr_radius``\n        # is set to -1  and ``lambda_lb`` to None.\n        self.previous_tr_radius = -1\n        self.lambda_lb = None\n\n        self.niter = 0\n        self.EPS = torch.finfo(x.dtype).eps\n\n        # ``k_easy`` and ``k_hard`` are parameters used\n        # to determine the stop criteria to the iterative\n        # subproblem solver. Take a look at pp. 194-197\n        # from reference _[1] for a more detailed description.\n        self.k_easy = k_easy\n        self.k_hard = k_hard\n\n        # Get Lapack function for cholesky decomposition.\n        # NOTE: cholesky_ex requires pytorch >= 1.9.0\n        if 'cholesky_ex' in dir(torch.linalg):\n            self.torch_cholesky = True\n        else:\n            # if we don't have torch cholesky, use potrf from scipy\n            self.cholesky, = get_lapack_funcs(('potrf',),\n                                              (self.hess.cpu().numpy(),))\n            self.torch_cholesky = False\n\n        # Get info about Hessian\n        self.dimension = len(self.hess)\n        self.hess_gershgorin_lb, self.hess_gershgorin_ub = gershgorin_bounds(self.hess)\n        self.hess_inf = norm(self.hess, float('inf'))\n        self.hess_fro = norm(self.hess, 'fro')\n\n        # A constant such that for vectors smaler than that\n        # backward substituition is not reliable. It was stabilished\n        # based on Golub, G. H., Van Loan, C. F. (2013).\n        # \"Matrix computations\". Forth Edition. JHU press., p.165.\n        self.CLOSE_TO_ZERO = self.dimension * self.EPS * self.hess_inf\n\n    def _initial_values(self, tr_radius):\n        \"\"\"Given a trust radius, return a good initial guess for\n        the damping factor, the lower bound and the upper bound.\n        The values were chosen accordingly to the guidelines on\n        section 7.3.8 (p. 192) from [1]_.\n        \"\"\"\n        hess_norm = torch.min(self.hess_fro, self.hess_inf)\n\n        # Upper bound for the damping factor\n        lambda_ub = self.jac_mag / tr_radius + torch.min(-self.hess_gershgorin_lb, hess_norm)\n        lambda_ub = torch.clamp(lambda_ub, min=0)\n\n        # Lower bound for the damping factor\n        lambda_lb = self.jac_mag / tr_radius - torch.min(self.hess_gershgorin_ub, hess_norm)\n        lambda_lb = torch.max(lambda_lb, -self.hess.diagonal().min())\n        lambda_lb = torch.clamp(lambda_lb, min=0)\n\n        # Improve bounds with previous info\n        if tr_radius < self.previous_tr_radius:\n            lambda_lb = torch.max(self.lambda_lb, lambda_lb)\n\n        # Initial guess for the damping factor\n        if lambda_lb == 0:\n            lambda_initial = lambda_lb.clone()\n        else:\n            lambda_initial = torch.max(\n                torch.sqrt(lambda_lb * lambda_ub),\n                lambda_lb + self.UPDATE_COEFF*(lambda_ub-lambda_lb))\n\n        return lambda_initial, lambda_lb, lambda_ub\n\n    def solve(self, tr_radius):\n        \"\"\"Solve quadratic subproblem\"\"\"\n\n        lambda_current, lambda_lb, lambda_ub = self._initial_values(tr_radius)\n        n = self.dimension\n        hits_boundary = True\n        already_factorized = False\n        self.niter = 0\n\n        while True:\n            # Compute Cholesky factorization\n            if already_factorized:\n                already_factorized = False\n            else:\n                H = self.hess.clone()\n                H.diagonal().add_(lambda_current)\n                if self.torch_cholesky:\n                    U, info = torch.linalg.cholesky_ex(H)\n                    U = U.t().contiguous()\n                else:\n                    U, info = self.cholesky(H.cpu().numpy(),\n                                            lower=False,\n                                            overwrite_a=False,\n                                            clean=True)\n                    U = H.new_tensor(U)\n\n            self.niter += 1\n\n            # Check if factorization succeeded\n            if info == 0 and self.jac_mag > self.CLOSE_TO_ZERO:\n                # Successful factorization\n\n                # Solve `U.T U p = s`\n                p = solve_cholesky(U, -self.jac, upper=True)\n                p_norm = norm(p)\n\n                # Check for interior convergence\n                if p_norm <= tr_radius and lambda_current == 0:\n                    hits_boundary = False\n                    break\n\n                # Solve `U.T w = p`\n                w = solve_triangular(U, p, transpose=True)\n                w_norm = norm(w)\n\n                # Compute Newton step accordingly to\n                # formula (4.44) p.87 from ref [2]_.\n                delta_lambda = (p_norm/w_norm)**2 * (p_norm-tr_radius)/tr_radius\n                lambda_new = lambda_current + delta_lambda\n\n                if p_norm < tr_radius:  # Inside boundary\n                    s_min, z_min = estimate_smallest_singular_value(U)\n\n                    ta, tb = self.get_boundaries_intersections(p, z_min, tr_radius)\n\n                    # Choose `step_len` with the smallest magnitude.\n                    # The reason for this choice is explained at\n                    # ref [3]_, p. 6 (Immediately before the formula\n                    # for `tau`).\n                    step_len = min(ta, tb, key=torch.abs)\n\n                    # Compute the quadratic term  (p.T*H*p)\n                    quadratic_term = p.dot(H.mv(p))\n\n                    # Check stop criteria\n                    relative_error = ((step_len**2 * s_min**2) /\n                                      (quadratic_term + lambda_current*tr_radius**2))\n                    if relative_error <= self.k_hard:\n                        p.add_(step_len * z_min)\n                        break\n\n                    # Update uncertanty bounds\n                    lambda_ub = lambda_current\n                    lambda_lb = torch.max(lambda_lb, lambda_current - s_min**2)\n\n                    # Compute Cholesky factorization\n                    H = self.hess.clone()\n                    H.diagonal().add_(lambda_new)\n                    if self.torch_cholesky:\n                        _, info = torch.linalg.cholesky_ex(H)\n                    else:\n                        _, info = self.cholesky(H.cpu().numpy(),\n                                                lower=False,\n                                                overwrite_a=False,\n                                                clean=True)\n\n                    if info == 0:\n                        lambda_current = lambda_new\n                        already_factorized = True\n                    else:\n                        lambda_lb = torch.max(lambda_lb, lambda_new)\n                        lambda_current = torch.max(\n                            torch.sqrt(lambda_lb * lambda_ub),\n                            lambda_lb + self.UPDATE_COEFF*(lambda_ub-lambda_lb))\n\n                else:  # Outside boundary\n                    # Check stop criteria\n                    relative_error = torch.abs(p_norm - tr_radius) / tr_radius\n                    if relative_error <= self.k_easy:\n                        break\n\n                    # Update uncertanty bounds\n                    lambda_lb = lambda_current\n\n                    # Update damping factor\n                    lambda_current = lambda_new\n\n            elif info == 0 and self.jac_mag <= self.CLOSE_TO_ZERO:\n                # jac_mag very close to zero\n\n                # Check for interior convergence\n                if lambda_current == 0:\n                    p = self.jac.new_zeros(n)\n                    hits_boundary = False\n                    break\n\n                s_min, z_min = estimate_smallest_singular_value(U)\n                step_len = tr_radius\n\n                # Check stop criteria\n                if step_len**2 * s_min**2 <= self.k_hard * lambda_current * tr_radius**2:\n                    p = step_len * z_min\n                    break\n\n                # Update uncertainty bounds and dampening factor\n                lambda_ub = lambda_current\n                lambda_lb = torch.max(lambda_lb, lambda_current - s_min**2)\n                lambda_current = torch.max(\n                    torch.sqrt(lambda_lb * lambda_ub),\n                    lambda_lb + self.UPDATE_COEFF*(lambda_ub-lambda_lb))\n\n            else:\n                # Unsuccessful factorization\n\n                delta, v = singular_leading_submatrix(H, U, info)\n                v_norm = norm(v)\n\n                lambda_lb = torch.max(lambda_lb, lambda_current + delta/v_norm**2)\n\n                # Update damping factor\n                lambda_current = torch.max(\n                    torch.sqrt(lambda_lb * lambda_ub),\n                    lambda_lb + self.UPDATE_COEFF*(lambda_ub-lambda_lb))\n\n        self.lambda_lb = lambda_lb\n        self.lambda_current = lambda_current\n        self.previous_tr_radius = tr_radius\n\n        return p, hits_boundary\n"
  },
  {
    "path": "torchmin/trustregion/krylov.py",
    "content": "\"\"\"\nTODO: this module is not yet complete. It is not ready for use.\n\"\"\"\nimport numpy as np\nfrom scipy.linalg import eigh_tridiagonal, get_lapack_funcs\nimport torch\n\nfrom .base import _minimize_trust_region, BaseQuadraticSubproblem\n\n\ndef _minimize_trust_krylov(fun, x0, **trust_region_options):\n    \"\"\"Minimization of scalar function of one or more variables using\n    the GLTR Krylov subspace trust-region algorithm.\n\n    .. warning::\n        This minimizer is in early stages and has not been rigorously\n        tested. It may change in the near future.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    initial_tr_radius : float\n        Initial trust-region radius.\n    max_tr_radius : float\n        Maximum value of the trust-region radius. No steps that are longer\n        than this value will be proposed.\n    eta : float\n        Trust region related acceptance stringency for proposed steps.\n    gtol : float\n        Gradient norm must be less than ``gtol`` before successful\n        termination.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    Notes\n    -----\n    This trust-region solver is based on the GLTR algorithm as\n    described in [1]_ and [2]_.\n\n    References\n    ----------\n    .. [1] F. Lenders, C. Kirches, and A. Potschka, \"trlib: A vector-free\n           implementation of the GLTR method for...\",\n           arXiv:1611.04718.\n    .. [2] N. Gould, S. Lucidi, M. Roma, P. Toint: “Solving the Trust-Region\n           Subproblem using the Lanczos Method”,\n           SIAM J. Optim., 9(2), 504–525, 1999.\n    .. [3] J. Nocedal and  S. Wright, \"Numerical optimization\",\n           Springer Science & Business Media. pp. 83-91, 2006.\n\n    \"\"\"\n    return _minimize_trust_region(fun, x0,\n                                  subproblem=KrylovSubproblem,\n                                  **trust_region_options)\n\n\nclass KrylovSubproblem(BaseQuadraticSubproblem):\n    \"\"\"The GLTR trust region sub-problem defined on an expanding\n    Krylov subspace.\n\n    Based on the implementation of GLTR described in [1]_.\n\n    References\n    ----------\n    .. [1] F. Lenders, C. Kirches, and A. Potschka, \"trlib: A vector-free\n           implementation of the GLTR method for...\",\n           arXiv:1611.04718.\n    .. [2] N. Gould, S. Lucidi, M. Roma, P. Toint: “Solving the Trust-Region\n           Subproblem using the Lanczos Method”,\n           SIAM J. Optim., 9(2), 504–525, 1999.\n    .. [3] J. Nocedal and  S. Wright, \"Numerical optimization\",\n           Springer Science & Business Media. pp. 83-91, 2006.\n    \"\"\"\n    hess_prod = True\n    max_lanczos = None\n    max_ms_iters = 500  # max iterations of the Moré-Sorensen loop\n\n    def __init__(self, x, fun, k_easy=0.1, k_hard=0.2, tol=1e-5, ortho=True,\n                 debug=False):\n        super().__init__(x, fun)\n        self.eps = torch.finfo(x.dtype).eps\n        self.k_easy = k_easy\n        self.k_hard = k_hard\n        self.tol = tol\n        self.ortho = ortho\n        self._debug = debug\n\n    def tridiag_subproblem(self, Ta, Tb, tr_radius):\n        \"\"\"Solve the GLTR tridiagonal subproblem.\n\n        Based on Algorithm 5.2 of [2]_. We factorize as follows:\n\n        .. math::\n            T + lambd * I = LDL^T\n\n        Where `D` is diagonal and `L` unit (lower) bi-diagonal.\n        \"\"\"\n        device, dtype = Ta.device, Ta.dtype\n\n        # convert to numpy\n        Ta = Ta.cpu().numpy()\n        Tb = Tb.cpu().numpy()\n        tr_radius = float(tr_radius)\n\n        # right hand side\n        rhs = np.zeros_like(Ta)\n        rhs[0] = - float(self.jac_mag)\n\n        # get LAPACK routines for factorizing and solving sym-PD tridiagonal\n        ptsv, pttrs = get_lapack_funcs(('ptsv', 'pttrs'), (Ta, Tb, rhs))\n\n        eig0 = None\n        lambd_lb = 0.\n        lambd = 0.\n        for _ in range(self.max_ms_iters):\n            lambd = max(lambd, lambd_lb)\n\n            # factor T + lambd * I = LDL^T and solve LDL^T p = rhs\n            d, e, p, info = ptsv(Ta + lambd, Tb, rhs)\n            assert info >= 0  # sanity check\n            if info > 0:\n                assert eig0 is None  # sanity check; should only happen once\n                # estimate smallest eigenvalue and continue\n                eig0 = eigh_tridiagonal(\n                    Ta, Tb, eigvals_only=True, select='i',\n                    select_range=(0,0), lapack_driver='stebz').item()\n                lambd_lb = max(1e-3 - eig0, 0)\n                continue\n\n            p_norm = np.linalg.norm(p)\n            if p_norm < tr_radius:\n                # TODO: add extra checks\n                status = 0\n                break\n            elif abs(p_norm - tr_radius) / tr_radius <= self.k_easy:\n                status = 1\n                break\n\n            # solve LDL^T q = p and compute <q, p>\n            v, info = pttrs(d, e, p)\n            q_norm2 = v.dot(p)\n\n            # update lambd\n            lambd += (p_norm**2 / q_norm2) * (p_norm - tr_radius) / tr_radius\n        else:\n            status = -1\n\n        p = torch.tensor(p, device=device, dtype=dtype)\n\n        return p, status, lambd\n\n    def solve(self, tr_radius):\n        g = self.jac\n        gamma_0 = self.jac_mag\n        n, = g.shape\n        m = n if self.max_lanczos is None else min(n, self.max_lanczos)\n        dtype = g.dtype\n        device = g.device\n        h_best = None\n        error_best = float('inf')\n\n        # Lanczos Q matrix buffer\n        Q = torch.zeros(m, n, dtype=dtype, device=device)\n        Q[0] = g / gamma_0\n\n        # Lanczos T matrix buffers\n        # a and b are the diagonal and off-diagonal entries of T, respectively\n        a = torch.zeros(m, dtype=dtype, device=device)\n        b = torch.zeros(m, dtype=dtype, device=device)\n\n        # first lanczos iteration\n        r = self.hessp(Q[0])\n        torch.dot(Q[0], r, out=a[0])\n        r.sub_(Q[0] * a[0])\n        torch.linalg.norm(r, out=b[0])\n        if b[0] < self.eps:\n            raise RuntimeError('initial beta is zero.')\n\n        # remaining iterations\n        for i in range(1, m):\n            torch.div(r, b[i-1], out=Q[i])\n            r = self.hessp(Q[i])\n            r.sub_(Q[i-1] * b[i-1])\n            torch.dot(Q[i], r, out=a[i])\n            r.sub_(Q[i] * a[i])\n            if self.ortho:\n                # Re-orthogonalize with Gram-Schmidt\n                r.addmv_(Q[:i+1].T, Q[:i+1].mv(r), alpha=-1)\n            torch.linalg.norm(r, out=b[i])\n            if b[i] < self.eps:\n                # This should never occur when self.ortho=True\n                raise RuntimeError('reducible T matrix encountered.')\n\n            # GLTR sub-problem\n            h, status, lambd = self.tridiag_subproblem(a[:i+1], b[:i], tr_radius)\n\n            if status >= 0:\n                # convergence check; see Algorithm 1 of [1]_ and\n                # Algorithm 5.1 of [2]_. Equivalent to the following:\n                #     p = Q[:i+1].T.mv(h)\n                #     error = torch.linalg.norm(self.hessp(p) + lambd * p + g)\n                error = b[i] * h[-1].abs()\n                if self._debug:\n                    print('iter %3d - status: %d - lambd: %0.4e - error: %0.4e'\n                          % (i+1, status, lambd, error))\n                if error < error_best:\n                    # we've found a new best\n                    hits_boundary = status != 0\n                    h_best = h\n                    error_best = error\n                    if error_best <= self.tol:\n                        break\n\n            elif self._debug:\n                print('iter %3d - status: %d - lambd: %0.4e' %\n                      (i+1, status, lambd))\n\n        if h_best is None:\n            # TODO: what should we do here?\n            raise RuntimeError('gltr solution not found')\n\n        # project h back to R^n\n        p_best = Q[:i+1].T.mv(h_best)\n\n        return p_best, hits_boundary\n"
  },
  {
    "path": "torchmin/trustregion/ncg.py",
    "content": "\"\"\"\nNewton-CG trust-region optimization.\n\nCode ported from SciPy to PyTorch\n\nCopyright (c) 2001-2002 Enthought, Inc.  2003-2019, SciPy Developers.\nAll rights reserved.\n\"\"\"\nimport torch\nfrom torch.linalg import norm\n\nfrom .base import _minimize_trust_region, BaseQuadraticSubproblem\n\n\ndef _minimize_trust_ncg(\n        fun, x0, **trust_region_options):\n    \"\"\"Minimization of scalar function of one or more variables using\n    the Newton conjugate gradient trust-region algorithm.\n\n    Parameters\n    ----------\n    fun : callable\n        Scalar objective function to minimize.\n    x0 : Tensor\n        Initialization point.\n    initial_trust_radius : float\n        Initial trust-region radius.\n    max_trust_radius : float\n        Maximum value of the trust-region radius. No steps that are longer\n        than this value will be proposed.\n    eta : float\n        Trust region related acceptance stringency for proposed steps.\n    gtol : float\n        Gradient norm must be less than ``gtol`` before successful\n        termination.\n\n    Returns\n    -------\n    result : OptimizeResult\n        Result of the optimization routine.\n\n    Notes\n    -----\n    This is algorithm (7.2) of Nocedal and Wright 2nd edition.\n    Only the function that computes the Hessian-vector product is required.\n    The Hessian itself is not required, and the Hessian does\n    not need to be positive semidefinite.\n\n    \"\"\"\n    return _minimize_trust_region(fun, x0,\n                                  subproblem=CGSteihaugSubproblem,\n                                  **trust_region_options)\n\n\nclass CGSteihaugSubproblem(BaseQuadraticSubproblem):\n    \"\"\"Quadratic subproblem solved by a conjugate gradient method\"\"\"\n    hess_prod = True\n\n    def solve(self, trust_radius):\n        \"\"\"Solve the subproblem using a conjugate gradient method.\n\n        Parameters\n        ----------\n        trust_radius : float\n            We are allowed to wander only this far away from the origin.\n\n        Returns\n        -------\n        p : Tensor\n            The proposed step.\n        hits_boundary : bool\n            True if the proposed step is on the boundary of the trust region.\n\n        \"\"\"\n\n        # get the norm of jacobian and define the origin\n        p_origin = torch.zeros_like(self.jac)\n\n        # define a default tolerance\n        tolerance = self.jac_mag * self.jac_mag.sqrt().clamp(max=0.5)\n\n        # Stop the method if the search direction\n        # is a direction of nonpositive curvature.\n        if self.jac_mag < tolerance:\n            hits_boundary = False\n            return p_origin, hits_boundary\n\n        # init the state for the first iteration\n        z = p_origin\n        r = self.jac\n        d = -r\n\n        # Search for the min of the approximation of the objective function.\n        while True:\n\n            # do an iteration\n            Bd = self.hessp(d)\n            dBd = d.dot(Bd)\n            if dBd <= 0:\n                # Look at the two boundary points.\n                # Find both values of t to get the boundary points such that\n                # ||z + t d|| == trust_radius\n                # and then choose the one with the predicted min value.\n                ta, tb = self.get_boundaries_intersections(z, d, trust_radius)\n                pa = z + ta * d\n                pb = z + tb * d\n                p_boundary = torch.where(self(pa).lt(self(pb)), pa, pb)\n                hits_boundary = True\n                return p_boundary, hits_boundary\n            r_squared = r.dot(r)\n            alpha = r_squared / dBd\n            z_next = z + alpha * d\n            if norm(z_next) >= trust_radius:\n                # Find t >= 0 to get the boundary point such that\n                # ||z + t d|| == trust_radius\n                ta, tb = self.get_boundaries_intersections(z, d, trust_radius)\n                p_boundary = z + tb * d\n                hits_boundary = True\n                return p_boundary, hits_boundary\n            r_next = r + alpha * Bd\n            r_next_squared = r_next.dot(r_next)\n            if r_next_squared.sqrt() < tolerance:\n                hits_boundary = False\n                return z_next, hits_boundary\n            beta_next = r_next_squared / r_squared\n            d_next = -r_next + beta_next * d\n\n            # update the state for the next iteration\n            z = z_next\n            r = r_next\n            d = d_next"
  }
]