Showing preview only (3,706K chars total). Download the full file or copy to clipboard to get everything.
Repository: bayesian-optimization/BayesianOptimization
Branch: master
Commit: 6868d3a03df1
Files: 66
Total size: 3.5 MB
Directory structure:
gitextract_x9fosayw/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ └── workflows/
│ ├── build_docs.yml
│ ├── format_and_lint.yml
│ ├── python-publish.yml
│ └── run_tests.yml
├── .gitignore
├── .pre-commit-config.yaml
├── LICENSE
├── README.md
├── bayes_opt/
│ ├── __init__.py
│ ├── acquisition.py
│ ├── bayesian_optimization.py
│ ├── constraint.py
│ ├── domain_reduction.py
│ ├── exception.py
│ ├── logger.py
│ ├── parameter.py
│ ├── py.typed
│ ├── target_space.py
│ └── util.py
├── docsrc/
│ ├── Makefile
│ ├── conf.py
│ ├── index.rst
│ ├── make.bat
│ ├── reference/
│ │ ├── acquisition/
│ │ │ ├── ConstantLiar.rst
│ │ │ ├── ExpectedImprovement.rst
│ │ │ ├── GPHedge.rst
│ │ │ ├── ProbabilityOfImprovement.rst
│ │ │ └── UpperConfidenceBound.rst
│ │ ├── acquisition.rst
│ │ ├── bayes_opt.rst
│ │ ├── constraint.rst
│ │ ├── domain_reduction.rst
│ │ ├── exception.rst
│ │ ├── other.rst
│ │ ├── parameter.rst
│ │ └── target_space.rst
│ └── requirements.txt
├── examples/
│ ├── acquisition_functions.ipynb
│ ├── advanced-tour.ipynb
│ ├── async_optimization.py
│ ├── async_optimization_dummies.py
│ ├── basic-tour.ipynb
│ ├── constraints.ipynb
│ ├── domain_reduction.ipynb
│ ├── duplicate_point.py
│ ├── exploitation_vs_exploration.ipynb
│ ├── parameter_types.ipynb
│ ├── sklearn_example.py
│ ├── typed_hyperparameter_tuning.py
│ └── visualization.ipynb
├── pyproject.toml
├── ruff.toml
├── scripts/
│ ├── check.sh
│ ├── check_precommit.sh
│ └── format.sh
└── tests/
├── test_acquisition.py
├── test_bayesian_optimization.py
├── test_constraint.py
├── test_logger.py
├── test_notebooks_run.py
├── test_parameter.py
├── test_seq_domain_red.py
├── test_target_space.py
└── test_util.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.md
================================================
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: bug, enhancement
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
Ex: Using `scipy==1.8` with `bayesian-optimization==1.2.0` results in `TypeError: 'float' object is not subscriptable`.
**To Reproduce**
A concise, self-contained code snippet that reproduces the bug you would like to report.
Ex:
```python
from bayes_opt import BayesianOptimization
black_box_function = lambda x, y: -x ** 2 - (y - 1) ** 2 + 1
pbounds = {'x': (2, 4), 'y': (-3, 3)}
optimizer = BayesianOptimization(
f=black_box_function,
pbounds=pbounds
)
optimizer.maximize()
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Environment (please complete the following information):**
- OS: [e.g. Arch Linux, macOS, Windows]
- `python` Version [e.g. 3.8.9]
- `numpy` Version [e.g. 1.21.6]
- `scipy` Version [e.g. 1.8.0]
- `bayesian-optimization` Version [e.g. 1.2.0]
**Additional context**
Add any other context about the problem here.
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md
================================================
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: enhancement
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**References or alternative approaches**
If this feature was described in literature, please add references here. Additionally, feel free to add descriptions of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
**Are you able and willing to implement this feature yourself and open a pull request?**
- [ ] Yes, I can provide this feature.
================================================
FILE: .github/workflows/build_docs.yml
================================================
name: docs
on:
release:
types: [published]
push:
branches:
- master
pull_request:
concurrency:
group: ${{ github.workflow }}
jobs:
build-docs-and-publish:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v3
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
python-version: "3.10"
- name: Get tag
uses: olegtarasov/get-tag@v2.1
- name: Install pandoc
run: sudo apt-get install -y pandoc
- name: Install package and test dependencies
run: uv sync --extra dev
- name: build sphinx docs
run: |
cd docsrc
uv run make github
- name: Determine directory to publish docs to
id: docs-publish-dir
uses: jannekem/run-python-script-action@v1
with:
script: |
import os, re
github_ref = os.environ.get('GITHUB_REF')
m = re.match(r'^refs/tags/v([0-9]+\.[0-9]+\.[0-9]+(-dev\.[0-9]+)?)$',
github_ref)
if m:
target = m.group(1)
elif github_ref == 'refs/heads/master':
target = 'master'
else:
target = ''
set_output('target', target)
- name: Deploy
uses: peaceiris/actions-gh-pages@v3
if: steps.docs-publish-dir.outputs.target != ''
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: ./docs/html
destination_dir: ${{ steps.docs-publish-dir.outputs.target }}
keep_files: false
outputs:
docs-target: ${{ steps.docs-publish-dir.outputs.target }}
update-versions:
name: Update docs versions JSON
needs: build-docs-and-publish
if: needs.build-docs-and-publish.outputs.docs-target != ''
runs-on: Ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v3
with:
ref: gh-pages
- name: Write versions to JSON file
uses: jannekem/run-python-script-action@v1
with:
script: |
import json
import re
# dependency of sphinx, so should be installed
from packaging import version as version_
from pathlib import Path
cwd = Path.cwd()
versions = sorted((item.name for item in cwd.iterdir()
if item.is_dir() and not item.name.startswith('.')),
reverse=True)
# Filter out master and dev versions
parseable_versions = []
for version in versions:
try:
version_.parse(version)
except version_.InvalidVersion:
continue
parseable_versions.append(version)
if parseable_versions:
max_version = max(parseable_versions, key=version_.parse)
else:
max_version = None
target_dir = Path('gh-pages')
target_dir.mkdir(parents=True)
versions = [
dict(
version=version,
title=version + ' (stable)' if version == max_version else version,
aliases=['stable'] if version == max_version else [],
) for version in versions
]
target_file = target_dir / 'versions.json'
with target_file.open('w') as f:
json.dump(versions, f)
- name: Publish versions JSON to GitHub pages
uses: peaceiris/actions-gh-pages@v3
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
publish_dir: gh-pages
keep_files: true
================================================
FILE: .github/workflows/format_and_lint.yml
================================================
name: Code format and lint
on:
push:
branches: [ "master" ]
pull_request:
permissions:
contents: read
jobs:
check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install uv
uses: astral-sh/setup-uv@v6
with:
python-version: "3.9"
- name: Install dependencies
run: uv sync --extra dev
- name: Run pre-commit
run: uv run pre-commit run --all-files --show-diff-on-failure --color=always
================================================
FILE: .github/workflows/python-publish.yml
================================================
# This workflow will upload a Python Package using uv when a release is created
# Note that you must manually update the version number in pyproject.toml before attempting this.
name: Upload Python Package
on:
release:
types: [published]
permissions:
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v6
- run: uv build
- name: Publish to pypi
env:
PYPI_API_TOKEN: ${{ secrets.PYPI_API_TOKEN }}
run: uv publish --token "$PYPI_API_TOKEN"
================================================
FILE: .github/workflows/run_tests.yml
================================================
# This workflow will install Python dependencies and run tests in multiple versions of Python
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
name: tests
on:
push:
branches: [ "master" ]
pull_request:
permissions:
contents: read
jobs:
build:
name: Python ${{ matrix.python-version }} - numpy ${{ matrix.numpy-version }}
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
numpy-version: [">=1.25,<2", ">=2"]
exclude:
- python-version: "3.13" # numpy<2 is not supported on 3.13+
numpy-version: ">=1.25,<2"
- python-version: "3.14"
numpy-version: ">=1.25,<2"
steps:
- uses: actions/checkout@v3
- name: Install uv and set the python version
uses: astral-sh/setup-uv@v6
with:
python-version: ${{ matrix.python-version }}
- name: Install test dependencies
run: |
uv sync --extra dev
uv pip install "numpy${{ matrix.numpy-version}}"
- name: Run pytest
run: uv run --no-sync pytest --cov-report xml --cov=bayes_opt/
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
================================================
FILE: .gitignore
================================================
.ipynb_checkpoints
*.pyc
*.egg-info/
build/
dist/
scratch/
.idea/
.DS_Store
bo_eg*.png
gif/
# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
*temp*
docs/*
docsrc/.ipynb_checkpoints/*
docsrc/*.ipynb
docsrc/static/*
docsrc/README.md
poetry.lock
uv.lock
# Add log files and optimizer state files to gitignore
examples/logs.log
examples/optimizer_state.json
================================================
FILE: .pre-commit-config.yaml
================================================
repos:
- hooks:
- id: ruff
name: ruff-lint
- id: ruff-format
name: ruff-format
args: [--check]
repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.12.3
================================================
FILE: LICENSE
================================================
The MIT License (MIT)
Copyright (c) 2014 Fernando M. F. Nogueira
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
================================================
FILE: README.md
================================================
<div align="center">
<img src="https://raw.githubusercontent.com/bayesian-optimization/BayesianOptimization/master/docsrc/static/func.png"><br><br>
</div>
# Bayesian Optimization
[](https://bayesian-optimization.github.io/BayesianOptimization/index.html)
[](https://codecov.io/github/bayesian-optimization/BayesianOptimization?branch=master)
[](https://pypi.python.org/pypi/bayesian-optimization)
Pure Python implementation of bayesian global optimization with gaussian
processes.
This is a constrained global optimization package built upon bayesian inference
and gaussian processes, that attempts to find the maximum value of an unknown
function in as few iterations as possible. This technique is particularly
suited for optimization of high cost functions and situations where the balance
between exploration and exploitation is important.
## Installation
* pip (via PyPI):
```console
$ pip install bayesian-optimization
```
* Conda (via conda-forge):
```console
$ conda install -c conda-forge bayesian-optimization
```
## How does it work?
See the [documentation](https://bayesian-optimization.github.io/BayesianOptimization/) for how to use this package.
Bayesian optimization works by constructing a posterior distribution of functions (gaussian process) that best describes the function you want to optimize. As the number of observations grows, the posterior distribution improves, and the algorithm becomes more certain of which regions in parameter space are worth exploring and which are not, as seen in the picture below.
As you iterate over and over, the algorithm balances its needs of exploration and exploitation taking into account what it knows about the target function. At each step a Gaussian Process is fitted to the known samples (points previously explored), and the posterior distribution, combined with a exploration strategy (such as UCB (Upper Confidence Bound), or EI (Expected Improvement)), are used to determine the next point that should be explored (see the gif below).
This process is designed to minimize the number of steps required to find a combination of parameters that are close to the optimal combination. To do so, this method uses a proxy optimization problem (finding the maximum of the acquisition function) that, albeit still a hard problem, is cheaper (in the computational sense) and common tools can be employed. Therefore Bayesian Optimization is most adequate for situations where sampling the function to be optimized is a very expensive endeavor. See the references for a proper discussion of this method.
This project is under active development. If you run into trouble, find a bug or notice
anything that needs correction, please let us know by filing an issue.
## Basic tour of the Bayesian Optimization package
### 1. Specifying the function to be optimized
This is a function optimization package, therefore the first and most important ingredient is, of course, the function to be optimized.
**DISCLAIMER:** We know exactly how the output of the function below depends on its parameter. Obviously this is just an example, and you shouldn't expect to know it in a real scenario. However, it should be clear that you don't need to. All you need in order to use this package (and more generally, this technique) is a function `f` that takes a known set of parameters and outputs a real number.
```python
def black_box_function(x, y):
"""Function with unknown internals we wish to maximize.
This is just serving as an example, for all intents and
purposes think of the internals of this function, i.e.: the process
which generates its output values, as unknown.
"""
return -x ** 2 - (y - 1) ** 2 + 1
```
### 2. Getting Started
All we need to get started is to instantiate a `BayesianOptimization` object specifying a function to be optimized `f`, and its parameters with their corresponding bounds, `pbounds`. This is a constrained optimization technique, so you must specify the minimum and maximum values that can be probed for each parameter in order for it to work
```python
from bayes_opt import BayesianOptimization
# Bounded region of parameter space
pbounds = {'x': (2, 4), 'y': (-3, 3)}
optimizer = BayesianOptimization(
f=black_box_function,
pbounds=pbounds,
random_state=1,
)
```
The BayesianOptimization object will work out of the box without much tuning needed. The main method you should be aware of is `maximize`, which does exactly what you think it does.
There are many parameters you can pass to maximize, nonetheless, the most important ones are:
- `n_iter`: How many steps of bayesian optimization you want to perform. The more steps the more likely to find a good maximum you are.
- `init_points`: How many steps of **random** exploration you want to perform. Random exploration can help by diversifying the exploration space.
```python
optimizer.maximize(
init_points=2,
n_iter=3,
)
```
| iter | target | x | y |
-------------------------------------------------
| 1 | -7.135 | 2.834 | 1.322 |
| 2 | -7.78 | 2.0 | -1.186 |
| 3 | -19.0 | 4.0 | 3.0 |
| 4 | -16.3 | 2.378 | -2.413 |
| 5 | -4.441 | 2.105 | -0.005822 |
=================================================
The best combination of parameters and target value found can be accessed via the property `optimizer.max`.
```python
print(optimizer.max)
>>> {'target': -4.441293113411222, 'params': {'y': -0.005822117636089974, 'x': 2.104665051994087}}
```
While the list of all parameters probed and their corresponding target values is available via the property `optimizer.res`.
```python
for i, res in enumerate(optimizer.res):
print("Iteration {}: \n\t{}".format(i, res))
>>> Iteration 0:
>>> {'target': -7.135455292718879, 'params': {'y': 1.3219469606529488, 'x': 2.8340440094051482}}
>>> Iteration 1:
>>> {'target': -7.779531005607566, 'params': {'y': -1.1860045642089614, 'x': 2.0002287496346898}}
>>> Iteration 2:
>>> {'target': -19.0, 'params': {'y': 3.0, 'x': 4.0}}
>>> Iteration 3:
>>> {'target': -16.29839645063864, 'params': {'y': -2.412527795983739, 'x': 2.3776144540856503}}
>>> Iteration 4:
>>> {'target': -4.441293113411222, 'params': {'y': -0.005822117636089974, 'x': 2.104665051994087}}
```
## Minutiae
### Citation
If you used this package in your research, please cite it:
```
@Misc{,
author = {Fernando Nogueira},
title = {{Bayesian Optimization}: Open source constrained global optimization tool for {Python}},
year = {2014--},
url = " https://github.com/bayesian-optimization/BayesianOptimization"
}
```
If you used any of the advanced functionalities, please additionally cite the corresponding publication:
For the `SequentialDomainTransformer`:
```
@article{
author = {Stander, Nielen and Craig, Kenneth},
year = {2002},
month = {06},
pages = {},
title = {On the robustness of a simple domain reduction scheme for simulation-based optimization},
volume = {19},
journal = {International Journal for Computer-Aided Engineering and Software (Eng. Comput.)},
doi = {10.1108/02644400210430190}
}
```
For constrained optimization:
```
@inproceedings{gardner2014bayesian,
title={Bayesian optimization with inequality constraints.},
author={Gardner, Jacob R and Kusner, Matt J and Xu, Zhixiang Eddie and Weinberger, Kilian Q and Cunningham, John P},
booktitle={ICML},
volume={2014},
pages={937--945},
year={2014}
}
```
For optimization over non-float parameters:
```
@article{garrido2020dealing,
title={Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes},
author={Garrido-Merch{\'a}n, Eduardo C and Hern{\'a}ndez-Lobato, Daniel},
journal={Neurocomputing},
volume={380},
pages={20--35},
year={2020},
publisher={Elsevier}
}
```
================================================
FILE: bayes_opt/__init__.py
================================================
"""Pure Python implementation of bayesian global optimization with gaussian processes."""
from __future__ import annotations
import importlib.metadata
from bayes_opt import acquisition
from bayes_opt.bayesian_optimization import BayesianOptimization
from bayes_opt.constraint import ConstraintModel
from bayes_opt.domain_reduction import SequentialDomainReductionTransformer
from bayes_opt.logger import ScreenLogger
from bayes_opt.target_space import TargetSpace
__version__ = importlib.metadata.version("bayesian-optimization")
__all__ = [
"BayesianOptimization",
"ConstraintModel",
"ScreenLogger",
"SequentialDomainReductionTransformer",
"TargetSpace",
"acquisition",
]
================================================
FILE: bayes_opt/acquisition.py
================================================
"""Acquisition functions for Bayesian Optimization.
The acquisition functions in this module can be grouped the following way:
- One of the base acquisition functions
(:py:class:`UpperConfidenceBound<bayes_opt.acquisition.UpperConfidenceBound>`,
:py:class:`ProbabilityOfImprovement<bayes_opt.acquisition.ProbabilityOfImprovement>` and
:py:class:`ExpectedImprovement<bayes_opt.acquisition.ExpectedImprovement>`) is always dictating the basic
behavior of the suggestion step. They can be used alone or combined with a meta acquisition function.
- :py:class:`GPHedge<bayes_opt.acquisition.GPHedge>` is a meta acquisition function that combines multiple
base acquisition functions and determines the most suitable one for a particular problem.
- :py:class:`ConstantLiar<bayes_opt.acquisition.ConstantLiar>` is a meta acquisition function that can be
used for parallelized optimization and discourages sampling near a previously suggested, but not yet
evaluated, point.
- :py:class:`AcquisitionFunction<bayes_opt.acquisition.AcquisitionFunction>` is the base class for all
acquisition functions. You can implement your own acquisition function by subclassing it. See the
`Acquisition Functions notebook <../acquisition.html>`__ to understand the many ways this class can be
modified.
"""
from __future__ import annotations
import abc
import warnings
from copy import deepcopy
from typing import TYPE_CHECKING, Any, Literal, NoReturn
import numpy as np
from numpy.random import RandomState
from packaging import version
from scipy import __version__ as scipy_version
from scipy.optimize._differentialevolution import DifferentialEvolutionSolver, minimize
from scipy.special import softmax
from scipy.stats import norm
from sklearn.gaussian_process import GaussianProcessRegressor
from bayes_opt.exception import (
ConstraintNotSupportedError,
NoValidPointRegisteredError,
TargetSpaceEmptyError,
)
from bayes_opt.target_space import TargetSpace
from bayes_opt.util import ensure_rng
if TYPE_CHECKING:
from collections.abc import Callable, Sequence
from numpy.typing import NDArray
from scipy.optimize import OptimizeResult
from bayes_opt.constraint import ConstraintModel
Float = np.floating[Any]
class AcquisitionFunction(abc.ABC):
"""Base class for acquisition functions.
Parameters
----------
random_state : int, RandomState, default None
Set the random state for reproducibility.
"""
def __init__(self, random_state: int | RandomState | None = None) -> None:
if random_state is not None:
msg = (
"Providing a random_state to an acquisition function during initialization is deprecated "
"and will be ignored. The random_state is instead provided automatically during the "
"suggest() call."
)
warnings.warn(msg, DeprecationWarning, stacklevel=2)
self.i = 0
@abc.abstractmethod
def base_acq(self, *args: Any, **kwargs: Any) -> NDArray[Float]:
"""Provide access to the base acquisition function."""
def _fit_gp(self, gp: GaussianProcessRegressor, target_space: TargetSpace) -> None:
# Sklearn's GP throws a large number of warnings at times, but
# we don't really need to see them here.
with warnings.catch_warnings():
warnings.simplefilter("ignore")
gp.fit(target_space.params, target_space.target)
if target_space.constraint is not None:
target_space.constraint.fit(target_space.params, target_space._constraint_values)
def get_acquisition_params(self) -> dict[str, Any]:
"""
Get the parameters of the acquisition function.
Returns
-------
dict
The parameters of the acquisition function.
"""
error_msg = (
"Custom AcquisitionFunction subclasses must implement their own get_acquisition_params method."
)
raise NotImplementedError(error_msg)
def set_acquisition_params(self, params: dict[str, Any]) -> None:
"""
Set the parameters of the acquisition function.
Parameters
----------
params : dict
The parameters of the acquisition function.
"""
error_msg = (
"Custom AcquisitionFunction subclasses must implement their own set_acquisition_params method."
)
raise NotImplementedError(error_msg)
def suggest(
self,
gp: GaussianProcessRegressor,
target_space: TargetSpace,
n_random: int = 10_000,
n_smart: int = 10,
fit_gp: bool = True,
random_state: int | RandomState | None = None,
) -> NDArray[Float]:
"""Suggest a promising point to probe next.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
target_space : TargetSpace
The target space to probe.
n_random : int, default 10_000
Number of random samples to use.
n_smart : int, default 10
Controls the number of runs for the smart optimization. If all parameters are continuous,
this is the number of random starting points for the L-BFGS-B optimizer. If there are
discrete parameters, n_smart of the best points are used as starting points for the
differential evolution optimizer, with the remaining points being random samples.
fit_gp : bool, default True
Whether to fit the Gaussian Process to the target space.
Set to False if the GP is already fitted.
random_state : int, RandomState, default None
Random state to use for the optimization.
Returns
-------
np.ndarray
Suggested point to probe next.
"""
random_state = ensure_rng(random_state)
if len(target_space) == 0:
msg = (
"Cannot suggest a point without previous samples. Use "
" target_space.random_sample() to generate a point and "
" target_space.probe(*) to evaluate it."
)
raise TargetSpaceEmptyError(msg)
self.i += 1
if fit_gp:
self._fit_gp(gp=gp, target_space=target_space)
acq = self._get_acq(gp=gp, constraint=target_space.constraint)
return self._acq_min(acq, target_space, n_random=n_random, n_smart=n_smart, random_state=random_state)
def _get_acq(
self, gp: GaussianProcessRegressor, constraint: ConstraintModel | None = None
) -> Callable[[NDArray[Float]], NDArray[Float]]:
"""Prepare the acquisition function for minimization.
Transforms a base_acq Callable, which takes `mean` and `std` as
input, into an acquisition function that only requires an array of
parameters.
Handles GP predictions and constraints.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
constraint : ConstraintModel, default None
A fitted constraint model, if constraints are present and the
acquisition function supports them.
Returns
-------
Callable
Function to minimize.
"""
dim = gp.X_train_.shape[1]
if constraint is not None:
def acq(x: NDArray[Float]) -> NDArray[Float]:
x = x.reshape(-1, dim)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
mean: NDArray[Float]
std: NDArray[Float]
p_constraints: NDArray[Float]
mean, std = gp.predict(x, return_std=True)
p_constraints = constraint.predict(x)
return -1 * self.base_acq(mean, std) * p_constraints
else:
def acq(x: NDArray[Float]) -> NDArray[Float]:
x = x.reshape(-1, dim)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
mean: NDArray[Float]
std: NDArray[Float]
mean, std = gp.predict(x, return_std=True)
return -1 * self.base_acq(mean, std)
return acq
def _acq_min(
self,
acq: Callable[[NDArray[Float]], NDArray[Float]],
space: TargetSpace,
random_state: RandomState,
n_random: int = 10_000,
n_smart: int = 10,
) -> NDArray[Float]:
"""Find the maximum of the acquisition function.
Uses a combination of random sampling (cheap) and either 'L-BFGS-B' or differential evolution
optimization (smarter, but expensive). First samples `n_random` (1e5) points at random, then
uses the best points as starting points for the smart optimizer.
Parameters
----------
acq : Callable
Acquisition function to use. Should accept an array of parameters `x`.
space : TargetSpace
The target space over which to optimize.
random_state : RandomState
Random state to use for the optimization.
n_random : int
Number of random samples to use.
n_smart : int
Controls the number of runs for the smart optimization. If all parameters are continuous,
this is the number of random starting points for the L-BFGS-B optimizer. Otherwise, n_smart
of the best points are used as starting points for the differential evolution optimizer, with
the remaining points being random samples.
Returns
-------
np.ndarray
Parameters maximizing the acquisition function.
"""
if n_random == 0 and n_smart == 0:
error_msg = "Either n_random or n_smart needs to be greater than 0."
raise ValueError(error_msg)
x_min_r, min_acq_r, x_seeds = self._random_sample_minimize(
acq, space, random_state, n_random=max(n_random, n_smart), n_x_seeds=n_smart
)
if n_smart:
x_min_s, min_acq_s = self._smart_minimize(acq, space, x_seeds=x_seeds, random_state=random_state)
# Either n_random or n_smart is not 0 => at least one of x_min_r and x_min_s is not None
if min_acq_r > min_acq_s:
return x_min_s
return x_min_r
def _random_sample_minimize(
self,
acq: Callable[[NDArray[Float]], NDArray[Float]],
space: TargetSpace,
random_state: RandomState,
n_random: int,
n_x_seeds: int = 0,
) -> tuple[NDArray[Float] | None, float, NDArray[Float]]:
"""Random search to find the minimum of `acq` function.
Parameters
----------
acq : Callable
Acquisition function to use. Should accept an array of parameters `x`.
space : TargetSpace
The target space over which to optimize.
random_state : RandomState
Random state to use for the optimization.
n_random : int
Number of random samples to use.
n_x_seeds : int
Number of top points to return, for use as starting points for L-BFGS-B.
Returns
-------
x_min : np.ndarray
Random sample minimizing the acquisition function.
min_acq : float
Acquisition function value at `x_min`
"""
if n_random == 0:
return None, np.inf, space.random_sample(n_x_seeds, random_state=random_state)
x_tries = space.random_sample(n_random, random_state=random_state)
ys = acq(x_tries)
x_min = x_tries[ys.argmin()]
min_acq = ys.min()
if n_x_seeds != 0:
idxs = np.argsort(ys)[:n_x_seeds]
x_seeds = x_tries[idxs]
else:
x_seeds = []
return x_min, min_acq, x_seeds
def _smart_minimize(
self,
acq: Callable[[NDArray[Float]], NDArray[Float]],
space: TargetSpace,
x_seeds: NDArray[Float],
random_state: RandomState,
) -> tuple[NDArray[Float] | None, float]:
"""Random search to find the minimum of `acq` function.
Parameters
----------
acq : Callable
Acquisition function to use. Should accept an array of parameters `x`.
space : TargetSpace
The target space over which to optimize.
x_seeds : np.ndarray
Starting points for the smart optimizer.
If all parameters are continuous, this is the number of random starting points for the L-BFGS-B
optimizer. Otherwise, n_smart of the best points are used as starting points for the differential
evolution optimizer, with the remaining points being random samples.
random_state : RandomState
Random state to use for the optimization.
Returns
-------
x_min : np.ndarray
Minimal result of the L-BFGS-B optimizer.
min_acq : float
Acquisition function value at `x_min`
"""
continuous_dimensions = space.continuous_dimensions
continuous_bounds = space.bounds[continuous_dimensions]
min_acq: float | None = None
x_try: NDArray[Float]
x_min: NDArray[Float]
# Case of continous optimization
if all(continuous_dimensions):
for x_try in x_seeds:
res: OptimizeResult = minimize(acq, x_try, bounds=continuous_bounds, method="L-BFGS-B")
if not res.success:
continue
# Store it if better than previous minimum(maximum).
if min_acq is None or np.squeeze(res.fun) < min_acq:
x_try = res.x
x_min = x_try
min_acq = np.squeeze(res.fun)
# Case of mixed-integer optimization
else:
xinit = space.random_sample(15 * len(space.bounds), random_state=random_state)
if len(x_seeds) > 0:
n_seeds = min(len(x_seeds), len(xinit))
xinit[:n_seeds] = x_seeds[:n_seeds]
de_parameters = {"func": acq, "bounds": space.bounds, "polish": False, "init": xinit}
if version.parse(scipy_version) < version.parse("1.15.0"):
de_parameters["seed"] = random_state
else:
de_parameters["rng"] = random_state
de = DifferentialEvolutionSolver(**de_parameters)
res_de: OptimizeResult = de.solve()
# Check if success
if not res_de.success:
msg = f"Differential evolution optimization failed. Message: {res_de.message}"
raise RuntimeError(msg)
x_min = res_de.x
min_acq = np.squeeze(res_de.fun)
# Refine the identification of continous parameters with deterministic search
if any(continuous_dimensions):
x_try = x_min.copy()
def continuous_acq(x: NDArray[Float], x_try=x_try) -> NDArray[Float]:
x_try[continuous_dimensions] = x
return acq(x_try)
res: OptimizeResult = minimize(
continuous_acq, x_min[continuous_dimensions], bounds=continuous_bounds
)
if res.success and np.squeeze(res.fun) < min_acq:
x_try[continuous_dimensions] = res.x
x_min = x_try
min_acq = np.squeeze(res.fun)
if min_acq is None:
min_acq = np.inf
x_min = np.array([np.nan] * space.bounds.shape[0])
# Clip output to make sure it lies within the bounds. Due to floating
# point technicalities this is not always the case.
return np.clip(x_min, space.bounds[:, 0], space.bounds[:, 1]), min_acq
class UpperConfidenceBound(AcquisitionFunction):
r"""Upper Confidence Bound acquisition function.
The upper confidence bound is calculated as
.. math::
\text{UCB}(x) = \mu(x) + \kappa \sigma(x).
Parameters
----------
kappa : float, default 2.576
Governs the exploration/exploitation tradeoff. Lower prefers
exploitation, higher prefers exploration.
exploration_decay : float, default None
Decay rate for kappa. If None, no decay is applied.
exploration_decay_delay : int, default None
Delay for decay. If None, decay is applied from the start.
"""
def __init__(
self,
kappa: float = 2.576,
exploration_decay: float | None = None,
exploration_decay_delay: int | None = None,
random_state: int | RandomState | None = None,
) -> None:
if kappa < 0:
error_msg = "kappa must be greater than or equal to 0."
raise ValueError(error_msg)
if exploration_decay is not None and not (0 < exploration_decay <= 1):
error_msg = "exploration_decay must be greater than 0 and less than or equal to 1."
raise ValueError(error_msg)
if exploration_decay_delay is not None and (
not isinstance(exploration_decay_delay, int) or exploration_decay_delay < 0
):
error_msg = "exploration_decay_delay must be an integer greater than or equal to 0."
raise ValueError(error_msg)
super().__init__(random_state=random_state)
self.kappa = kappa
self.exploration_decay = exploration_decay
self.exploration_decay_delay = exploration_decay_delay
def base_acq(self, mean: NDArray[Float], std: NDArray[Float]) -> NDArray[Float]:
"""Calculate the upper confidence bound.
Parameters
----------
mean : np.ndarray
Mean of the predictive distribution.
std : np.ndarray
Standard deviation of the predictive distribution.
Returns
-------
np.ndarray
Acquisition function value.
"""
return mean + self.kappa * std
def suggest(
self,
gp: GaussianProcessRegressor,
target_space: TargetSpace,
n_random: int = 10_000,
n_smart: int = 10,
fit_gp: bool = True,
random_state: int | RandomState | None = None,
) -> NDArray[Float]:
"""Suggest a promising point to probe next.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
target_space : TargetSpace
The target space to probe.
n_random : int, default 10_000
Number of random samples to use.
n_smart : int, default 10
Number of starting points for the L-BFGS-B optimizer.
fit_gp : bool, default True
Whether to fit the Gaussian Process to the target space.
Set to False if the GP is already fitted.
random_state : int, RandomState, default None
Random state to use for the optimization.
Returns
-------
np.ndarray
Suggested point to probe next.
"""
if target_space.constraint is not None:
msg = (
f"Received constraints, but acquisition function {type(self)} "
"does not support constrained optimization."
)
raise ConstraintNotSupportedError(msg)
x_max = super().suggest(
gp=gp,
target_space=target_space,
n_random=n_random,
n_smart=n_smart,
fit_gp=fit_gp,
random_state=random_state,
)
self.decay_exploration()
return x_max
def decay_exploration(self) -> None:
"""Decay kappa by a constant rate.
Adjust exploration/exploitation trade-off by reducing kappa.
Note
----
This method is called automatically at the end of each ``suggest()`` call.
"""
if self.exploration_decay is not None and (
self.exploration_decay_delay is None or self.exploration_decay_delay <= self.i
):
self.kappa = self.kappa * self.exploration_decay
def get_acquisition_params(self) -> dict[str, Any]:
"""Get the current acquisition function parameters.
Returns
-------
dict
Dictionary containing the current acquisition function parameters.
"""
return {
"kappa": self.kappa,
"exploration_decay": self.exploration_decay,
"exploration_decay_delay": self.exploration_decay_delay,
}
def set_acquisition_params(self, params: dict[str, Any]) -> None:
"""Set the acquisition function parameters.
Parameters
----------
params : dict
Dictionary containing the acquisition function parameters.
"""
self.kappa = params["kappa"]
self.exploration_decay = params["exploration_decay"]
self.exploration_decay_delay = params["exploration_decay_delay"]
class ProbabilityOfImprovement(AcquisitionFunction):
r"""Probability of Improvement acqusition function.
Calculated as
.. math:: \text{POI}(x) = \Phi\left( \frac{\mu(x)-y_{\text{max}} - \xi }{\sigma(x)} \right)
where :math:`\Phi` is the CDF of the normal distribution.
Parameters
----------
xi : float, positive
Governs the exploration/exploitation tradeoff. Lower prefers
exploitation, higher prefers exploration.
exploration_decay : float, default None
Decay rate for xi. If None, no decay is applied.
exploration_decay_delay : int, default None
Delay for decay. If None, decay is applied from the start.
random_state : int, RandomState, default None
Set the random state for reproducibility.
"""
def __init__(
self,
xi: float,
exploration_decay: float | None = None,
exploration_decay_delay: int | None = None,
random_state: int | RandomState | None = None,
) -> None:
if xi < 0:
error_msg = "xi must be greater than or equal to 0."
raise ValueError(error_msg)
if exploration_decay is not None and not (0 < exploration_decay <= 1):
error_msg = "exploration_decay must be greater than 0 and less than or equal to 1."
raise ValueError(error_msg)
if exploration_decay_delay is not None and (
not isinstance(exploration_decay_delay, int) or exploration_decay_delay < 0
):
error_msg = "exploration_decay_delay must be an integer greater than or equal to 0."
raise ValueError(error_msg)
super().__init__(random_state=random_state)
self.xi = xi
self.exploration_decay = exploration_decay
self.exploration_decay_delay = exploration_decay_delay
self.y_max = None
def base_acq(self, mean: NDArray[Float], std: NDArray[Float]) -> NDArray[Float]:
"""Calculate the probability of improvement.
Parameters
----------
mean : np.ndarray
Mean of the predictive distribution.
std : np.ndarray
Standard deviation of the predictive distribution.
Returns
-------
np.ndarray
Acquisition function value.
Raises
------
ValueError
If y_max is not set.
"""
if self.y_max is None:
msg = (
"y_max is not set. If you are calling this method outside "
"of suggest(), you must set y_max manually."
)
raise ValueError(msg)
z = (mean - self.y_max - self.xi) / std
return norm.cdf(z)
def suggest(
self,
gp: GaussianProcessRegressor,
target_space: TargetSpace,
n_random: int = 10_000,
n_smart: int = 10,
fit_gp: bool = True,
random_state: int | RandomState | None = None,
) -> NDArray[Float]:
"""Suggest a promising point to probe next.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
target_space : TargetSpace
The target space to probe.
n_random : int, default 10_000
Number of random samples to use.
n_smart : int, default 10
Number of starting points for the L-BFGS-B optimizer.
fit_gp : bool, default True
Whether to fit the Gaussian Process to the target space.
Set to False if the GP is already fitted.
random_state : int, RandomState, default None
Random state to use for the optimization.
Returns
-------
np.ndarray
Suggested point to probe next.
"""
y_max = target_space._target_max()
if y_max is None and not target_space.empty:
# If target space is empty, let base class handle the error
msg = (
"Cannot suggest a point without an allowed point. Use "
"target_space.random_sample() to generate a point until "
" at least one point that satisfies the constraints is found."
)
raise NoValidPointRegisteredError(msg)
self.y_max = y_max
x_max = super().suggest(
gp=gp,
target_space=target_space,
n_random=n_random,
n_smart=n_smart,
fit_gp=fit_gp,
random_state=random_state,
)
self.decay_exploration()
return x_max
def decay_exploration(self) -> None:
r"""Decay xi by a constant rate.
Adjust exploration/exploitation trade-off by reducing xi.
Note
----
This method is called automatically at the end of each ``suggest()`` call.
"""
if self.exploration_decay is not None and (
self.exploration_decay_delay is None or self.exploration_decay_delay <= self.i
):
self.xi = self.xi * self.exploration_decay
def get_acquisition_params(self) -> dict[str, Any]:
"""Get the current acquisition function parameters.
Returns
-------
dict
Dictionary containing the current acquisition function parameters.
"""
return {
"xi": self.xi,
"exploration_decay": self.exploration_decay,
"exploration_decay_delay": self.exploration_decay_delay,
}
def set_acquisition_params(self, params: dict[str, Any]) -> None:
"""Set the acquisition function parameters.
Parameters
----------
params : dict
Dictionary containing the acquisition function parameters.
"""
self.xi = params["xi"]
self.exploration_decay = params["exploration_decay"]
self.exploration_decay_delay = params["exploration_decay_delay"]
class ExpectedImprovement(AcquisitionFunction):
r"""Expected Improvement acqusition function.
Similar to Probability of Improvement (`ProbabilityOfImprovement`), but also considers the
magnitude of improvement.
Calculated as
.. math::
\text{EI}(x) = (\mu(x)-y_{\text{max}} - \xi) \Phi\left(
\frac{\mu(x)-y_{\text{max}} - \xi }{\sigma(x)} \right)
+ \sigma(x) \phi\left(
\frac{\mu(x)-y_{\text{max}} - \xi }{\sigma(x)} \right)
where :math:`\Phi` is the CDF and :math:`\phi` the PDF of the normal
distribution.
Parameters
----------
xi : float, positive
Governs the exploration/exploitation tradeoff. Lower prefers
exploitation, higher prefers exploration.
exploration_decay : float, default None
Decay rate for xi. If None, no decay is applied.
exploration_decay_delay : int, default None
Delay for decay. If None, decay is applied from the start.
random_state : int, RandomState, default None
Set the random state for reproducibility.
"""
def __init__(
self,
xi: float,
exploration_decay: float | None = None,
exploration_decay_delay: int | None = None,
random_state: int | RandomState | None = None,
) -> None:
if xi < 0:
error_msg = "xi must be greater than or equal to 0."
raise ValueError(error_msg)
if exploration_decay is not None and not (0 < exploration_decay <= 1):
error_msg = "exploration_decay must be greater than 0 and less than or equal to 1."
raise ValueError(error_msg)
if exploration_decay_delay is not None and (
not isinstance(exploration_decay_delay, int) or exploration_decay_delay < 0
):
error_msg = "exploration_decay_delay must be an integer greater than or equal to 0."
raise ValueError(error_msg)
super().__init__(random_state=random_state)
self.xi = xi
self.exploration_decay = exploration_decay
self.exploration_decay_delay = exploration_decay_delay
self.y_max = None
def base_acq(self, mean: NDArray[Float], std: NDArray[Float]) -> NDArray[Float]:
"""Calculate the expected improvement.
Parameters
----------
mean : np.ndarray
Mean of the predictive distribution.
std : np.ndarray
Standard deviation of the predictive distribution.
Returns
-------
np.ndarray
Acquisition function value.
Raises
------
ValueError
If y_max is not set.
"""
if self.y_max is None:
msg = (
"y_max is not set. If you are calling this method outside "
"of suggest(), ensure y_max is set, or set it manually."
)
raise ValueError(msg)
a = mean - self.y_max - self.xi
z = a / std
return a * norm.cdf(z) + std * norm.pdf(z)
def suggest(
self,
gp: GaussianProcessRegressor,
target_space: TargetSpace,
n_random: int = 10_000,
n_smart: int = 10,
fit_gp: bool = True,
random_state: int | RandomState | None = None,
) -> NDArray[Float]:
"""Suggest a promising point to probe next.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
target_space : TargetSpace
The target space to probe.
n_random : int, default 10_000
Number of random samples to use.
n_smart : int, default 10
Number of starting points for the L-BFGS-B optimizer.
fit_gp : bool, default True
Whether to fit the Gaussian Process to the target space.
Set to False if the GP is already fitted.
random_state : int, RandomState, default None
Random state to use for the optimization.
Returns
-------
np.ndarray
Suggested point to probe next.
"""
y_max = target_space._target_max()
if y_max is None and not target_space.empty:
# If target space is empty, let base class handle the error
msg = (
"Cannot suggest a point without an allowed point. Use "
"target_space.random_sample() to generate a point until "
" at least one point that satisfies the constraints is found."
)
raise NoValidPointRegisteredError(msg)
self.y_max = y_max
x_max = super().suggest(
gp=gp,
target_space=target_space,
n_random=n_random,
n_smart=n_smart,
fit_gp=fit_gp,
random_state=random_state,
)
self.decay_exploration()
return x_max
def decay_exploration(self) -> None:
r"""Decay xi by a constant rate.
Adjust exploration/exploitation trade-off by reducing xi.
Note
----
This method is called automatically at the end of each ``suggest()`` call.
"""
if self.exploration_decay is not None and (
self.exploration_decay_delay is None or self.exploration_decay_delay <= self.i
):
self.xi = self.xi * self.exploration_decay
def get_acquisition_params(self) -> dict[str, Any]:
"""Get the current acquisition function parameters.
Returns
-------
dict
Dictionary containing the current acquisition function parameters.
"""
return {
"xi": self.xi,
"exploration_decay": self.exploration_decay,
"exploration_decay_delay": self.exploration_decay_delay,
}
def set_acquisition_params(self, params: dict[str, Any]) -> None:
"""Set the acquisition function parameters.
Parameters
----------
params : dict
Dictionary containing the acquisition function parameters.
"""
self.xi = params["xi"]
self.exploration_decay = params["exploration_decay"]
self.exploration_decay_delay = params["exploration_decay_delay"]
class ConstantLiar(AcquisitionFunction):
"""Constant Liar acquisition function.
Used for asynchronous optimization. It operates on a copy of the target space
that includes the previously suggested points that have not been evaluated yet.
A GP fitted to this target space is less likely to suggest the same point again,
since the variance of the predictive distribution is lower at these points.
This is discourages the optimization algorithm from suggesting the same point
to multiple workers.
Parameters
----------
base_acquisition : AcquisitionFunction
The acquisition function to use.
strategy : float or str, default 'max'
Strategy to use for the constant liar. If a float, the constant liar
will always register dummies with this value. If 'min'/'mean'/'max',
the constant liar will register dummies with the minimum/mean/maximum
target value in the target space.
random_state : int, RandomState, default None
Set the random state for reproducibility.
atol : float, default 1e-5
Absolute tolerance to eliminate a dummy point.
rtol : float, default 1e-8
Relative tolerance to eliminate a dummy point.
"""
def __init__(
self,
base_acquisition: AcquisitionFunction,
strategy: Literal["min", "mean", "max"] | float = "max",
random_state: int | RandomState | None = None,
atol: float = 1e-5,
rtol: float = 1e-8,
) -> None:
super().__init__(random_state)
self.base_acquisition = base_acquisition
self.dummies = []
if not isinstance(strategy, float) and strategy not in ["min", "mean", "max"]:
error_msg = f"Received invalid argument {strategy} for strategy."
raise ValueError(error_msg)
self.strategy: Literal["min", "mean", "max"] | float = strategy
self.atol = atol
self.rtol = rtol
def base_acq(self, *args: Any, **kwargs: Any) -> NDArray[Float]:
"""Calculate the acquisition function.
Calls the base acquisition function's `base_acq` method.
Returns
-------
np.ndarray
Acquisition function value.
"""
return self.base_acquisition.base_acq(*args, **kwargs)
def _copy_target_space(self, target_space: TargetSpace) -> TargetSpace:
"""Create a copy of the target space.
Parameters
----------
target_space : TargetSpace
The target space to copy.
Returns
-------
TargetSpace
A copy of the target space.
"""
keys = target_space.keys
pbounds = {key: bound for key, bound in zip(keys, target_space.bounds)}
target_space_copy = TargetSpace(
None, pbounds=pbounds, allow_duplicate_points=target_space._allow_duplicate_points
)
if target_space._constraint is not None:
target_space_copy.set_constraint(deepcopy(target_space.constraint))
target_space_copy._params = deepcopy(target_space._params)
target_space_copy._target = deepcopy(target_space._target)
return target_space_copy
def _remove_expired_dummies(self, target_space: TargetSpace) -> None:
"""Remove expired dummy points from the list of dummies.
Once a worker has evaluated a dummy point, the dummy is discarded. To
accomplish this, we compare every dummy point to the current target
space's parameters and remove it if it is close to any of them.
Parameters
----------
target_space : TargetSpace
The target space to compare the dummies to.
"""
dummies = []
for dummy in self.dummies:
close = np.isclose(dummy, target_space.params, rtol=self.rtol, atol=self.atol)
if not close.all(axis=1).any():
dummies.append(dummy)
self.dummies = dummies
def suggest(
self,
gp: GaussianProcessRegressor,
target_space: TargetSpace,
n_random: int = 10_000,
n_smart: int = 10,
fit_gp: bool = True,
random_state: int | RandomState | None = None,
) -> NDArray[Float]:
"""Suggest a promising point to probe next.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
target_space : TargetSpace
The target space to probe.
n_random : int, default 10_000
Number of random samples to use.
n_smart : int, default 10
Number of starting points for the L-BFGS-B optimizer.
fit_gp : bool, default True
Unused, since the GP is always fitted to the dummy target space.
Remains for compatibility with the base class.
random_state : int, RandomState, default None
Random state to use for the optimization.
Returns
-------
np.ndarray
Suggested point to probe next.
"""
if len(target_space) == 0:
msg = (
"Cannot suggest a point without previous samples. Use "
" target_space.random_sample() to generate a point and "
" target_space.probe(*) to evaluate it."
)
raise TargetSpaceEmptyError(msg)
if target_space.constraint is not None:
msg = (
f"Received constraints, but acquisition function {type(self)} "
"does not support constrained optimization."
)
raise ConstraintNotSupportedError(msg)
# Check if any dummies have been evaluated and remove them
self._remove_expired_dummies(target_space)
# Create a copy of the target space
dummy_target_space = self._copy_target_space(target_space)
dummy_target: float
# Choose the dummy target value
if isinstance(self.strategy, float):
dummy_target = self.strategy
elif self.strategy == "min":
dummy_target = target_space.target.min()
elif self.strategy == "mean":
dummy_target = target_space.target.mean()
elif self.strategy != "max":
error_msg = f"Received invalid argument {self.strategy} for strategy."
raise ValueError(error_msg)
else:
dummy_target = target_space.target.max()
# Register the dummies to the dummy target space
for dummy in self.dummies:
dummy_target_space.register(dummy, dummy_target)
# Fit the GP to the dummy target space and suggest a point
self._fit_gp(gp=gp, target_space=dummy_target_space)
x_max = self.base_acquisition.suggest(
gp,
dummy_target_space,
n_random=n_random,
n_smart=n_smart,
fit_gp=False,
random_state=random_state,
)
# Register the suggested point as a dummy
self.dummies.append(x_max)
return x_max
def get_acquisition_params(self) -> dict[str, Any]:
"""Get the current acquisition function parameters.
Returns
-------
dict
Dictionary containing the current acquisition function parameters.
"""
return {
"dummies": [dummy.tolist() for dummy in self.dummies],
"base_acquisition_params": self.base_acquisition.get_acquisition_params(),
"strategy": self.strategy,
"atol": self.atol,
"rtol": self.rtol,
}
def set_acquisition_params(self, params: dict[str, Any]) -> None:
"""Set the acquisition function parameters.
Parameters
----------
params : dict
Dictionary containing the acquisition function parameters.
"""
self.dummies = [np.array(dummy) for dummy in params["dummies"]]
self.base_acquisition.set_acquisition_params(params["base_acquisition_params"])
self.strategy = params["strategy"]
self.atol = params["atol"]
self.rtol = params["rtol"]
class GPHedge(AcquisitionFunction):
"""GPHedge acquisition function.
At each suggestion step, GPHedge samples suggestions from each base
acquisition function acq_i. Then a candidate is selected from the
suggestions based on the on the cumulative rewards of each acq_i.
After evaluating the candidate, the gains are updated (in the next
iteration) based on the updated expectation value of the candidates.
For more information, see:
Brochu et al., "Portfolio Allocation for Bayesian Optimization",
https://arxiv.org/abs/1009.5419
Parameters
----------
base_acquisitions : Sequence[AcquisitionFunction]
Sequence of base acquisition functions.
random_state : int, RandomState, default None
Set the random state for reproducibility.
"""
def __init__(
self, base_acquisitions: Sequence[AcquisitionFunction], random_state: int | RandomState | None = None
) -> None:
super().__init__(random_state)
self.base_acquisitions = list(base_acquisitions)
self.n_acq = len(self.base_acquisitions)
self.gains = np.zeros(self.n_acq)
self.previous_candidates = None
def base_acq(self, *args: Any, **kwargs: Any) -> NoReturn:
"""Raise an error, since the base acquisition function is ambiguous."""
msg = (
"GPHedge base acquisition function is ambiguous."
" You may use self.base_acquisitions[i].base_acq(mean, std)"
" to get the base acquisition function for the i-th acquisition."
)
raise TypeError(msg)
def _sample_idx_from_softmax_gains(self, random_state: RandomState) -> int:
"""Sample an index weighted by the softmax of the gains.
Parameters
----------
random_state : RandomState
Random state to use for the sampling.
Returns
-------
int
Index of the selected base acquisition function.
"""
cumsum_softmax_g = np.cumsum(softmax(self.gains))
r = random_state.rand()
return np.argmax(r <= cumsum_softmax_g) # Returns the first True value
def _update_gains(self, gp: GaussianProcessRegressor) -> None:
"""Update the gains of the base acquisition functions.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
"""
with warnings.catch_warnings():
warnings.simplefilter("ignore")
rewards = gp.predict(self.previous_candidates)
self.gains += rewards
self.previous_candidates = None
def suggest(
self,
gp: GaussianProcessRegressor,
target_space: TargetSpace,
n_random: int = 10_000,
n_smart: int = 10,
fit_gp: bool = True,
random_state: int | RandomState | None = None,
) -> NDArray[Float]:
"""Suggest a promising point to probe next.
Parameters
----------
gp : GaussianProcessRegressor
A fitted Gaussian Process.
target_space : TargetSpace
The target space to probe.
n_random : int, default 10_000
Number of random samples to use.
n_smart : int, default 10
Number of starting points for the L-BFGS-B optimizer.
fit_gp : bool, default True
Whether to fit the Gaussian Process to the target space.
Set to False if the GP is already fitted.
random_state : int, RandomState, default None
Random state to use for the optimization.
Returns
-------
np.ndarray
Suggested point to probe next.
"""
if len(target_space) == 0:
msg = (
"Cannot suggest a point without previous samples. Use "
" target_space.random_sample() to generate a point and "
" target_space.probe(*) to evaluate it."
)
raise TargetSpaceEmptyError(msg)
self.i += 1
random_state = ensure_rng(random_state)
if fit_gp:
self._fit_gp(gp=gp, target_space=target_space)
# Update the gains of the base acquisition functions
if self.previous_candidates is not None:
self._update_gains(gp)
# Suggest a point using each base acquisition function
x_max = [
base_acq.suggest(
gp=gp,
target_space=target_space,
n_random=n_random // self.n_acq,
n_smart=n_smart // self.n_acq,
fit_gp=False,
random_state=random_state,
)
for base_acq in self.base_acquisitions
]
self.previous_candidates = np.array(x_max)
idx = self._sample_idx_from_softmax_gains(random_state=random_state)
return x_max[idx]
def get_acquisition_params(self) -> dict[str, Any]:
"""Get the current acquisition function parameters.
Returns
-------
dict
Dictionary containing the current acquisition function parameters.
"""
return {
"base_acquisitions_params": [acq.get_acquisition_params() for acq in self.base_acquisitions],
"gains": self.gains.tolist(),
"previous_candidates": self.previous_candidates.tolist()
if self.previous_candidates is not None
else None,
}
def set_acquisition_params(self, params: dict[str, Any]) -> None:
"""Set the acquisition function parameters.
Parameters
----------
params : dict
Dictionary containing the acquisition function parameters.
"""
for acq, acq_params in zip(self.base_acquisitions, params["base_acquisitions_params"]):
acq.set_acquisition_params(acq_params)
self.gains = np.array(params["gains"])
self.previous_candidates = (
np.array(params["previous_candidates"]) if params["previous_candidates"] is not None else None
)
================================================
FILE: bayes_opt/bayesian_optimization.py
================================================
"""Main module.
Holds the `BayesianOptimization` class, which handles the maximization of a
function over a specific target space.
"""
from __future__ import annotations
import json
from collections import deque
from collections.abc import Iterable
from os import PathLike
from pathlib import Path
from typing import TYPE_CHECKING, Any
from warnings import warn
import numpy as np
from scipy.optimize import NonlinearConstraint
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern
from bayes_opt import acquisition
from bayes_opt.domain_reduction import DomainTransformer
from bayes_opt.logger import ScreenLogger
from bayes_opt.parameter import wrap_kernel
from bayes_opt.target_space import TargetSpace
from bayes_opt.util import ensure_rng
if TYPE_CHECKING:
from collections.abc import Callable, Mapping
from numpy.random import RandomState
from numpy.typing import NDArray
from bayes_opt.acquisition import AcquisitionFunction
from bayes_opt.constraint import ConstraintModel
from bayes_opt.domain_reduction import DomainTransformer
from bayes_opt.parameter import BoundsMapping, ParamsType
Float = np.floating[Any]
class BayesianOptimization:
"""Handle optimization of a target function over a specific target space.
This class takes the function to optimize as well as the parameters bounds
in order to find which values for the parameters yield the maximum value
using bayesian optimization.
Parameters
----------
f: function or None.
Function to be maximized.
pbounds: dict
Dictionary with parameters names as keys and a tuple with minimum
and maximum values.
acquisition_function: AcquisitionFunction, optional(default=None)
The acquisition function to use for suggesting new points to evaluate.
If None, defaults to UpperConfidenceBound for unconstrained problems
and ExpectedImprovement for constrained problems.
constraint: NonlinearConstraint.
Note that the names of arguments of the constraint function and of
f need to be the same.
random_state: int or numpy.random.RandomState, optional(default=None)
If the value is an integer, it is used as the seed for creating a
numpy.random.RandomState. Otherwise the random state provided is used.
When set to None, an unseeded random state is generated.
verbose: int, optional(default=2)
The level of verbosity.
bounds_transformer: DomainTransformer, optional(default=None)
If provided, the transformation is applied to the bounds.
allow_duplicate_points: bool, optional (default=False)
If True, the optimizer will allow duplicate points to be registered.
This behavior may be desired in high noise situations where repeatedly probing
the same point will give different answers. In other situations, the acquisition
may occasionally generate a duplicate point.
"""
def __init__(
self,
f: Callable[..., float] | None,
pbounds: Mapping[str, tuple[float, float]],
acquisition_function: AcquisitionFunction | None = None,
constraint: NonlinearConstraint | None = None,
random_state: int | RandomState | None = None,
verbose: int = 2,
bounds_transformer: DomainTransformer | None = None,
allow_duplicate_points: bool = False,
):
self._random_state = ensure_rng(random_state)
self._allow_duplicate_points = allow_duplicate_points
self._queue: deque[ParamsType] = deque()
if acquisition_function is None:
if constraint is None:
self._acquisition_function = acquisition.UpperConfidenceBound(kappa=2.576)
else:
self._acquisition_function = acquisition.ExpectedImprovement(xi=0.01)
else:
self._acquisition_function = acquisition_function
# Data structure containing the function to be optimized, the
# bounds of its domain, and a record of the evaluations we have
# done so far
self._space = TargetSpace(
f,
pbounds,
constraint=constraint,
random_state=random_state,
allow_duplicate_points=self._allow_duplicate_points,
)
if constraint is None:
self.is_constrained = False
else:
self.is_constrained = True
# Internal GP regressor
self._gp = GaussianProcessRegressor(
kernel=wrap_kernel(Matern(nu=2.5), transform=self._space.kernel_transform),
alpha=1e-6,
normalize_y=True,
n_restarts_optimizer=5,
random_state=self._random_state,
)
self._verbose = verbose
self._bounds_transformer = bounds_transformer
if self._bounds_transformer:
if not isinstance(self._bounds_transformer, DomainTransformer):
msg = "The transformer must be an instance of DomainTransformer"
raise TypeError(msg)
self._bounds_transformer.initialize(self._space)
self._sorting_warning_already_shown = False # TODO: remove in future version
# Initialize logger
self.logger = ScreenLogger(verbose=self._verbose, is_constrained=self.is_constrained)
@property
def space(self) -> TargetSpace:
"""Return the target space associated with the optimizer."""
return self._space
@property
def acquisition_function(self) -> AcquisitionFunction:
"""Return the acquisition function associated with the optimizer."""
return self._acquisition_function
@property
def constraint(self) -> ConstraintModel | None:
"""Return the constraint associated with the optimizer, if any."""
if self.is_constrained:
return self._space.constraint
return None
@property
def max(self) -> dict[str, Any] | None:
"""Get maximum target value found and corresponding parameters.
See `TargetSpace.max` for more information.
"""
return self._space.max()
@property
def res(self) -> list[dict[str, Any]]:
"""Get all target values and constraint fulfillment for all parameters.
See `TargetSpace.res` for more information.
"""
return self._space.res()
def predict(
self,
params: dict[str, Any] | Iterable[dict[str, Any]],
return_std=False,
return_cov=False,
fit_gp=True,
) -> float | NDArray[Float] | tuple[float | NDArray[Float], float | NDArray[Float]]:
"""Predict the target function value at given parameters.
Parameters
---------
params: dict or iterable of dicts
The parameters where the prediction is made.
return_std: bool, optional(default=False)
If True, the standard deviation of the prediction is returned.
return_cov: bool, optional(default=False)
If True, the covariance of the prediction is returned.
fit_gp: bool, optional(default=True)
If True, the internal Gaussian Process model is fitted before
making the prediction.
Returns
-------
mean: float or np.ndarray
The predicted mean of the target function at the given parameters.
When params is a dict, returns a scalar. When params is an iterable,
returns a 1D array.
std_or_cov: float or np.ndarray (only if return_std or return_cov is True)
The predicted standard deviation or covariance of the target function
at the given parameters.
"""
# Validate param types
if isinstance(params, dict):
params_array = self._space.params_to_array(params).reshape(1, -1)
single_param = True
elif isinstance(params, Iterable) and not isinstance(params, str):
# convert iterable of dicts to 2D array
params_array = np.array([self._space.params_to_array(p) for p in params])
single_param = False
else:
msg = f"params must be a dict or iterable of dicts, got {type(params).__name__}"
raise TypeError(msg)
# Validate mutual exclusivity of return_std and return_cov
if return_std and return_cov:
msg = "return_std and return_cov cannot both be True"
raise ValueError(msg)
if fit_gp:
if len(self._space) == 0:
msg = (
"The Gaussian Process model cannot be fitted with zero observations. To use predict(), "
"without fitting the GP, set fit_gp=False. The predictions will then be made using the "
"GP prior."
)
raise RuntimeError(msg)
self.acquisition_function._fit_gp(self._gp, self._space)
res = self._gp.predict(params_array, return_std=return_std, return_cov=return_cov)
if return_std or return_cov:
mean, std_or_cov = res
else:
mean = res
# Shape semantics: dict input returns scalars, list input returns arrays
# Ensure list input always returns arrays (convert scalar to 1D if needed)
if not single_param and mean.ndim == 0:
mean = np.atleast_1d(mean)
# ruff complains when nesting conditionals, so this three-way split is necessary
if not single_param and (return_std or return_cov) and std_or_cov.ndim == 0:
std_or_cov = np.atleast_1d(std_or_cov)
if single_param and mean.ndim > 0:
mean = mean[0]
if single_param and return_std and std_or_cov.ndim > 0:
std_or_cov = std_or_cov[0]
if return_std or return_cov:
return mean, std_or_cov
return mean
def register(
self, params: ParamsType, target: float, constraint_value: float | NDArray[Float] | None = None
) -> None:
"""Register an observation with known target.
Parameters
----------
params: dict or list
The parameters associated with the observation.
target: float
Value of the target function at the observation.
constraint_value: float or None
Value of the constraint function at the observation, if any.
"""
# TODO: remove in future version
if isinstance(params, np.ndarray) and not self._sorting_warning_already_shown:
msg = (
"You're attempting to register an np.ndarray. In previous versions, the optimizer internally"
" sorted parameters by key and expected any registered array to respect this order."
" In the current and any future version the order as given by the pbounds dictionary will be"
" used. If you wish to retain sorted parameters, please manually sort your pbounds"
" dictionary before constructing the optimizer."
)
warn(msg, stacklevel=1)
self._sorting_warning_already_shown = True
self._space.register(params, target, constraint_value)
self.logger.log_optimization_step(
self._space.keys, self._space.res()[-1], self._space.params_config, self.max
)
def probe(self, params: ParamsType, lazy: bool = True) -> None:
"""Evaluate the function at the given points.
Useful to guide the optimizer.
Parameters
----------
params: dict or list
The parameters where the optimizer will evaluate the function.
lazy: bool, optional(default=True)
If True, the optimizer will evaluate the points when calling
maximize(). Otherwise it will evaluate it at the moment.
"""
# TODO: remove in future version
if isinstance(params, np.ndarray) and not self._sorting_warning_already_shown:
msg = (
"You're attempting to register an np.ndarray. In previous versions, the optimizer internally"
" sorted parameters by key and expected any registered array to respect this order."
" In the current and any future version the order as given by the pbounds dictionary will be"
" used. If you wish to retain sorted parameters, please manually sort your pbounds"
" dictionary before constructing the optimizer."
)
warn(msg, stacklevel=1)
self._sorting_warning_already_shown = True
params = self._space.array_to_params(params)
if lazy:
self._queue.append(params)
else:
self._space.probe(params)
self.logger.log_optimization_step(
self._space.keys, self._space.res()[-1], self._space.params_config, self.max
)
def random_sample(self, n: int = 1) -> list[dict[str, float | NDArray[Float]]]:
"""Generate a random sample of parameters from the target space.
Parameters
----------
n: int, optional(default=1)
Number of random samples to generate.
Returns
-------
list of dict
List of randomly sampled parameters.
"""
return [
self._space.array_to_params(self._space.random_sample(random_state=self._random_state))
for _ in range(n)
]
def suggest(self) -> dict[str, float | NDArray[Float]]:
"""Suggest a promising point to probe next."""
if len(self._space) == 0:
return self.random_sample(1)[0]
# Finding argmax of the acquisition function.
suggestion = self._acquisition_function.suggest(
gp=self._gp, target_space=self._space, fit_gp=True, random_state=self._random_state
)
return self._space.array_to_params(suggestion)
def _prime_queue(self, init_points: int) -> None:
"""Ensure the queue is not empty.
Parameters
----------
init_points: int
Number of parameters to prime the queue with.
"""
if not self._queue and self._space.empty:
init_points = max(init_points, 1)
self._queue.extend(self.random_sample(init_points))
def maximize(self, init_points: int = 5, n_iter: int = 25) -> None:
r"""
Maximize the given function over the target space.
Parameters
----------
init_points : int, optional(default=5)
Number of random points to probe before starting the optimization.
n_iter: int, optional(default=25)
Number of iterations where the method attempts to find the maximum
value.
Warning
-------
The maximize loop only fits the GP when suggesting a new point to
probe based on the acquisition function. This means that the GP may
not be fitted on all points registered to the target space when the
method completes. If you intend to use the GP model after the
optimization routine, make sure to call predict() with fit_gp=True.
"""
# Log optimization start
self.logger.log_optimization_start(self._space.keys)
# Prime the queue with random points
self._prime_queue(init_points)
iteration = 0
while self._queue or iteration < n_iter:
try:
x_probe = self._queue.popleft()
except IndexError:
x_probe = self.suggest()
iteration += 1
self.probe(x_probe, lazy=False)
if self._bounds_transformer and iteration > 0:
# The bounds transformer should only modify the bounds after
# the init_points points (only for the true iterations)
self.set_bounds(self._bounds_transformer.transform(self._space))
# Log optimization end
self.logger.log_optimization_end()
def set_bounds(self, new_bounds: BoundsMapping) -> None:
"""Modify the bounds of the search space.
Parameters
----------
new_bounds : dict
A dictionary with the parameter name and its new bounds
"""
self._space.set_bounds(new_bounds)
def set_gp_params(self, **params: Any) -> None:
"""Set parameters of the internal Gaussian Process Regressor."""
if "kernel" in params:
params["kernel"] = wrap_kernel(kernel=params["kernel"], transform=self._space.kernel_transform)
self._gp.set_params(**params)
def save_state(self, path: str | PathLike[str]) -> None:
"""Save complete state for reconstruction of the optimizer.
Parameters
----------
path : str or PathLike
Path to save the optimization state
"""
random_state = None
if self._random_state is not None:
state_dict = self._random_state.get_state(legacy=False)
random_state = {
"bit_generator": state_dict["bit_generator"],
"state": state_dict["state"]["key"].tolist(),
"pos": state_dict["state"]["pos"],
"has_gauss": state_dict["has_gauss"],
"cached_gaussian": state_dict["gauss"],
}
# Get constraint values if they exist
constraint_values = self._space._constraint_values.tolist() if self.is_constrained else None
acquisition_params = self._acquisition_function.get_acquisition_params()
state = {
"pbounds": {key: self._space._bounds[i].tolist() for i, key in enumerate(self._space.keys)},
# Add current transformed bounds if using bounds transformer
"transformed_bounds": (self._space.bounds.tolist() if self._bounds_transformer else None),
"keys": self._space.keys,
"params": np.array(self._space.params).tolist(),
"target": self._space.target.tolist(),
"constraint_values": constraint_values,
"gp_params": {
"kernel": self._gp.kernel.get_params(),
"alpha": self._gp.alpha,
"normalize_y": self._gp.normalize_y,
"n_restarts_optimizer": self._gp.n_restarts_optimizer,
},
"allow_duplicate_points": self._allow_duplicate_points,
"verbose": self._verbose,
"random_state": random_state,
"acquisition_params": acquisition_params,
}
with Path(path).open("w") as f:
json.dump(state, f, indent=2)
def load_state(self, path: str | PathLike[str]) -> None:
"""Load optimizer state from a JSON file.
Parameters
----------
path : str or PathLike
Path to the JSON file containing the optimizer state.
"""
with Path(path).open("r") as file:
state = json.load(file)
params_array = np.asarray(state["params"], dtype=np.float64)
target_array = np.asarray(state["target"], dtype=np.float64)
constraint_array = (
np.array(state["constraint_values"]) if state["constraint_values"] is not None else None
)
for i in range(len(params_array)):
params = self._space.array_to_params(params_array[i])
target = target_array[i]
constraint = constraint_array[i] if constraint_array is not None else None
self.register(params=params, target=target, constraint_value=constraint)
self._acquisition_function.set_acquisition_params(state["acquisition_params"])
if state.get("transformed_bounds") and self._bounds_transformer:
new_bounds = {
key: bounds for key, bounds in zip(self._space.keys, np.array(state["transformed_bounds"]))
}
self._space.set_bounds(new_bounds)
self._bounds_transformer.initialize(self._space)
# Construct the GP kernel
kernel = Matern(**state["gp_params"]["kernel"])
# Re-construct the GP parameters
gp_params = {k: v for k, v in state["gp_params"].items() if k != "kernel"}
gp_params["kernel"] = kernel
# Set the GP parameters
self.set_gp_params(**gp_params)
if len(self._space):
self._gp.fit(self._space.params, self._space.target)
if state["random_state"] is not None:
random_state_tuple = (
state["random_state"]["bit_generator"],
np.array(state["random_state"]["state"], dtype=np.uint32),
state["random_state"]["pos"],
state["random_state"]["has_gauss"],
state["random_state"]["cached_gaussian"],
)
self._random_state.set_state(random_state_tuple)
================================================
FILE: bayes_opt/constraint.py
================================================
"""Constraint handling."""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
import numpy as np
from scipy.stats import norm
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern
from bayes_opt.parameter import wrap_kernel
if TYPE_CHECKING:
from collections.abc import Callable
from numpy.random import RandomState
from numpy.typing import NDArray
Float = np.floating[Any]
class ConstraintModel:
"""Model constraints using GP regressors.
This class takes the function to optimize as well as the parameters bounds
in order to find which values for the parameters yield the maximum value
using bayesian optimization.
Parameters
----------
fun : None or Callable -> float or np.ndarray
The constraint function. Should be float-valued or array-valued (if
multiple constraints are present). Needs to take the same parameters
as the optimization target with the same argument names.
lb : float or np.ndarray
The lower bound on the constraints. Should have the same
dimensionality as the return value of the constraint function.
ub : float or np.ndarray
The upper bound on the constraints. Should have the same
dimensionality as the return value of the constraint function.
random_state : np.random.RandomState or int or None, default=None
Random state to use.
Note
----
In case of multiple constraints, this model assumes conditional
independence. This means that the overall probability of fulfillment is a
simply the product of the individual probabilities.
"""
def __init__(
self,
fun: Callable[..., float] | Callable[..., NDArray[Float]] | None,
lb: float | NDArray[Float],
ub: float | NDArray[Float],
transform: Callable[[Any], Any] | None = None,
random_state: int | RandomState | None = None,
) -> None:
self.fun = fun
self._lb = np.atleast_1d(lb)
self._ub = np.atleast_1d(ub)
if np.any(self._lb >= self._ub):
msg = "Lower bounds must be less than upper bounds."
raise ValueError(msg)
self._model = [
GaussianProcessRegressor(
kernel=wrap_kernel(Matern(nu=2.5), transform) if transform is not None else Matern(nu=2.5),
alpha=1e-6,
normalize_y=True,
n_restarts_optimizer=5,
random_state=random_state,
)
for _ in range(len(self._lb))
]
@property
def lb(self) -> NDArray[Float]:
"""Return lower bounds."""
return self._lb
@property
def ub(self) -> NDArray[Float]:
"""Return upper bounds."""
return self._ub
@property
def model(self) -> list[GaussianProcessRegressor]:
"""Return GP regressors of the constraint function."""
return self._model
def eval(self, **kwargs: Any) -> float | NDArray[Float]: # noqa: D417
r"""Evaluate the constraint function.
Parameters
----------
\*\*kwargs : any
Function arguments to evaluate the constraint function on.
Returns
-------
Value of the constraint function.
Raises
------
TypeError
If the kwargs' keys don't match the function argument names.
"""
if self.fun is None:
error_msg = "No constraint function was provided."
raise ValueError(error_msg)
try:
return self.fun(**kwargs)
except TypeError as e:
msg = (
"Encountered TypeError when evaluating constraint "
"function. This could be because your constraint function "
"doesn't use the same keyword arguments as the target "
f"function. Original error message:\n\n{e}"
)
e.args = (msg,)
raise
def fit(self, X: NDArray[Float], Y: NDArray[Float]) -> None:
"""Fit internal GPRs to the data.
Parameters
----------
X : np.ndarray of shape (n_samples, n_features)
Parameters of the constraint function.
Y : np.ndarray of shape (n_samples, n_constraints)
Values of the constraint function.
Returns
-------
None
"""
if len(self._model) == 1:
self._model[0].fit(X, Y)
else:
for i, gp in enumerate(self._model):
gp.fit(X, Y[:, i])
def predict(self, X: NDArray[Float]) -> NDArray[Float]:
r"""Calculate the probability that the constraint is fulfilled at `X`.
Note that this does not try to approximate the values of the
constraint function (for this, see `ConstraintModel.approx()`.), but
probability that the constraint function is fulfilled. That is, this
function calculates
.. math::
p = \text{Pr}\left\{c^{\text{low}} \leq \tilde{c}(x) \leq
c^{\text{up}} \right\} = \int_{c^{\text{low}}}^{c^{\text{up}}}
\mathcal{N}(c, \mu(x), \sigma^2(x)) \, dc.
with :math:`\mu(x)`, :math:`\sigma^2(x)` the mean and variance at
:math:`x` as given by the GP and :math:`c^{\text{low}}`,
:math:`c^{\text{up}}` the lower and upper bounds of the constraint
respectively.
Note
----
In case of multiple constraints, we assume conditional independence.
This means we calculate the probability of constraint fulfilment
individually, with the joint probability given as their product.
Parameters
----------
X : np.ndarray of shape (n_samples, n_features)
Parameters for which to predict the probability of constraint
fulfilment.
Returns
-------
np.ndarray of shape (n_samples,)
Probability of constraint fulfilment.
"""
X_shape = X.shape
X = X.reshape((-1, self._model[0].n_features_in_))
result: NDArray[Float]
y_mean: NDArray[Float]
y_std: NDArray[Float]
p_lower: NDArray[Float]
p_upper: NDArray[Float]
if len(self._model) == 1:
y_mean, y_std = self._model[0].predict(X, return_std=True)
p_lower = (
norm(loc=y_mean, scale=y_std).cdf(self._lb[0]) if self._lb[0] != -np.inf else np.array([0])
)
p_upper = (
norm(loc=y_mean, scale=y_std).cdf(self._ub[0]) if self._ub[0] != np.inf else np.array([1])
)
result = p_upper - p_lower
return result.reshape(X_shape[:-1])
result = np.ones(X.shape[0])
for j, gp in enumerate(self._model):
y_mean, y_std = gp.predict(X, return_std=True)
p_lower = (
norm(loc=y_mean, scale=y_std).cdf(self._lb[j]) if self._lb[j] != -np.inf else np.array([0])
)
p_upper = (
norm(loc=y_mean, scale=y_std).cdf(self._ub[j]) if self._ub[j] != np.inf else np.array([1])
)
result = result * (p_upper - p_lower)
return result.reshape(X_shape[:-1])
def approx(self, X: NDArray[Float]) -> NDArray[Float]:
"""
Approximate the constraint function using the internal GPR model.
Parameters
----------
X : np.ndarray of shape (n_samples, n_features)
Parameters for which to estimate the constraint function value.
Returns
-------
np.ndarray of shape (n_samples, n_constraints)
Constraint function value estimates.
"""
X_shape = X.shape
X = X.reshape((-1, self._model[0].n_features_in_))
if len(self._model) == 1:
return self._model[0].predict(X).reshape(X_shape[:-1])
result = np.column_stack([gp.predict(X) for gp in self._model])
return result.reshape(*X_shape[:-1], len(self._lb))
def allowed(self, constraint_values: NDArray[Float]) -> NDArray[np.bool_]:
"""Check whether `constraint_values` fulfills the specified limits.
Parameters
----------
constraint_values : np.ndarray of shape (n_samples, n_constraints)
The values of the constraint function.
Returns
-------
np.ndarrray of shape (n_samples,)
Specifying wheter the constraints are fulfilled.
"""
if self._lb.size == 1:
return np.less_equal(self._lb, constraint_values) & np.less_equal(constraint_values, self._ub)
return np.all(constraint_values <= self._ub, axis=-1) & np.all(constraint_values >= self._lb, axis=-1)
================================================
FILE: bayes_opt/domain_reduction.py
================================================
"""Implement domain transformation.
In particular, this provides a base transformer class and a sequential domain
reduction transformer as based on Stander and Craig's "On the robustness of a
simple domain reduction scheme for simulation-based optimization"
"""
from __future__ import annotations
from abc import ABC, abstractmethod
from collections.abc import Iterable, Mapping, Sequence
from typing import TYPE_CHECKING, Any
from warnings import warn
import numpy as np
from bayes_opt.parameter import FloatParameter
from bayes_opt.target_space import TargetSpace
if TYPE_CHECKING:
from numpy.typing import NDArray
Float = np.floating[Any]
class DomainTransformer(ABC):
"""Base class."""
@abstractmethod
def __init__(self, **kwargs: Any) -> None:
"""To override with specific implementation."""
@abstractmethod
def initialize(self, target_space: TargetSpace) -> None:
"""To override with specific implementation."""
@abstractmethod
def transform(self, target_space: TargetSpace) -> dict[str, NDArray[Float]]:
"""To override with specific implementation."""
class SequentialDomainReductionTransformer(DomainTransformer):
"""Reduce the searchable space.
A sequential domain reduction transformer based on the work by Stander, N. and Craig, K:
"On the robustness of a simple domain reduction scheme for simulation-based optimization"
Parameters
----------
gamma_osc : float, default=0.7
Parameter used to scale (typically dampen) oscillations.
gamma_pan : float, default=1.0
Parameter used to scale (typically unitary) panning.
eta : float, default=0.9
Zooming parameter used to shrink the region of interest.
minimum_window : float or np.ndarray or dict, default=0.0
Minimum window size for each parameter. If a float is provided,
the same value is used for all parameters.
"""
def __init__(
self,
parameters: Iterable[str] | None = None,
gamma_osc: float = 0.7,
gamma_pan: float = 1.0,
eta: float = 0.9,
minimum_window: NDArray[Float] | Sequence[float] | Mapping[str, float] | float = 0.0,
) -> None:
# TODO: Ensure that this is only applied to continuous parameters
self.parameters = parameters
self.gamma_osc = gamma_osc
self.gamma_pan = gamma_pan
self.eta = eta
self.minimum_window_value = minimum_window
def initialize(self, target_space: TargetSpace) -> None:
"""Initialize all of the parameters.
Parameters
----------
target_space : TargetSpace
TargetSpace this DomainTransformer operates on.
"""
if isinstance(self.minimum_window_value, Mapping):
self.minimum_window_value = [self.minimum_window_value[key] for key in target_space.keys]
else:
self.minimum_window_value = self.minimum_window_value
any_not_float = any([not isinstance(p, FloatParameter) for p in target_space._params_config.values()])
if any_not_float:
msg = "Domain reduction is only supported for all-FloatParameter optimization."
raise ValueError(msg)
# Set the original bounds
self.original_bounds = np.copy(target_space.bounds)
self.bounds = [self.original_bounds]
self.minimum_window: NDArray[Float] | Sequence[float]
# Set the minimum window to an array of length bounds
if isinstance(self.minimum_window_value, (Sequence, np.ndarray)):
if len(self.minimum_window_value) != len(target_space.bounds):
error_msg = "Length of minimum_window must be the same as the number of parameters"
raise ValueError(error_msg)
self.minimum_window = self.minimum_window_value
else:
self.minimum_window = [self.minimum_window_value] * len(target_space.bounds)
# Set initial values
self.previous_optimal = np.mean(target_space.bounds, axis=1)
self.current_optimal = np.mean(target_space.bounds, axis=1)
self.r = target_space.bounds[:, 1] - target_space.bounds[:, 0]
self.previous_d = 2.0 * (self.current_optimal - self.previous_optimal) / self.r
self.current_d = 2.0 * (self.current_optimal - self.previous_optimal) / self.r
self.c = self.current_d * self.previous_d
self.c_hat = np.sqrt(np.abs(self.c)) * np.sign(self.c)
self.gamma = 0.5 * (self.gamma_pan * (1.0 + self.c_hat) + self.gamma_osc * (1.0 - self.c_hat))
self.contraction_rate = self.eta + np.abs(self.current_d) * (self.gamma - self.eta)
self.r = self.contraction_rate * self.r
# check if the minimum window fits in the original bounds
self._window_bounds_compatibility(self.original_bounds)
def _update(self, target_space: TargetSpace) -> None:
"""Update contraction rate, window size, and window center.
Parameters
----------
target_space : TargetSpace
TargetSpace this DomainTransformer operates on.
"""
# setting the previous
self.previous_optimal = self.current_optimal
self.previous_d = self.current_d
self.current_optimal = target_space.params_to_array(target_space.max()["params"])
self.current_d = 2.0 * (self.current_optimal - self.previous_optimal) / self.r
self.c = self.current_d * self.previous_d
self.c_hat = np.sqrt(np.abs(self.c)) * np.sign(self.c)
self.gamma = 0.5 * (self.gamma_pan * (1.0 + self.c_hat) + self.gamma_osc * (1.0 - self.c_hat))
self.contraction_rate = self.eta + np.abs(self.current_d) * (self.gamma - self.eta)
self.r = self.contraction_rate * self.r
def _trim(self, new_bounds: NDArray[Float], global_bounds: NDArray[Float]) -> NDArray[Float]:
"""
Adjust the new_bounds and verify that they adhere to global_bounds and minimum_window.
Parameters
----------
new_bounds : np.ndarray
The proposed new_bounds that (may) need adjustment.
global_bounds : np.ndarray
The maximum allowable bounds for each parameter.
Returns
-------
new_bounds : np.ndarray
The adjusted bounds after enforcing constraints.
"""
# sort bounds
new_bounds = np.sort(new_bounds)
pbounds: NDArray[Float]
# Validate each parameter's bounds against the global_bounds
for i, pbounds in enumerate(new_bounds):
# If the one of the bounds is outside the global bounds, reset the bound to the global bound
# This is expected to happen when the window is near the global bounds, no warning is issued
if pbounds[0] < global_bounds[i, 0]:
pbounds[0] = global_bounds[i, 0]
if pbounds[1] > global_bounds[i, 1]:
pbounds[1] = global_bounds[i, 1]
# If a lower bound is greater than the associated global upper bound,
# reset it to the global lower bound
if pbounds[0] > global_bounds[i, 1]:
pbounds[0] = global_bounds[i, 0]
warn(
"\nDomain Reduction Warning:\n"
"A parameter's lower bound is greater than the global upper bound."
"The offensive boundary has been reset."
"Be cautious of subsequent reductions.",
stacklevel=2,
)
# If an upper bound is less than the associated global lower bound,
# reset it to the global upper bound
if pbounds[1] < global_bounds[i, 0]:
pbounds[1] = global_bounds[i, 1]
warn(
"\nDomain Reduction Warning:\n"
"A parameter's lower bound is greater than the global upper bound."
"The offensive boundary has been reset."
"Be cautious of subsequent reductions.",
stacklevel=2,
)
# Adjust new_bounds to ensure they respect the minimum window width for each parameter
for i, pbounds in enumerate(new_bounds):
current_window_width = abs(pbounds[0] - pbounds[1])
# If the window width is less than the minimum allowable width, adjust it
# Note that when minimum_window < width of the global bounds one side
# always has more space than required
if current_window_width < self.minimum_window[i]:
width_deficit = (self.minimum_window[i] - current_window_width) / 2.0
available_left_space = abs(global_bounds[i, 0] - pbounds[0])
available_right_space = abs(global_bounds[i, 1] - pbounds[1])
# determine how much to expand on the left and right
expand_left = min(width_deficit, available_left_space)
expand_right = min(width_deficit, available_right_space)
# calculate the deficit on each side
expand_left_deficit = width_deficit - expand_left
expand_right_deficit = width_deficit - expand_right
# shift the deficit to the side with more space
adjust_left = expand_left + max(expand_right_deficit, 0)
adjust_right = expand_right + max(expand_left_deficit, 0)
# adjust the bounds
pbounds[0] -= adjust_left
pbounds[1] += adjust_right
return new_bounds
def _window_bounds_compatibility(self, global_bounds: NDArray[Float]) -> None:
"""Check if global bounds are compatible with the minimum window sizes.
Parameters
----------
global_bounds : np.ndarray
The maximum allowable bounds for each parameter.
Raises
------
ValueError
If global bounds are not compatible with the minimum window size.
"""
entry: NDArray[Float]
for i, entry in enumerate(global_bounds):
global_window_width = abs(entry[1] - entry[0])
if global_window_width < self.minimum_window[i]:
error_msg = "Global bounds are not compatible with the minimum window size."
raise ValueError(error_msg)
def _create_bounds(self, parameters: Iterable[str], bounds: NDArray[Float]) -> dict[str, NDArray[Float]]:
"""Create a dictionary of bounds for each parameter.
Parameters
----------
parameters : Iterable[str]
The parameters for which to create the bounds.
bounds : np.ndarray
The bounds for each parameter.
"""
return {param: bounds[i, :] for i, param in enumerate(parameters)}
def transform(self, target_space: TargetSpace) -> dict[str, NDArray[Float]]:
"""Transform the bounds of the target space.
Parameters
----------
target_space : TargetSpace
TargetSpace this DomainTransformer operates on.
Returns
-------
dict
The new bounds of each parameter.
"""
self._update(target_space)
new_bounds = np.array([self.current_optimal - 0.5 * self.r, self.current_optimal + 0.5 * self.r]).T
new_bounds = self._trim(new_bounds, self.original_bounds)
self.bounds.append(new_bounds)
return self._create_bounds(target_space.keys, new_bounds)
================================================
FILE: bayes_opt/exception.py
================================================
"""This module contains custom exceptions for Bayesian Optimization."""
from __future__ import annotations
__all__ = [
"BayesianOptimizationError",
"ConstraintNotSupportedError",
"NoValidPointRegisteredError",
"NotUniqueError",
"TargetSpaceEmptyError",
]
class BayesianOptimizationError(Exception):
"""Base class for exceptions in the Bayesian Optimization."""
class NotUniqueError(BayesianOptimizationError):
"""A point is non-unique."""
class ConstraintNotSupportedError(BayesianOptimizationError):
"""Raised when constrained optimization is not supported."""
class NoValidPointRegisteredError(BayesianOptimizationError):
"""Raised when an acquisition function depends on previous points but none are registered."""
class TargetSpaceEmptyError(BayesianOptimizationError):
"""Raised when the target space is empty."""
================================================
FILE: bayes_opt/logger.py
================================================
"""Contains classes and functions for logging."""
from __future__ import annotations
from collections.abc import Mapping
from typing import TYPE_CHECKING, Any
from colorama import Fore, just_fix_windows_console
if TYPE_CHECKING:
from bayes_opt.parameter import ParamsType
just_fix_windows_console()
class ScreenLogger:
"""Logger that outputs text, e.g. to log to a terminal.
Parameters
----------
verbose : int
Verbosity level of the logger.
is_constrained : bool
Whether the logger is associated with a constrained optimization
instance.
"""
_default_cell_size = 9
_default_precision = 4
_color_new_max = Fore.MAGENTA
_color_regular_message = Fore.RESET
_color_reset = Fore.RESET
def __init__(self, verbose: int = 2, is_constrained: bool = False) -> None:
self._verbose = verbose
self._is_constrained = is_constrained
self._header_length = None
self._iterations = 0
self._previous_max = None
self._previous_max_params = None
self._start_time = None
self._previous_time = None
@property
def verbose(self) -> int:
"""Return the verbosity level."""
return self._verbose
@verbose.setter
def verbose(self, v: int) -> None:
"""Set the verbosity level.
Parameters
----------
v : int
New verbosity level of the logger.
"""
self._verbose = v
@property
def is_constrained(self) -> bool:
"""Return whether the logger is constrained."""
return self._is_constrained
def _format_number(self, x: float) -> str:
"""Format a number.
Parameters
----------
x : number
Value to format.
Returns
-------
A stringified, formatted version of `x`.
"""
s = f"{x:.5e}" if abs(x) >= 10000000.0 else str(x)
if len(s) > self._default_cell_size:
# Convert to str representation of scientific notation
result = ""
width = self._default_cell_size
# Keep negative sign, exponent, and as many decimal places as possible
if x < 0:
result += "-"
width -= 1
s = s[1:]
if "e" in s:
e_pos = s.find("e")
end = s[e_pos:]
width -= len(end)
if "." in s:
dot_pos = s.find(".") + 1
result += s[:dot_pos]
width -= dot_pos
if width > 0:
result += s[dot_pos : dot_pos + width]
if "e" in s:
result += end
result = result.ljust(self._default_cell_size)
else:
result = s.ljust(self._default_cell_size)
return result
def _format_bool(self, x: bool) -> str:
"""Format a boolean.
Parameters
----------
x : boolean
Value to format.
Returns
-------
A stringified, formatted version of `x`.
"""
x_ = ("T" if x else "F") if self._default_cell_size < 5 else str(x)
return f"{x_:<{self._default_cell_size}}"
def _format_str(self, str_: str) -> str:
"""Format a str.
Parameters
----------
str_ : str
Value to format.
Returns
-------
A stringified, formatted version of `x`.
"""
s = f"{str_:^{self._default_cell_size}}"
if len(s) > self._default_cell_size:
return s[: self._default_cell_size - 3] + "..."
return s
def _print_step(
self,
keys: list[str],
result: dict[str, Any],
params_config: Mapping[str, ParamsType],
color: str = _color_regular_message,
) -> str:
"""Print a step.
Parameters
----------
result : dict[str, Any]
The result dictionary for the most recent step.
keys : list[str]
The parameter keys.
params_config : Mapping[str, ParamsType]
The configuration to map the key to the parameter for correct formatting.
color : str, optional
Color to use for the output.
(Default value = _color_regular_message, equivalent to Fore.RESET)
Returns
-------
A stringified, formatted version of the most recent optimization step.
"""
# iter, target, allowed [, *params]
cells: list[str | None] = [None] * (3 + len(keys))
cells[:2] = self._format_number(self._iterations), self._format_number(result["target"])
if self._is_constrained:
cells[2] = self._format_bool(result["allowed"])
params = result.get("params", {})
cells[3:] = [
self._format_number(val)
if isinstance(val, (int, float))
else params_config[key].to_string(val, self._default_cell_size)
for key, val in params.items()
]
return "| " + " | ".join(color + x + self._color_reset for x in cells if x is not None) + " |"
def _print_header(self, keys: list[str]) -> str:
"""Print the header of the log.
Parameters
----------
keys : list[str]
The parameter keys.
Returns
-------
A stringified, formatted version of the most header.
"""
# iter, target, allowed [, *params]
cells: list[str | None] = [None] * (3 + len(keys))
cells[:2] = self._format_str("iter"), self._format_str("target")
if self._is_constrained:
cells[2] = self._format_str("allowed")
cells[3:] = [self._format_str(key) for key in keys]
line = "| " + " | ".join(x for x in cells if x is not None) + " |"
self._header_length = len(line)
return line + "\n" + ("-" * self._header_length)
def _is_new_max(self, current_max: dict[str, Any] | None) -> bool:
"""Check if the step to log produced a new maximum.
Parameters
----------
current_max : dict[str, Any] | None
The current maximum target value and its parameters.
Returns
-------
boolean
"""
if current_max is None:
# During constrained optimization, there might not be a maximum
# value since the optimizer might've not encountered any points
# that fulfill the constraints.
return False
if self._previous_max is None:
self._previous_max = current_max["target"]
return current_max["target"] > self._previous_max
def _update_tracker(self, current_max: dict[str, Any] | None) -> None:
"""Update the tracker.
Parameters
----------
current_max : dict[str, Any] | None
The current maximum target value and its parameters.
"""
self._iterations += 1
if current_max is None:
return
if self._previous_max is None or current_max["target"] > self._previous_max:
self._previous_max = current_max["target"]
self._previous_max_params = current_max["params"]
def log_optimization_start(self, keys: list[str]) -> None:
"""Log the start of the optimization process.
Parameters
----------
keys : list[str]
The parameter keys.
"""
if self._verbose:
line = self._print_header(keys) + "\n"
print(line, end="")
def log_optimization_step(
self,
keys: list[str],
result: dict[str, Any],
params_config: Mapping[str, ParamsType],
current_max: dict[str, Any] | None,
) -> None:
"""Log an optimization step.
Parameters
----------
keys : list[str]
The parameter keys.
result : dict[str, Any]
The result dictionary for the most recent step.
params_config : Mapping[str, ParamsType]
The configuration to map the key to the parameter for correct formatting.
current_max : dict[str, Any] | None
The current maximum target value and its parameters.
"""
is_new_max = self._is_new_max(current_max)
self._update_tracker(current_max)
if self._verbose == 0:
return
if self._verbose == 2 or is_new_max:
color = self._color_new_max if is_new_max else self._color_regular_message
line = self._print_step(keys, result, params_config, color=color) + "\n"
if self._verbose:
print(line, end="")
def log_optimization_end(self) -> None:
"""Log the end of the optimization process."""
if self._verbose and self._header_length is not None:
line = "=" * self._header_length + "\n"
print(line, end="")
================================================
FILE: bayes_opt/parameter.py
================================================
"""Parameter classes for Bayesian optimization."""
from __future__ import annotations
import abc
from collections.abc import Sequence
from inspect import signature
from numbers import Number
from typing import TYPE_CHECKING, Any, Callable, Union
import numpy as np
from sklearn.gaussian_process import kernels
from bayes_opt.util import ensure_rng
if TYPE_CHECKING:
from collections.abc import Mapping
from numpy.typing import NDArray
Float = np.floating[Any]
Int = np.integer[Any]
FloatBoundsWithoutType = tuple[float, float]
FloatBoundsWithType = tuple[float, float, type[float]]
FloatBounds = Union[FloatBoundsWithoutType, FloatBoundsWithType]
IntBounds = tuple[Union[int, float], Union[int, float], type[int]]
CategoricalBounds = Sequence[Any]
Bounds = Union[FloatBounds, IntBounds, CategoricalBounds]
BoundsMapping = Mapping[str, Bounds]
# FIXME: categorical parameters can be of any type.
# This will make static type checking for parameters difficult.
ParamsType = Union[Mapping[str, Any], Sequence[Any], NDArray[Float]]
def is_numeric(value: Any) -> bool:
"""Check if a value is numeric."""
return isinstance(value, Number) or (
isinstance(value, np.generic)
and (np.isdtype(value.dtype, np.number) or np.issubdtype(value.dtype, np.number))
)
class BayesParameter(abc.ABC):
"""Base class for Bayesian optimization parameters.
Parameters
----------
name : str
The name of the parameter.
"""
def __init__(self, name: str, bounds: NDArray[Any]) -> None:
self.name = name
self._bounds = bounds
@property
def bounds(self) -> NDArray[Any]:
"""The bounds of the parameter in float space."""
return self._bounds
@property
@abc.abstractmethod
def is_continuous(self) -> bool:
"""Whether the parameter is continuous."""
def random_sample(
self, n_samples: int, random_state: np.random.RandomState | int | None
) -> NDArray[Float]:
"""Generate random samples from the parameter.
Parameters
----------
n_samples : int
The number of samples to generate.
random_state : np.random.RandomState | int | None
The random state to use for sampling.
Returns
-------
np.ndarray
The samples.
"""
random_state = ensure_rng(random_state)
return random_state.uniform(self.bounds[0], self.bounds[1], n_samples)
@abc.abstractmethod
def to_float(self, value: Any) -> float | NDArray[Float]:
"""Convert a parameter value to a float.
Parameters
----------
value : Any
The value to convert, should be the canonical representation of the parameter.
"""
@abc.abstractmethod
def to_param(self, value: float | NDArray[Float]) -> Any:
"""Convert a float value to a parameter.
Parameters
----------
value : np.ndarray
The value to convert, should be a float.
Returns
-------
Any
The canonical representation of the parameter.
"""
@abc.abstractmethod
def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
"""Transform a parameter value for use in a kernel.
Parameters
----------
value : np.ndarray
The value(s) to transform, should be a float.
Returns
-------
np.ndarray
"""
def to_string(self, value: Any, str_len: int) -> str:
"""Represent a parameter value as a string.
Parameters
----------
value : Any
The value to represent.
str_len : int
The maximum length of the string representation.
Returns
-------
str
"""
s = f"{value:<{str_len}}"
if len(s) > str_len:
return s[: str_len - 3] + "..."
return s
@property
@abc.abstractmethod
def dim(self) -> int:
"""The dimensionality of the parameter."""
class FloatParameter(BayesParameter):
"""A parameter with float values.
Parameters
----------
name : str
The name of the parameter.
bounds : tuple[float, float]
The bounds of the parameter.
"""
def __init__(self, name: str, bounds: tuple[float, float]) -> None:
super().__init__(name, np.array(bounds))
@property
def is_continuous(self) -> bool:
"""Whether the parameter is continuous."""
return True
def to_float(self, value: float) -> float:
"""Convert a parameter value to a float.
Parameters
----------
value : Any
The value to convert, should be the canonical representation of the parameter.
"""
return value
def to_param(self, value: float | NDArray[Float]) -> float:
"""Convert a float value to a parameter.
Parameters
----------
value : np.ndarray
The value to convert, should be a float.
Returns
-------
Any
The canonical representation of the parameter.
"""
return value.flatten()[0]
def to_string(self, value: float, str_len: int) -> str:
"""Represent a parameter value as a string.
Parameters
----------
value : Any
The value to represent.
str_len : int
The maximum length of the string representation.
Returns
-------
str
"""
s = f"{value:<{str_len}.{str_len}}"
if len(s) > str_len:
if "." in s and "e" not in s:
return s[:str_len]
return s[: str_len - 3] + "..."
return s
def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
"""Transform a parameter value for use in a kernel.
Parameters
----------
value : np.ndarray
The value(s) to transform, should be a float.
Returns
-------
np.ndarray
"""
return value
@property
def dim(self) -> int:
"""The dimensionality of the parameter."""
return 1
class IntParameter(BayesParameter):
"""A parameter with int values.
Parameters
----------
name : str
The name of the parameter.
bounds : tuple[int, int]
The bounds of the parameter.
"""
def __init__(self, name: str, bounds: tuple[int, int]) -> None:
super().__init__(name, np.array(bounds))
@property
def is_continuous(self) -> bool:
"""Whether the parameter is continuous."""
return False
def random_sample(
self, n_samples: int, random_state: np.random.RandomState | int | None
) -> NDArray[Float]:
"""Generate random samples from the parameter.
Parameters
----------
n_samples : int
The number of samples to generate.
random_state : np.random.RandomState | int | None
The random state to use for sampling.
Returns
-------
np.ndarray
The samples.
"""
random_state = ensure_rng(random_state)
return random_state.randint(self.bounds[0], self.bounds[1] + 1, n_samples).astype(float)
def to_float(self, value: int | float) -> float:
"""Convert a parameter value to a float.
Parameters
----------
value : Any
The value to convert, should be the canonical representation of the parameter.
"""
return float(value)
def to_param(self, value: int | float | NDArray[Int] | NDArray[Float]) -> int:
"""Convert a float value to a parameter.
Parameters
----------
value : np.ndarray
The value to convert, should be a float.
Returns
-------
Any
The canonical representation of the parameter.
"""
return int(np.round(np.squeeze(value)))
def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
"""Transform a parameter value for use in a kernel.
Parameters
----------
value : np.ndarray
The value(s) to transform, should be a float.
Returns
-------
np.ndarray
"""
return np.round(value)
@property
def dim(self) -> int:
"""The dimensionality of the parameter."""
return 1
class CategoricalParameter(BayesParameter):
"""A parameter with categorical values.
Parameters
----------
name : str
The name of the parameter.
categories : Sequence[Any]
The categories of the parameter.
"""
def __init__(self, name: str, categories: Sequence[Any]) -> None:
if len(categories) != len(set(categories)):
msg = "Categories must be unique."
raise ValueError(msg)
if len(categories) < 2:
msg = "At least two categories are required."
raise ValueError(msg)
self.categories = categories
lower = np.zeros(self.dim)
upper = np.ones(self.dim)
bounds = np.vstack((lower, upper)).T
super().__init__(name, bounds)
@property
def is_continuous(self) -> bool:
"""Whether the parameter is continuous."""
return False
def random_sample(
self, n_samples: int, random_state: np.random.RandomState | int | None
) -> NDArray[Float]:
"""Generate random float-format samples from the parameter.
Parameters
----------
n_samples : int
The number of samples to generate.
random_state : np.random.RandomState | int | None
The random state to use for sampling.
Returns
-------
np.ndarray
The samples.
"""
random_state = ensure_rng(random_state)
res = random_state.randint(0, len(self.categories), n_samples)
one_hot = np.zeros((n_samples, len(self.categories)))
one_hot[np.arange(n_samples), res] = 1
return one_hot.astype(float)
def to_float(self, value: Any) -> NDArray[Float]:
"""Convert a parameter value to a float.
Parameters
----------
value : Any
The value to convert, should be the canonical representation of the parameter.
"""
res = np.zeros(len(self.categories))
one_hot_index = [i for i, val in enumerate(self.categories) if val == value]
res[one_hot_index] = 1
return res.astype(float)
def to_param(self, value: float | NDArray[Float]) -> Any:
"""Convert a float value to a parameter.
Parameters
----------
value : np.ndarray
The value to convert, should be a float.
Returns
-------
Any
The canonical representation of the parameter.
"""
return self.categories[int(np.argmax(value))]
def to_string(self, value: Any, str_len: int) -> str:
"""Represent a parameter value as a string.
Parameters
----------
value : Any
The value to represent.
str_len : int
The maximum length of the string representation.
Returns
-------
str
"""
if not isinstance(value, str):
value = repr(value)
s = f"{value:<{str_len}}"
if len(s) > str_len:
return s[: str_len - 3] + "..."
return s
def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
"""Transform a parameter value for use in a kernel.
Parameters
----------
value : np.ndarray
The value(s) to transform, should be a float.
Returns
-------
np.ndarray
"""
value = np.atleast_2d(value)
res = np.zeros(value.shape)
res[:, np.argmax(value, axis=1)] = 1
return res
@property
def dim(self) -> int:
"""The dimensionality of the parameter."""
return len(self.categories)
def wrap_kernel(kernel: kernels.Kernel, transform: Callable[[Any], Any]) -> kernels.Kernel:
"""Wrap a kernel to transform input data before passing it to the kernel.
Parameters
----------
kernel : kernels.Kernel
The kernel to wrap.
transform : Callable
The transformation function to apply to the input data.
Returns
-------
kernels.Kernel
The wrapped kernel.
Notes
-----
See https://arxiv.org/abs/1805.03463 for more information.
"""
kernel_type = type(kernel)
class WrappedKernel(kernel_type):
@_copy_signature(getattr(kernel_type.__init__, "deprecated_original", kernel_type.__init__))
def __init__(self, **kwargs: Any) -> None:
super().__init__(**kwargs)
def __call__(self, X: Any, Y: Any = None, eval_gradient: bool = False) -> Any:
X = transform(X)
Y = transform(Y) if Y is not None else None
return super().__call__(X, Y, eval_gradient)
def __reduce__(self) -> str | tuple[Any, ...]:
return (wrap_kernel, (kernel, transform))
wrapped_instance = WrappedKernel.__new__(WrappedKernel)
wrapped_instance.__dict__.update(kernel.__dict__)
wrapped_instance._transform = transform
return wrapped_instance
def _copy_signature(source_fct: Callable[..., Any]) -> Callable[[Callable[..., Any]], Callable[..., Any]]:
"""Clone a signature from a source function to a target function.
via
https://stackoverflow.com/a/58989918/
"""
def copy(target_fct: Callable[..., Any]) -> Callable[..., Any]:
target_fct.__signature__ = signature(source_fct)
return target_fct
return copy
================================================
FILE: bayes_opt/py.typed
================================================
================================================
FILE: bayes_opt/target_space.py
================================================
"""Manages the optimization domain and holds points."""
from __future__ import annotations
from copy import deepcopy
from typing import TYPE_CHECKING, Any
from warnings import warn
import numpy as np
from colorama import Fore
from bayes_opt.constraint import ConstraintModel
from bayes_opt.exception import NotUniqueError
from bayes_opt.parameter import BayesParameter, CategoricalParameter, FloatParameter, IntParameter, is_numeric
from bayes_opt.util import ensure_rng
if TYPE_CHECKING:
from collections.abc import Callable, Mapping
from numpy.random import RandomState
from numpy.typing import NDArray
from scipy.optimize import NonlinearConstraint
from bayes_opt.parameter import BoundsMapping, ParamsType
Float = np.floating[Any]
Int = np.integer[Any]
def _hashable(x: NDArray[Float]) -> tuple[float, ...]:
"""Ensure that a point is hashable by a python dict."""
return tuple(map(float, x))
class TargetSpace:
"""Holds the param-space coordinates (X) and target values (Y).
Allows for constant-time appends.
Parameters
----------
target_func : function or None.
Function to be maximized.
pbounds : dict
Dictionary with parameters names as keys and a tuple with minimum
and maximum values.
random_state : int, RandomState, or None
optionally specify a seed for a random number generator
allow_duplicate_points: bool, optional (default=False)
If True, the optimizer will allow duplicate points to be registered.
This behavior may be desired in high noise situations where repeatedly probing
the same point will give different answers. In other situations, the acquisition
may occasionally generate a duplicate point.
Examples
--------
>>> def target_func(p1, p2):
>>> return p1 + p2
>>> pbounds = {"p1": (0, 1), "p2": (1, 100)}
>>> space = TargetSpace(target_func, pbounds, random_state=0)
>>> x = np.array([4, 5])
>>> y = target_func(x)
>>> space.register(x, y)
>>> assert self.max()["target"] == 9
>>> assert self.max()["params"] == {"p1": 1.0, "p2": 2.0}
"""
def __init__(
self,
target_func: Callable[..., float] | None,
pbounds: BoundsMapping,
constraint: NonlinearConstraint | None = None,
random_state: int | RandomState | None = None,
allow_duplicate_points: bool | None = False,
) -> None:
self._allow_duplicate_points = allow_duplicate_points or False
self.n_duplicate_points = 0
# The function to be optimized
self.target_func = target_func
# Get the name of the parameters
self._keys: list[str] = list(pbounds.keys())
self._params_config = self.make_params(pbounds)
self._dim = sum([self._params_config[key].dim for key in self._keys])
self._masks = self.make_masks()
self._bounds = self.calculate_bounds()
# preallocated memory for X and Y points
self._params: NDArray[Float] = np.empty(shape=(0, self.dim))
self._target: NDArray[Float] = np.empty(shape=(0,))
# keep track of unique points we have seen so far
self._cache: dict[tuple[float, ...], float | tuple[float, float | NDArray[Float]]] = {}
self._constraint: ConstraintModel | None = None
if constraint is None:
self._constraint = None
else:
self._constraint = ConstraintModel(
constraint.fun,
constraint.lb,
constraint.ub,
transform=self.kernel_transform,
random_state=random_state,
)
# preallocated memory for constraint fulfillment
self._constraint_values: NDArray[Float]
if self._constraint.lb.size == 1:
self._constraint_values = np.empty(shape=(0), dtype=float)
else:
self._constraint_values = np.empty(shape=(0, self._constraint.lb.size), dtype=float)
def __contains__(self, x: NDArray[Float]) -> bool:
"""Check if this parameter has already been registered.
Returns
-------
bool
"""
return _hashable(x) in self._cache
def __len__(self) -> int:
"""Return number of observations registered.
Returns
-------
int
"""
return len(self._target)
@property
def empty(self) -> bool:
"""Check if anything has been registered.
Returns
-------
bool
"""
return len(self) == 0
@property
def params(self) -> NDArray[Float]:
"""Get the parameter values registered to this TargetSpace.
Returns
-------
np.ndarray
"""
return self._params
@property
def target(self) -> NDArray[Float]:
"""Get the target function values registered to this TargetSpace.
Returns
-------
np.ndarray
"""
return self._target
@property
def dim(self) -> int:
"""Get the number of parameter names.
Returns
-------
int
"""
return self._dim
@property
def keys(self) -> list[str]:
"""Get the keys (or parameter names).
Returns
-------
list of str
"""
return self._keys
@property
def params_config(self) -> dict[str, BayesParameter]:
"""Get the parameters configuration."""
return self._params_config
@property
def bounds(self) -> NDArray[Float]:
"""Get the bounds of this TargetSpace.
Returns
-------
np.ndarray
"""
return self._bounds
@property
def constraint(self) -> ConstraintModel | None:
"""Get the constraint model.
Returns
-------
ConstraintModel
"""
return self._constraint
@property
def masks(self) -> dict[str, NDArray[np.bool_]]:
"""Get the masks for the parameters.
Returns
-------
dict
"""
return self._masks
@property
def continuous_dimensions(self) -> NDArray[np.bool_]:
"""Get the continuous parameters.
Returns
-------
dict
"""
result = np.zeros(self.dim, dtype=bool)
masks = self.masks
for key in self.keys:
result[masks[key]] = self._params_config[key].is_continuous
return result
def make_params(self, pbounds: BoundsMapping) -> dict[str, BayesParameter]:
"""Create a dictionary of parameters from a dictionary of bounds.
Parameters
----------
pbounds : dict
A dictionary with the parameter names as keys and a tuple with minimum
and maximum values.
Returns
-------
dict
A dictionary with the parameter names as keys and the corresponding
parameter objects as values.
"""
any_is_not_float = False # TODO: remove in an upcoming release
params: dict[str, BayesParameter] = {}
for key in pbounds:
pbound = pbounds[key]
if isinstance(pbound, BayesParameter):
res = pbound
if not isinstance(pbound, FloatParameter):
any_is_not_float = True
elif (len(pbound) == 2 and is_numeric(pbound[0]) and is_numeric(pbound[1])) or (
len(pbound) == 3 and pbound[-1] is float
):
res = FloatParameter(name=key, bounds=(float(pbound[0]), float(pbound[1])))
elif len(pbound) == 3 and pbound[-1] is int:
res = IntParameter(name=key, bounds=(int(pbound[0]), int(pbound[1])))
any_is_not_float = True
else:
# assume categorical variable with pbound as list of possible values
res = CategoricalParameter(name=key, categories=pbound)
any_is_not_float = True
params[key] = res
if any_is_not_float:
msg = (
"Non-float parameters are experimental and may not work as expected."
" Exercise caution when using them and please report any issues you encounter."
)
warn(msg, stacklevel=4)
return params
def make_masks(self) -> dict[str, NDArray[np.bool_]]:
"""Create a dictionary of masks for the parameters.
The mask can be used to select the corresponding parameters from an array.
Returns
-------
dict
A dictionary with the parameter names as keys and the corresponding
mask as values.
"""
masks = {}
pos = 0
for key in self._keys:
mask = np.zeros(self._dim)
mask[pos : pos + self._params_config[key].dim] = 1
masks[key] = mask.astype(bool)
pos = pos + self._params_config[key].dim
return masks
def calculate_bounds(self) -> NDArray[Float]:
"""Calculate the float bounds of the parameter space."""
bounds = np.empty((self._dim, 2))
for key in self._keys:
bounds[self.masks[key]] = self._params_config[key].bounds
return bounds
def params_to_array(self, params: Mapping[str, float | NDArray[Float]]) -> NDArray[Float]:
"""Convert a dict representation of parameters into an array version.
Parameters
----------
params : dict
a single point, with len(x) == self.dim.
Returns
-------
np.ndarray
Representation of the parameters as an array.
"""
if set(params) != set(self.keys):
error_msg = f"Parameters' keys ({params}) do not match the expected set of keys ({self.keys})."
raise ValueError(error_msg)
return self._to_float(params)
@property
def constraint_values(self) -> NDArray[Float]:
"""Get the constraint values registered to this TargetSpace.
Returns
-------
np.ndarray
"""
if self._constraint is None:
error_msg = "TargetSpace belongs to an unconstrained optimization"
raise AttributeError(error_msg)
return self._constraint_values
def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
"""Transform floating-point suggestions to values used in the kernel.
Vectorized.
"""
value = np.atleast_2d(value)
res = [self._params_config[p].kernel_transform(value[:, self.masks[p]]) for p in self._keys]
return np.hstack(res)
def array_to_params(self, x: NDArray[Float]) -> dict[str, float | NDArray[Float]]:
"""Convert an array representation of parameters into a dict version.
Parameters
----------
x : np.ndarray
a single point, with len(x) == self.dim.
Returns
-------
dict
Representation of the parameters as dictionary.
"""
if len(x) != self._dim:
error_msg = (
f"Size of array ({len(x)}) is different than the expected number of parameters ({self._dim})."
)
raise ValueError(error_msg)
return self._to_params(x)
def _to_float(self, value: Mapping[str, float | NDArray[Float]]) -> NDArray[Float]:
if set(value) != set(self.keys):
msg = f"Parameters' keys ({value}) do not match the expected set of keys ({self.keys})."
raise ValueError(msg)
res = np.zeros(self._dim)
for key in self._keys:
p = self._params_config[key]
res[self.masks[key]] = p.to_float(value[key])
return res
def _to_params(self, value: NDArray[Float]) -> dict[str, float | NDArray[Float]]:
res: dict[str, float | NDArray[Float]] = {}
for key in self._keys:
p = self._params_config[key]
mask = self.masks[key]
res[key] = p.to_param(value[mask])
return res
@property
def mask(self) -> NDArray[np.bool_]:
"""Return a boolean array of valid points.
Points are valid if they satisfy both the constraint and boundary conditions.
Returns
-------
np.ndarray
"""
mask = np.ones_like(self.target, dtype=bool)
# mask points that don't satisfy the constraint
if self._constraint is not None:
mask &= self._constraint.allowed(self._constraint_values)
# mask points that are outside the bounds
if self._bounds is not None:
within_bounds = np.all(
(self._bounds[:, 0] <= self._params) & (self._params <= self._bounds[:, 1]), axis=1
)
mask &= within_bounds
return mask
def _as_array(self, x: Any) -> NDArray[Float]:
try:
x = np.asarray(x, dtype=float)
except TypeError:
x = self.params_to_array(x)
x = x.ravel()
if x.size != self.dim:
msg = f"Size of array ({len(x)}) is different than the expected number of ({self.dim})."
raise ValueError(msg)
return x
def register(
self, params: ParamsType, target: float, constraint_value: float | NDArray[Float] | None = None
) -> None:
"""Append a point and its target value to the known data.
Parameters
----------
params : np.ndarray
a single point, with len(x) == self.dim.
target : float
target function value
constraint_value : float or np.ndarray or None
Constraint function value
Raises
------
NotUniqueError:
if the point is not unique
Notes
-----
runs in amortized constant time
Examples
--------
>>> target_func = lambda p1, p2: p1 + p2
>>> pbounds = {"p1": (0, 1), "p2": (1, 100)}
>>> space = TargetSpace(target_func, pbounds)
>>> len(space)
0
>>> x = np.array([0, 0])
>>> y = 1
>>> space.register(x, y)
>>> len(space)
1
"""
x = self._as_array(params)
if x in self:
if self._allow_duplicate_points:
self.n_duplicate_points = self.n_duplicate_points + 1
print(
Fore.RED + f"Data point {x} is not unique. {self.n_duplicate_points}"
" duplicates registered. Continuing ..." + Fore.RESET
)
else:
error_msg = (
f"Data point {x} is not unique. You can set"
' "allow_duplicate_points=True" to avoid this error'
)
raise NotUniqueError(error_msg)
# if x is not within the bounds of the parameter space, warn the user
if self._bounds is not None and not np.all((self._bounds[:, 0] <= x) & (x <= self._bounds[:, 1])):
for key in self.keys:
if not np.all(
(self._params_config[key].bounds[..., 0] <= x[self.masks[key]])
& (x[self.masks[key]] <= self._params_config[key].bounds[..., 1])
):
msg = (
f"\nData point {x} is outside the bounds of the parameter {key}."
f"\n\tBounds:\n{self._params_config[key].bounds}"
)
warn(msg, stacklevel=2)
# Make copies of the data, so as not to modify the originals incase something fails
# during the registration process. This prevents out-of-sync data.
params_copy: NDArray[Float] = np.concatenate([self._params, x.reshape(1, -1)])
target_copy: NDArray[Float] = np.concatenate([self._target, [target]])
cache_copy = self._cache.copy() # shallow copy suffices
if self._constraint is None:
# Insert data into unique dictionary
cache_copy[_hashable(x.ravel())] = target
else:
if constraint_value is None:
msg = (
"When registering a point to a constrained TargetSpace"
" a constraint value needs to be present."
)
raise ValueError(msg)
# Insert data into unique dictionary
cache_copy[_hashable(x.ravel())] = (target, constraint_value)
constraint_values_copy: NDArray[Float] = np.concatenate(
[self._constraint_values, [constraint_value]]
)
self._constraint_values = constraint_values_copy
# Operations passed, update the variables
self._params = params_copy
self._target = target_copy
self._cache = cache_copy
def probe(self, params: ParamsType) -> float | tuple[float, float | NDArray[Float]]:
"""Evaluate the target function on a point and register the result.
Notes
-----
If `params` has been previously seen and duplicate points are not allowed,
returns a cached value of `result`.
Parameters
----------
params : np.ndarray
a single point, with len(x) == self.dim
Returns
-------
result : float | Tuple(float, float)
target function value, or Tuple(target function value, constraint value)
Example
-------
>>> target_func = lambda p1, p2: p1 + p2
>>> pbounds = {"p1": (0, 1), "p2": (1, 100)}
>>> space = TargetSpace(target_func, pbounds)
>>> space.probe([1, 5])
>>> assert self.max()["target"] == 6
>>> assert self.max()["params"] == {"p1": 1.0, "p2": 5.0}
"""
x = self._as_array(params)
if x in self and not self._allow_duplicate_points:
return self._cache[_hashable(x.ravel())]
dict_params = self.array_to_params(x)
if self.target_func is None:
error_msg = "No target function has been provided."
raise ValueError(error_msg)
target = self.target_func(**dict_params)
if self._constraint is None:
self.register(x, target)
return target
constraint_value = self._constraint.eval(**dict_params)
self.register(x, target, constraint_value)
return target, constraint_value
def random_sample(
self, n_samples: int = 0, random_state: np.random.RandomState | int | None = None
) -> NDArray[Float]:
"""
Sample a random point from within the bounds of the space.
Parameters
----------
n_samples : int, optional
Number of samples to draw. If 0, a single sample is drawn,
and a 1D array is returned. If n_samples > 0, an array of
shape (n_samples, dim) is returned.
random_state : np.random.RandomState | int | None
The random state to use for sampling.
Returns
-------
data: ndarray
[1 x dim] array with dimensions corresponding to `self._keys`
Examples
--------
>>> target_func = lambda p1, p2: p1 + p2
>>> pbounds = {"p1": (0, 1), "p2": (1, 100)}
>>> space = TargetSpace(target_func, pbounds, random_state=0)
>>> space.random_sample()
array([[ 0.54488318, 55.33253689]])
"""
random_state = ensure_rng(random_state)
flatten = n_samples == 0
n_samples = max(1, n_samples)
data = np.empty((n_samples, self._dim))
for key, mask in self.masks.items():
smpl = self._params_config[key].random_sample(n_samples, random_state)
data[:, mask] = smpl.reshape(n_samples, self._params_config[key].dim)
if flatten:
return data.ravel()
return data
def _target_max(self) -> float | None:
"""Get the maximum target value within the current parameter bounds.
If there is a constraint present, the maximum value that fulfills the
constraint within the parameter bounds is returned.
Returns
-------
max: float
The maximum target value.
"""
if len(self.target) == 0:
return None
if len(self.target[self.mask]) == 0:
return None
return self.target[self.mask].max()
def max(self) -> dict[str, Any] | None:
"""Get maximum target value found and corresponding parameters.
If there is a constraint present, the maximum value that fulfills the
constraint within the parameter bounds is returned.
Returns
-------
res: dict
A dictionary with the keys 'target' and 'params'. The value of
'target' is the maximum target value, and the value of 'params' is
a dictionary with the parameter names as keys and the parameter
values as values.
"""
target_max = self._target_max()
if target_max is None:
return None
target = self.target[self.mask]
params = self.params[self.mask]
target_max_idx = np.argmax(target)
res = {"target": target_max, "params": self.array_to_params(params[target_max_idx])}
if self._constraint is not None:
constraint_values = self.constraint_values[self.mask]
res["constraint"] = constraint_values[target_max_idx]
return res
def res(self) -> list[dict[str, Any]]:
"""Get all target values and constraint fulfillment for all parameters.
Returns
-------
res: list
A list of dictionaries with the keys 'target', 'params', and
'constraint'. The value of 'target' is the target value, the value
of 'params' is a dictionary with the parameter names as keys and the
parameter values as values, and the value of 'constraint' is the
constraint fulfillment.
Notes
-----
Does not report if points are within the bounds of the parameter space.
"""
if self._constraint is None:
params = [self.array_to_params(p) for p in self.params]
return [{"target": target, "params": param} for target, param in zip(self.target, params)]
params = [self.array_to_params(p) for p in self.params]
return [
{"target": target, "constraint": constraint_value, "params": param, "allowed": allowed}
for target, constraint_value, param, allowed in zip(
self.target,
self._constraint_values,
params,
self._constraint.allowed(self._constraint_values),
)
]
def set_bounds(self, new_bounds: BoundsMapping) -> None:
"""Change the lower and upper search bounds.
Parameters
----------
new_bounds : dict
A dictionary with the parameter name and its new bounds
"""
new_params_config = self.make_params(new_bounds)
dims = 0
params_config = deepcopy(self._params_config)
for key in self.keys:
if key in new_bounds:
if not isinstance(new_params_config[key], type(self._params_config[key])):
msg = (
f"Parameter type {type(new_params_config[key])} of"
" new bounds does not match parameter type"
f" {type(self._params_config[key])} of old bounds"
)
raise ValueError(msg)
params_config[key] = new_params_config[key]
dims = dims + params_config[key].dim
if dims != self.dim:
msg = f"Dimensions of new bounds ({dims}) does not match dimensions of old bounds ({self.dim})."
raise ValueError(msg)
self._params_config = params_config
self._bounds = self.calculate_bounds()
================================================
FILE: bayes_opt/util.py
================================================
"""Contains utility functions."""
from __future__ import annotations
import numpy as np
def ensure_rng(random_state: int | np.random.RandomState | None = None) -> np.random.RandomState:
"""Create a random number generator based on an optional seed.
Parameters
----------
random_state : np.random.RandomState or int or None, default=None
Random state to use. if `None`, will create an unseeded random state.
If `int`, creates a state using the argument as seed. If a
`np.random.RandomState` simply returns the argument.
Returns
-------
np.random.RandomState
"""
if random_state is None:
random_state = np.random.RandomState()
elif isinstance(random_state, int):
random_state = np.random.RandomState(random_state)
elif not isinstance(random_state, np.random.RandomState):
error_msg = "random_state should be an instance of np.random.RandomState, an int, or None."
raise TypeError(error_msg)
return random_state
================================================
FILE: docsrc/Makefile
================================================
# Minimal makefile for Sphinx documentation
#
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXOPTS ?=
SPHINXBUILD ?= sphinx-build
SOURCEDIR = .
BUILDDIR = ../docs
# Put it first so that "make" without argument is like "make help".
# help:
# @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
.PHONY: help Makefile
github:
# @cp ../README.md .
@make html
@cp -a ../docs/html/. ../docs
@cp -r ../docsrc/ ../docs
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
================================================
FILE: docsrc/conf.py
================================================
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
import time
import shutil
from glob import glob
from pathlib import Path
# sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('..'))
# copy the latest example files:
this_file_loc = Path(__file__).parent
notebooks = glob(str(this_file_loc.parent / 'examples' / '*.ipynb'))
for notebook in notebooks:
shutil.copy(notebook, this_file_loc)
# -- Project information -----------------------------------------------------
project = 'bayesian-optimization'
author = 'Fernando Nogueira'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.coverage',
'sphinx.ext.githubpages',
'nbsphinx',
'IPython.sphinxext.ipython_console_highlighting',
'sphinx.ext.mathjax',
"sphinx.ext.napoleon",
'sphinx_autodoc_typehints',
'sphinx.ext.intersphinx',
'sphinx_immaterial',
]
source_suffix = {
'.rst': 'restructuredtext',
}
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# Link types to the corresponding documentations
intersphinx_mapping = {
'python': ('https://docs.python.org/3', None),
'numpy': ('https://numpy.org/doc/stable', None),
'scipy': ('https://docs.scipy.org/doc/scipy/reference', None),
'sklearn': ('https://scikit-learn.org/stable', None),
}
napoleon_use_rtype = False
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
#
html_title = "Bayesian Optimization"
html_theme = "sphinx_immaterial"
copyright = f"{time.strftime('%Y')}, Fernando Nogueira and the bayesian-optimization developers"
# material theme options (see theme.conf for more information)
html_theme_options = {
"icon": {
"repo": "fontawesome/brands/github",
"edit": "material/file-edit-outline",
},
"site_url": "https://bayesian-optimization.github.io/BayesianOptimization/",
"repo_url": "https://github.com/bayesian-optimization/BayesianOptimization/",
"repo_name": "bayesian-optimization",
"edit_uri": "blob/master/docsrc",
"globaltoc_collapse": True,
"features": [
"navigation.expand",
# "navigation.tabs",
# "toc.integrate",
"navigation.sections",
# "navigation.instant",
# "header.autohide",
"navigation.top",
# "navigation.tracking",
# "search.highlight",
"search.share",
"toc.follow",
"toc.sticky",
"content.tabs.link",
"announce.dismiss",
],
"palette": [
{
"media": "(prefers-color-scheme: light)",
"scheme": "default",
"primary": "light-blue",
"accent": "light-green",
"toggle": {
"icon": "material/lightbulb-outline",
"name": "Switch to dark mode",
},
},
{
"media": "(prefers-color-scheme: dark)",
"scheme": "slate",
"primary": "deep-orange",
"accent": "lime",
"toggle": {
"icon": "material/lightbulb",
"name": "Switch to light mode",
},
},
],
# BEGIN: version_dropdown
"version_dropdown": True,
"version_json": '../versions.json',
# END: version_dropdown
"scope": "/", # share preferences across subsites
"toc_title_is_page_title": True,
# BEGIN: social icons
"social": [
{
"icon": "fontawesome/brands/github",
"link": "https://github.com/bayesian-optimization/BayesianOptimization",
"name": "Source on github.com",
},
{
"icon": "fontawesome/brands/python",
"link": "https://pypi.org/project/bayesian-optimization/",
},
{
"icon": "fontawesome/brands/python",
"link": "https://anaconda.org/conda-forge/bayesian-optimization",
}
],
# END: social icons
}
html_favicon = 'func.ico'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
# html_static_path = ['_static']
## extensions configuration
### sphinx-autodoc-typehints
typehints_use_signature = True
"""
If True, typehints for parameters in the signature are shown.
see more: https://github.com/tox-dev/sphinx-autodoc-typehints/blob/main/README.md#options
"""
typehints_use_signature_return = True
"""
If True, return annotations in the signature are shown.
see more: https://github.com/tox-dev/sphinx-autodoc-typehints/blob/main/README.md#options
"""
### autodoc
autodoc_typehints = "both"
"""
This value controls how to represent typehints. The setting takes the following values:
- `signature`: Show typehints in the signature
- `description`: Show typehints as content of the function or method
The typehints of overloaded functions or methods will still be represented in the signature.
- `none`: Do not show typehints
- `both`: Show typehints in the signature and as content of the function or method
Overloaded functions or methods will not have typehints included in the description
because it is impossible to accurately represent all possible overloads as a list of parameters.
see more: https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints
"""
autodoc_typehints_format = "short"
"""
This value controls the format of typehints. The setting takes the following values:
- `fully-qualified`: Show the module name and its name of typehints
- `short`: Suppress the leading module names of the typehints
(e.g. io.StringIO -> StringIO)
see more: https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html#confval-autodoc_typehints_format
"""
================================================
FILE: docsrc/index.rst
================================================
.. toctree::
:hidden:
Quickstart <self>
.. toctree::
:hidden:
:maxdepth: 3
:caption: Example Notebooks:
Basic Tour </basic-tour>
Advanced Tour </advanced-tour>
Constrained Bayesian Optimization </constraints>
Parameter Types </parameter_types>
Sequential Domain Reduction </domain_reduction>
Acquisition Functions </acquisition_functions>
Exploration vs. Exploitation </exploitation_vs_exploration>
Visualization of a 1D-Optimization </visualization>
.. toctree::
:hidden:
:maxdepth: 2
:caption: API reference:
reference/bayes_opt
reference/acquisition
reference/constraint
reference/domain_reduction
reference/target_space
reference/parameter
reference/exception
reference/other
.. raw:: html
<div align="center">
<img src="https://raw.githubusercontent.com/bayesian-optimization/BayesianOptimization/master/docsrc/static/func.png"><br><br>
</div>
Bayesian Optimization
=====================
|tests| |Codecov| |Pypi| |PyPI - Python Version|
Pure Python implementation of bayesian global optimization with gaussian
processes.
This is a constrained global optimization package built upon bayesian
inference and gaussian processes, that attempts to find the maximum value
of an unknown function in as few iterations as possible. This technique
is particularly suited for optimization of high cost functions and
situations where the balance between exploration and exploitation is
important.
Installation
------------
pip (via PyPI)
~~~~~~~~~~~~~~
.. code:: console
$ pip install bayesian-optimization
Conda (via conda-forge)
~~~~~~~~~~~~~~~~~~~~~~~
.. code:: console
$ conda install -c conda-forge bayesian-optimization
How does it work?
-----------------
Bayesian optimization works by constructing a posterior distribution of
functions (gaussian process) that best describes the function you want
to optimize. As the number of observations grows, the posterior
distribution improves, and the algorithm becomes more certain of which
regions in parameter space are worth exploring and which are not, as
seen in the picture below.
.. image:: ./static/bo_example.png
:alt: BayesianOptimization in action
As you iterate over and over, the algorithm balances its needs of
exploration and exploitation taking into account what it knows about the
target function. At each step a Gaussian Process is fitted to the known
samples (points previously explored), and the posterior distribution,
combined with a exploration strategy (such as UCB (Upper Confidence
Bound), or EI (Expected Improvement)), are used to determine the next
point that should be explored (see the gif below).
.. image:: ./static/bayesian_optimization.gif
:alt: BayesianOptimization in action
This process is designed to minimize the number of steps required to
find a combination of parameters that are close to the optimal
combination. To do so, this method uses a proxy optimization problem
(finding the maximum of the acquisition function) that, albeit still a
hard problem, is cheaper (in the computational sense) and common tools
can be employed. Therefore Bayesian Optimization is most adequate for
situations where sampling the function to be optimized is a very
expensive endeavor. See the references for a proper discussion of this
method.
This project is under active development, if you find a bug, or anything
that needs correction, please let us know by filing an
`issue on GitHub <https://github.com/bayesian-optimization/BayesianOptimization/issues>`__
.
Quick Index
-----------
See below for a quick tour over the basics of the Bayesian Optimization
package. More detailed information, other advanced features, and tips on
usage/implementation can be found in the
`examples <examples.html>`__
section. We suggest that you:
- Follow the `basic tour
notebook <basic-tour.html>`__
to learn how to use the package's most important features.
- Take a look at the `advanced tour
notebook <advanced-tour.html>`__
to learn how to make the package more flexible or how to use observers.
- To learn more about acquisition functions, a central building block
of bayesian optimization, see the `acquisition functions
notebook <acquisition_functions.html>`__
- If you want to optimize over integer-valued or categorical
parameters, see the `parameter types
notebook <parameter_types.html>`__.
- Check out this
`notebook <visualization.html>`__
with a step by step visualization of how this method works.
- To understand how to use bayesian optimization when additional
constraints are present, see the `constrained optimization
notebook <constraints.html>`__.
- Explore the `domain reduction
notebook <domain_reduction.html>`__
to learn more about how search can be sped up by dynamically changing
parameters' bounds.
- Explore this
`notebook <exploitation_vs_exploration.html>`__
exemplifying the balance between exploration and exploitation and how
to control it.
- Go over this
`script <https://github.com/bayesian-optimization/BayesianOptimization/blob/master/examples/sklearn_example.py>`__
for examples of how to tune parameters of Machine Learning models
using cross validation and bayesian optimization.
- Finally, take a look at this
`script <https://github.com/bayesian-optimization/BayesianOptimization/blob/master/examples/async_optimization.py>`__
for ideas on how to implement bayesian optimization in a distributed
fashion using this package.
Citation
--------
If you used this package in your research, please cite it:
::
@Misc{,
author={Fernando Nogueira},
title={{Bayesian Optimization}: Open source constrained global optimization tool for {Python}},
year={2014--},
url="https://github.com/bayesian-optimization/BayesianOptimization"
}
If you used any of the advanced functionalities, please additionally
cite the corresponding publication:
For the ``SequentialDomainTransformer``:
::
@article{
author={Stander, Nielen and Craig, Kenneth},
year={2002},
month={06},
pages={},
title={On the robustness of a simple domain reduction scheme for simulation-based optimization},
volume={19},
journal={International Journal for Computer-Aided Engineering and Software (Eng. Comput.)},
doi={10.1108/02644400210430190}
}
For constrained optimization:
::
@inproceedings{gardner2014bayesian,
title={Bayesian optimization with inequality constraints.},
author={Gardner, Jacob R and Kusner, Matt J and Xu, Zhixiang Eddie and Weinberger, Kilian Q and Cunningham, John P},
booktitle={ICML},
volume={2014},
pages={937--945},
year={2014}
}
For optimization over non-float parameters:
::
@article{garrido2020dealing,
title={Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes},
author={Garrido-Merch{\'a}n, Eduardo C and Hern{\'a}ndez-Lobato, Daniel},
journal={Neurocomputing},
volume={380},
pages={20--35},
year={2020},
publisher={Elsevier}
}
.. |tests| image:: https://github.com/bayesian-optimization/BayesianOptimization/actions/workflows/run_tests.yml/badge.svg
.. |Codecov| image:: https://codecov.io/github/bayesian-optimization/BayesianOptimization/badge.svg?branch=master&service=github
:target: https://codecov.io/github/bayesian-optimization/BayesianOptimization?branch=master
.. |Pypi| image:: https://img.shields.io/pypi/v/bayesian-optimization.svg
:target: https://pypi.python.org/pypi/bayesian-optimization
.. |PyPI - Python Version| image:: https://img.shields.io/pypi/pyversions/bayesian-optimization
================================================
FILE: docsrc/make.bat
================================================
@ECHO OFF
pushd %~dp0
REM Command file for Sphinx documentation
if "%SPHINXBUILD%" == "" (
set SPHINXBUILD=sphinx-build
)
set SOURCEDIR=.
set BUILDDIR=../docs
%SPHINXBUILD% >NUL 2>NUL
if errorlevel 9009 (
echo.
echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
echo.installed, then set the SPHINXBUILD environment variable to point
echo.to the full path of the 'sphinx-build' executable. Alternatively you
echo.may add the Sphinx directory to PATH.
echo.
echo.If you don't have Sphinx installed, grab it from
echo.https://www.sphinx-doc.org/
exit /b 1
)
if "%1" == "" goto help
if "%1" == "github" (
%SPHINXBUILD% -M html %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
robocopy %BUILDDIR%/html ../docs /E > nul
echo.Generated files copied to ../docs
goto end
)
%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
goto end
:help
%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
:end
popd
================================================
FILE: docsrc/reference/acquisition/ConstantLiar.rst
================================================
:py:class:`bayes_opt.acquisition.ConstantLiar`
----------------------------------------------
.. autoclass:: bayes_opt.acquisition.ConstantLiar
:members:
================================================
FILE: docsrc/reference/acquisition/ExpectedImprovement.rst
================================================
:py:class:`bayes_opt.acquisition.ExpectedImprovement`
-----------------------------------------------------
.. autoclass:: bayes_opt.acquisition.ExpectedImprovement
:members:
================================================
FILE: docsrc/reference/acquisition/GPHedge.rst
================================================
:py:class:`bayes_opt.acquisition.GPHedge`
-----------------------------------------
.. autoclass:: bayes_opt.acquisition.GPHedge
:members:
================================================
FILE: docsrc/reference/acquisition/ProbabilityOfImprovement.rst
================================================
:py:class:`bayes_opt.acquisition.ProbabilityOfImprovement`
----------------------------------------------------------
.. autoclass:: bayes_opt.acquisition.ProbabilityOfImprovement
:members:
================================================
FILE: docsrc/reference/acquisition/UpperConfidenceBound.rst
================================================
:py:class:`bayes_opt.acquisition.UpperConfidenceBound`
------------------------------------------------------
.. autoclass:: bayes_opt.acquisition.UpperConfidenceBound
:members:
================================================
FILE: docsrc/reference/acquisition.rst
================================================
:py:mod:`bayes_opt.acquisition`
-------------------------------
.. automodule:: bayes_opt.acquisition
:members: AcquisitionFunction
.. toctree::
:hidden:
acquisition/UpperConfidenceBound
acquisition/ProbabilityOfImprovement
acquisition/ExpectedImprovement
acquisition/GPHedge
acquisition/ConstantLiar
================================================
FILE: docsrc/reference/bayes_opt.rst
================================================
:py:class:`bayes_opt.BayesianOptimization`
------------------------------------------
.. autoclass:: bayes_opt.BayesianOptimization
:members:
================================================
FILE: docsrc/reference/constraint.rst
================================================
:py:class:`bayes_opt.ConstraintModel`
------------------------------------------------
See the `Constrained Optimization notebook <../constraints.html#2.-Advanced-Constrained-Optimization>`__ for a complete example.
.. autoclass:: bayes_opt.constraint.ConstraintModel
:members:
================================================
FILE: docsrc/reference/domain_reduction.rst
================================================
:py:class:`bayes_opt.SequentialDomainReductionTransformer`
----------------------------------------------------------
See the `Sequential Domain Reduction notebook <../domain_reduction.html>`__ for a complete example.
.. autoclass:: bayes_opt.SequentialDomainReductionTransformer
:members:
================================================
FILE: docsrc/reference/exception.rst
================================================
:py:mod:`bayes_opt.exception`
-------------------------------
.. automodule:: bayes_opt.exception
:members:
================================================
FILE: docsrc/reference/other.rst
================================================
Other
-----
.. autoclass:: bayes_opt.ScreenLogger
:members:
================================================
FILE: docsrc/reference/parameter.rst
================================================
:py:mod:`bayes_opt.parameter`
--------------------------------
.. automodule:: bayes_opt.parameter
:members:
================================================
FILE: docsrc/reference/target_space.rst
================================================
:py:class:`bayes_opt.TargetSpace`
---------------------------------
.. autoclass:: bayes_opt.TargetSpace
:members:
================================================
FILE: docsrc/requirements.txt
================================================
sphinx
nbsphinx
sphinx_rtd_theme
================================================
FILE: examples/acquisition_functions.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Acquisition functions\n",
"\n",
"An acquisition function dictates the exploration policy of the optimizer. In practice, there are many different acquisition policies and which one is the best might depend on your situation. This package comes with several acquisition functions out-of-the-box, but it is also very easy to write your acquisition function. This notebook showcases how to do that using various examples."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's begin with a few import statements and some function definitions we will use later."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from bayes_opt import BayesianOptimization\n",
"from bayes_opt import acquisition\n",
"import numpy as np\n",
"\n",
"import matplotlib.pyplot as plt\n",
"from matplotlib import gridspec\n",
"\n",
"SEED = 42"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"def target(x):\n",
" return np.exp(-(x - 2)**2) + np.exp(-(x - 6)**2/10) + 1/ (x**2 + 1)\n",
"\n",
"x = np.linspace(-2, 10, 1000).reshape(-1, 1)\n",
"y = target(x)\n",
"\n",
"def posterior(optimizer, grid):\n",
" mu, sigma = optimizer._gp.predict(grid, return_std=True)\n",
" return mu, sigma\n",
"\n",
"def plot_gp(optimizer, x, y):\n",
" acquisition_function_ = optimizer.acquisition_function\n",
" fig = plt.figure(figsize=(16, 10))\n",
" steps = len(optimizer.space)\n",
" fig.suptitle(\n",
" 'Gaussian Process and Utility Function After {} Steps'.format(steps),\n",
" fontsize=30\n",
" )\n",
"\n",
" gs = gridspec.GridSpec(2, 1, height_ratios=[3, 1])\n",
" axis = plt.subplot(gs[0])\n",
" acq = plt.subplot(gs[1])\n",
"\n",
" x_obs = np.array([[res[\"params\"][\"x\"]] for res in optimizer.res])\n",
" y_obs = np.array([res[\"target\"] for res in optimizer.res])\n",
"\n",
" acquisition_function_._fit_gp(optimizer._gp, optimizer._space)\n",
" mu, sigma = posterior(optimizer, x)\n",
"\n",
" axis.plot(x, y, linewidth=3, label='Target')\n",
" axis.plot(x_obs.flatten(), y_obs, 'D', markersize=8, label=u'Observations', color='r')\n",
" axis.plot(x, mu, '--', color='k', label='Prediction')\n",
"\n",
" axis.fill(np.concatenate([x, x[::-1]]),\n",
" np.concatenate([mu - 1.9600 * sigma, (mu + 1.9600 * sigma)[::-1]]),\n",
" alpha=.6, fc='c', ec='None', label='95% confidence interval')\n",
"\n",
" axis.set_xlim((-2, 10))\n",
" axis.set_ylim((None, None))\n",
" axis.set_ylabel('f(x)', fontdict={'size':20})\n",
" axis.set_xlabel('x', fontdict={'size':20})\n",
"\n",
" utility = -1 * acquisition_function_._get_acq(gp=optimizer._gp)(x)\n",
" x = x.flatten()\n",
"\n",
" acq.plot(x, utility, label='Utility Function', color='purple')\n",
" acq.plot(x[np.argmax(utility)], np.max(utility), '*', markersize=15,\n",
" label=u'Next Best Guess', markerfacecolor='gold', markeredgecolor='k', markeredgewidth=1)\n",
" acq.set_xlim((-2, 10))\n",
" #acq.set_ylim((0, np.max(utility) + 0.5))\n",
" acq.set_ylabel('Utility', fontdict={'size':20})\n",
" acq.set_xlabel('x', fontdict={'size':20})\n",
"\n",
" axis.legend(loc=2, bbox_to_anchor=(1.01, 1), borderaxespad=0.)\n",
" acq.legend(loc=2, bbox_to_anchor=(1.01, 1), borderaxespad=0.)\n",
" return fig, fig.axes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Most (simple) acquisition operate directly on the estimates of mean and standard deviation of the GP. For this, all it takes is to implement a `.base_acq` function, which combines the `mean`/`std` values into an acquisition policy. One very simple such function could simple disregard the `std` and always opt to investigate the maximum of the GP `mean`."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"class GreedyAcquisition(acquisition.AcquisitionFunction):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" def base_acq(self, mean, std):\n",
" return mean # disregard std"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"| iter | target | x |\n",
"-------------------------------------\n",
"| \u001b[39m1 \u001b[39m | \u001b[39m1.2141683\u001b[39m | \u001b[39m2.4944814\u001b[39m |\n",
"| \u001b[39m2 \u001b[39m | \u001b[39m0.3240816\u001b[39m | \u001b[39m9.4085716\u001b[39m |\n",
"| \u001b[39m3 \u001b[39m | \u001b[39m0.9616628\u001b[39m | \u001b[39m6.7839273\u001b[39m |\n",
"| \u001b[39m4 \u001b[39m | \u001b[39m1.2141034\u001b[39m | \u001b[39m2.4945789\u001b[39m |\n",
"| \u001b[35m5 \u001b[39m | \u001b[35m1.2213315\u001b[39m | \u001b[35m2.4836524\u001b[39m |\n",
"| \u001b[35m6 \u001b[39m | \u001b[35m1.3813338\u001b[39m | \u001b[35m2.1554093\u001b[39m |\n",
"| \u001b[35m7 \u001b[39m | \u001b[35m1.3904450\u001b[39m | \u001b[35m1.8852594\u001b[39m |\n",
"| \u001b[35m8 \u001b[39m | \u001b[35m1.4018873\u001b[39m | \u001b[35m2.0042391\u001b[39m |\n",
"| \u001b[39m9 \u001b[39m | \u001b[39m1.4018855\u001b[39m | \u001b[39m2.0045401\u001b[39m |\n",
"| \u001b[39m10 \u001b[39m | \u001b[39m1.4018849\u001b[39m | \u001b[39m2.0046339\u001b[39m |\n",
"=====================================\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAABh0AAAOzCAYAAACs2MdPAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjMsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvZiW1igAAAAlwSFlzAAAPYQAAD2EBqD+naQABAABJREFUeJzs3Qd0VNXWwPGdHgIk9N6VJiLFgmBDBRG72PXZ22fvXWnq02fH3rEXFLCL7akoggICCgrSSe+9l/nWPmTy7kxmkplkJjOT/H9rXci0O2faLWefs3eYzWazCQAAAAAAAAAAQDOFN3cFAAAAAAAAAAAAiqADAAAAAAAAAADwCYIOAAAAAAAAAADAJwg6AAAAAAAAAAAAnyDoAAAAAAAAAAAAfIKgAwAAAAAAAAAA8AmCDgAAAAAAAAAAwCcIOgAAAAAAAAAAAJ8g6AAAAAAAAAAAAHyCoAMAr82ePVvCwsLqlh9++CHQTQIAn5s8ebLDtg7/o9t963uj+4WGePNeertuIJixHQkd+fn58vDDD8uRRx4pvXv3lpiYGIfP7rXXXgt0EwEAAEJGZKAbgNYtPT1d1q1bJzt37pS8vDwpLy+XDh06SKdOnaR79+4yZswY6devX6CbCQAAAKCN0mDnKaecIjk5OYFuCnzAZrPJ77//LmvXrpWMjAxzXc+ePc255/jx4wMeANyxY4dpn54r6zmy0nNkPT8eMmSIDB8+XBISEgLaRgAAmougA3xu8+bN8tJLL8nixYtly5Ytjd5fD64OPfRQOeOMM+S4446Tdu3atUg7gZZ0wQUXyOuvv97gffQEyB6UGzx4sOy///5y9NFHmxF3gT45AuB7Omr2wgsvrLt82GGHNXnmmD7u8MMPr7s8cOBA06mB0OX8mTbVmjVrZOzYsT5pExBozz//vFxxxRX1jrHmz5/f5HXq+YqegxQXF/ughXA1g2TlypXy22+/meXXX3+VtLQ0h/ts375dBg0a1OznqqyslHnz5skTTzwhycnJLu+jA96uv/56ufbaayUqKkpairbnueeeM/t+d22z0+P+oUOHyoQJE2TatGlm6datW4u1FQAAXyC9EnxGD57OOeccGTFihJma7EnAQWVmZsrChQvl9NNPNyNQ7rnnnroRH0BbG5VVWFgoiYmJsnTpUnn00Udl6tSp5je1ZMmSQDcPQCuinXTWtCGhHKAgfQ38RX8X1u+W/m4QWK5SHH344YfNChjceeedDo/fZ5995Omnn5bPP/9cvvnmm7pFO34VKeAap0GF888/X0aOHCmdO3c2x7N33XWXfPzxx/UCDr6ix8/aSX/LLbc02KmflJQkN998s0ycOLHRzn9fBsv0eP7+++/36Dn1nOCff/6RN998U/71r3+Z968hbKsAAMGImQ7wCT0oP++889xOSW7fvr0ZnaGLpljSqaTZ2dlSU1PjcD/tcL3vvvvMgb6mZIqPj2+hVwAELz3pmD59ujmpnTVrVqCbAwAA0OI2bdpkRsk7KyoqMgOY9FzEWwUFBaYj3E7T2uhzxMbGNru9bZkGFt54440Wez5NoaQzw7Zu3epwvc6g13RFes6psynKysrqblu9erV5zC+//OLXWQQa1HrggQdc3tarVy/p0aOHaWdubq553/Q76SoIAQBAqCHogGbTERiaHqK6utrh+lGjRskll1xiUsOMHj263uMqKirMaO4vv/zSnChokMFOZzro7QhO2vnNqK7mOffcc+udHOsJhU5B//vvv+Wjjz4yuV6t9D3XKeEXX3xxC7cWAOrPLvBXJ4g/1x3KdDboW2+95fXj9txzT7+0B55paso01NdQIWdNYdmUoIN2PFvPOXQdBBz8S1OJaqDIl3RkvzXgoJ/hgw8+KJdeeqnExcWZ63Q2y4svvmiCAPbgg6YFvuiii+STTz4Rf3jnnXfqBRwGDBggt912m8yYMcMEHax036fZAjQQoufHX3/9tRmwBwBAKCLogGbRvJzaAWoNOGg++ieffNKkWgoPd5/BKzo6WqZMmWIWPSjUEwmdcmoNPgCtlY660u++O5pmTEeIXXbZZQ4nG3qSctpppzELCADaGO1Ea2i/AbRmOlJdBzpZZ1FrznstFKy+//572bVrl+nQ9XY2qfOgKfiO1kzQdFVap0yXAw44QPbaay+JiIjw2XNox7wOYrM+51dffWVqBlrpd+aGG24whaQ1XZHWf1Cffvqp+f74ooaOlQY2NNWTldZq02CCPRDirpaDLpqeSmc/vPDCC/Lzzz/7tG0AALQEajqgyTSVktZhsB+wKR2FrQdFOoq7oYCDMz041JEoOm3auTgc0FbpaDtNNWalaclacro6AABAoH377bcOufBPPvlkh5mfOkK8KcdHznXkGNThG1oUesWKFSZ17qpVq0wBZZ1RsPfee3t1jugJHahjdfvtt9cLOFgddthhZhCP1d133y3++M6mpKTUXdY0Su+9957bgIMrWg9DX89nn33m8/YBAOBvBB3QZJrqxTorQQMHOjW1OSOEYmJi5NlnnzUF4XQmBNDW6Qm15hd2HtEFAADQVlMraXHdM888UyIjIx1SLHnLmuNfUYzeN3TmuxZ11nM7f/rzzz/NzHvrbAbn2QWu3Hrrrea+dprOSNOb+pIWH7fS72tCQoJPnwMAgGBGeiU0iY62fuWVVxyu0/yY48aN88n6TznllCYVEFu/fr3J56mjlqqqqqRLly4mV6Ye9DrnzAxmpaWlsm7dOvnrr7/MtFq9rAXGdPSVjhwaMWKE9O/fv0mzU7ROgOYK1doB+h7paBstnjZ48GATMNIRNf6WlJQkGzZsMAXdtB1KP6u+ffvKxIkT/dIGfQ+1hsjGjRtNHll9Dn0vdbST9aQj2OjJ7zHHHGNmAVlPsLylKZqWLVtm3vvU1FQzrV2nuevrb+x3pSdiWthOvz96sqR5xfU31ZTvoDv6m12+fLlpW1ZWlkmjoCese+yxh4wZM8aMDmsqTbego+y0gL3+nvQ16PbgoIMOavZ2Qd8X/U3t2LHDFP7TdutvSturKbR0RJ/mLvaWpnvQbYC+HzpKUDtV9HuqvxF9TzQ1gbWjxZ/096K/V/3t6LZff0v6Hup2Q1MUDBs2zC/Pq69fPzf9DmqnhX5WkyZNMr/b5tJtjxYK1VGzOltP173ffvuZzwtAy9Dfnu5f9NhN9wF6jKP7Fd0v+eo4QLf7+lvX7YjuW3SEte5bdLs1duxY83coaKl9sXW/qc+n/+sMgu7du5v3S4/zA9Epr/tXrXVlp9tsTTWmxzLTpk2Tzz//3Fyvx7d6rKP7d08FW/0Y/ax1hoB+1rrP1WMIPabQtER6rO5rup/V902PNzQAM3DgQDn77LMlVFiLgCudhd+xY8dGH6f30XSl1mCWfsdGjhzps7bpMbeVnr+FIt0+6/ZAvyOZmZkm1Z9uE3R7oMejvqTnpvr91/2Cbuvs+wWt9dScgI0en+t59R9//GFegx5b6wBH/X3p+rX2kQ7y8vUsHABo82xAE9x77716hF63xMXF2fLy8lq0DTU1NbalS5farr76atvw4cMd2uNqGT16tO21116zVVZWevwchx12mMM6vDF//nyHx+rlxmzevNl27rnn2tq3b9/o6+nTp4/twgsvtC1fvrzR9X777be2qVOn2sLDwxtcZ1hYmG3EiBG22267zbZr1y6365s1a5bD477//vsGn1/f8y+//NJ28cUX2wYOHNhoGyZOnGhbvHix+Yw9df755zusZ/v27eZ6/V7ecMMNbt/T6Oho2xVXXGHLzMy0+ZNz+/Q99NRTTz3l8NgOHTp4/PoTExNtl112ma1Tp071XvuJJ57o9jn1/Z8wYYL5PBr7TVVXVzfpPdHHvfPOO7YDDzzQFhER0eB3Yvz48bZHH33UlpOT49G6y8vLbU888YRtr732anC9++23n+3jjz/2uu0LFiww39PGfqf6usaNG2ebM2eOLTs7u8F1lpWV2R588EHbHnvs0eh627VrZzviiCNszz33nM0fNm3aZJs9e7b5bCIjIxtsS69evczr8/SzUbrNcPd70O9EQ9t0/V7+9NNPTXpdv/zyi23SpElu1z1q1CjbwoULfbIPaMp+Qp+vqZzfU93WNvZ8ni6u1tXQZ+iKN++lJ+t23g95uljfY93f2a/X/aNuL72l+w7dj1j3zVVVVTZf8OQz9ZbuG6zr1H2HN7z5HN0dK+i2bu7cubYuXbq43W6eccYZth07djTpNer2/5lnnrHts88+De7D9HkOOugg20svvWQrLi52WEdjxyruFlfHes3ZjvhrX+zuN/Hbb7+ZfYu75+rfv79Hx7O+9uKLLzq04/rrr6+77d1333W47ZJLLvHqN+DNZ9uUx1m/++7oZ/f666+bY5KGPuuRI0eadnj6Wbs7FykpKTG/wcGDB9d7joSEBFtLcH5e+3Grt/Q4xboe/T546u2333Z4rB4f+JKee1nXr8fzvubLbZWzTz75xHbooYc2eBw4YMAA22OPPWa2655wt3+vqKgwx8Ddu3d3+TwxMTG2M88805aUlOTV+5Ofn2+74447bL179270PYmPj7cdd9xxtvfee8+r5wAAuEfQAU2iB8XNOWn1hZtuuqlJB1kHH3ywLT09PeiCDm+88YY5oPL29Zxzzjlu16md9hqUacr7pCfhvgo6nHLKKU1qw4wZM2xFRUUevNuuO903bNjg8cG4dvQ29YTH30GH559/3uGxsbGxja5fX8s333xjTh7dvWZXQQcN0hx11FFefU7777+/LSUlxav34++//zYdJd5+Jzx531asWOHyRLqh5fjjj/fou6YnVSeffHKTvs/6ebizc+dOhw5QbxZvAqme+PTTT5vUjr59+5oOq6aedGpHoW7PPHku7Sz0tuNLAyONBV7tyzXXXGO2nwQdWnfQQTtKrLfpd8RbjzzyiMM67r77bpuvtMagg3YYjR071qPPqlu3brbff//d68BiUzrhnLcngQ46+Htf7Oo3MW/evAYHAFiXSy+9tMkDDprCOVi8atWqutu0A71jx44OHYd6XagEHf755x/bmDFjvFqfdrRnZGQ06VxEg3kNDcgIpaCD7qd14Jt1PXo85Sl9L6yP1QFK3gx4asxpp53m8TlbMAUd9Dx58uTJXq1v2LBhti1btjTaXlf799zcXBMA9uR59Pf9xRdfePTerF271gwEaMrxLADAN5g/hial2lizZo3Ddccee2yLt8M5B6vS6fg6LVanm+s08N69e9e7jxa6PuKII0yKkGChOT/PP/98kwLHStO06LTVAw880Exh1bQi3kz7nDlzZr1CxPZURpqyRtc7evRol++Tvz8rnZarr00/K22LpmpxtmjRIjnxxBPNlFhvabqbI4880qHuiE4Z15RC+h3RKflWmpZrxowZZlpvsLEWoVNdu3Zt9DH6G9X3zp6+yv76NYWMpufRGizOdBqzTl92VTNiwIAB5rGaMsj5O7hy5UqTysD6Xjfk+++/N2m0XKWJ0u/FPvvs4/a5GvPpp5/K4YcfbtLnWOkUap02rekJdHq7c2oifZxuF1x9V60uueQSWbx4cb3rNfWB/kb1N6VpyvR1eEq3RZomQtMXWelr189s3333rWt3U1I1ecvVe6Dp3TQdibZFPxtN8eD82WiqIv3+NDUnsm4D3377bYftuX4XNIWTcwqU6upq81nod88T//73v2XWrFn1tiW6LbSnB9DXaPfUU0/J/fff36TXgdCh3zlNE2H36quver2/efnll+v+1rQz1sK2qJ+iQ7d1a9eurbtOU8bZU5s5537XdEgnnHCCSa3jCS3Qqtt/V/siPc7R37puT3R/Fsxaal9s9fzzz8t1111ntq1Kt4d6rKTHTH369Kl3/5deekmeeOIJaQmbN282qV3sdF+o+yI7basWlbbT74ur/XQw0tRfmjZQ0wla6TGqpnvRfb/un6zbKaXpZ/Q4StPEeEPfm6OOOsqkmXE+ftHnCeZ0o67od72kpKTusrbfm9+3HmNZizoXFxdLYmKiz9rnXOdwwYIFJp1VMNPfmx7L/vDDDw7X6/5Nz0N1+6Pnjs7Ho5oWVL+T+r83dJ+raa40LZr1PEe/k7oNcv7u63dYz9f0XKKx1Hp6Huh8DqXH/3oepNs2XfTY1vk5AAA+5KPgBdqQr7/+ut6IgG3btrV4O6666iozsklTDH344YduR3fpqDqdrumcXubaa68NmpkOziOODj/8cNsPP/zgMkWDjt7SkXw6mnLIkCFuR83o646KinJY7+WXX27766+/XN5fU6N89tlntv/7v/8zo5x8OdPh2GOPNSMWr7zyStvnn3/uNpWRppfSKbA6kt+6fh2N2hjnkf763tjT0MycOdOWnJxc7/VqGinnaezPPvusLdhmOujUZufZOo2tv2fPnuZ/fS/1u+I8HVlHFf3444+NzkjR0YzOI5f0vbz99tvrTbfWkYiNpRXRkWydO3d2eJzO8NGZSzozxVlhYaHZ5miKKE0r1dD7tn79evN5W9d9yCGHmO91aWmpw30LCgpMugb7+2Rf9Pvvjo7it95XX79+X92lAElLS7N98MEHtvPOO8+0y91Mh4ceeshhvTq1XNvmKl2RjsDTz0Nnv+goWP3++nqmg7ZZR7vq7/aFF14wv0tXI1r1PXzrrbfM6DZr+3UUc2MjBZ1Hutl/r7ocffTRJm2cdR36vdI0I84j1g444IBGX8+yZcvq/c41XZe2wfocml7llVdesXXt2rXu89XRbk3dBwTjTAfdT+r3UBfnUdT6Wdpvc15+/vnnoJvpsHXr1rr2aRod6/3dvQ5drCOklaY0tD5WUwF6SlM8Wh+r76kvtbaZDvbfuf62dBam875FZ5vpzBHnY5dbb7210Xbp9tma5so+IlbTgboaQa3b148++sh29tlnm8c5H6Ppd16/L/q7cP6MG/p+uToW9fZYsiX2xdb763bO/t5p+h5NMee8z1y9enW9lII6wjwrK8vmb3fddZfD8+pn2ti5SUO/RX1t1s/MeRug30F3n639b+cZTrqOhr4Xrvbnqampth49ejisR7dlmh5Ij32c2/z+++/XS794zDHHNLi/dd7HWI95NIXZunXrHO6vKW48HUXeXL6Y6bBkyRKHdeiMUW85p3PU75KvrFy5st7r1GPZ//znP+Y43Bd8ua3S4yDdBljXozOH9VjQ+TtsT52rx1POx4ANpVpytV9r6NhMfwv6/M7n8Zras6H3UM85nfc/+ttyNatZt5d6DqLnm7r97NevnxefAACgIQQd4LWnn3663kldIGjHgeZp9JR2DFpTrmgnYGM51lsi6KAHOc4BB0+nrOv9NPe6K5rP2Lpe7Xj3lB7g+bKmgwZJnE9gG7JmzRqHXM96QtxYx6pzp7suuo7G0r3cd9999Q6WgynooJ2vzq/L1Wfp6vXriY2nue81cOf8+JdffrnBx2gAyblz6PHHH2/wMc6dFtqJ/Mcff3jURv296nfDFf1+7L333vVSpTTW+a3BmKFDhzo8zl06j1tuucXhfq+++qrNU9ox4y4VgqbEsAZgNPWUp/S+vkwFoDSI7E0udf1ta8eH9b1prNPC+aTTvtxzzz0NPk63d86pFHT6fEPbSK3T4NxJox0r7uhrdw42NGUfEIxBB0/qwDT1+Vo66NDUdbsKSjmn9fOUBhStj9WAnS85vw/aYdhQJ5InwaJABh3s2zjddzTEufNMO2Yb+s1qB9egQYMcHqODORo6jnHeD2zcuNHlbc19v7x9z1pqX+xq+zZt2rR6tS2s9DbnNEBaO8mfdP+mOePtz6cBZFcDnXRbb83Zrqn0PM397u0xbVO2Ua5ogN26Dh1c0dD3XGknq3OqqUWLFrm9v7uUUP7+3Dzhi6CDpqa1rmPKlCler8O5holuf3xJ2+TqM9BAn9Z80NoaX331VaPnpI3xxbZKB944px51DoB5kna0ocFi7o7/Gjs20zRkzkE6DSy4Y72vDubxNLWycjdADwDgPYIO8JoeHDmPHAgVWlDZ2nbtmA900EFH71rv56viVdddd53DenXUta805QTNW3qSbX2OxjopXHW663vbGO2s1hEt/nqv3LXPkxNU7WC1nmzroiPQXXWOuHr9OhreU87BAB2F6gmdReTc0eluhKWeVDl3PnkacGiMcyFJndXjKW2DNde/u9lDWgPDGtDxVbFYa90N7YQIRXqybH0dmse4Ia5OOhsqbG6lM5Ssj3vggQc8HgWpnVI6Q6MxGqxzdVLsawQdWn7drljry2jnrSf7AO38s86saqxjvCncdc54urj6DgQ66NBYZ7idFk+2Pk4D8O7oKFjrfXW2krd1htxp6aBDS+yLlfPnoscaWkeiMTqy2TlQ0ZLH7Zr33Z0bb7zR431DoIMOzgNKpk+f7vEAAp0hYa1h0dB74irooMV4g4Evgg46O7kpxxFW2rHe1GNnTyQmJnpUxFiXPffc03bRRRfZ3nzzTY9+j77cVmmQ1jprSmfdeFocWgOS1sBvQ9sfV/s1T4/NnI/pdB/s6n3S/bOns5gBAP5FTQc0KdesVUJCglePX7p0qXz77beNLtbcjr6iuR2t9QusOWIDxbm2hKtc+8G03pZy5plnOtRd8Paz0jy4J510UqP309yemhvUavXq1RIIeg6mNRg0x+9tt91mchbv2rXL4T4XXXSRqU3QGM1pe+mll3r0vJrbd/ny5Q45ce+9916PHnvjjTdK//79HfLruspDrZzzP996660mL6wvWNet+XkfeOABjx+rbdD6F3Yff/xxXV5rd78pzaXtbb0Jd6zrDbXfqbU2wvTp05u1bdW6C54444wzHC7//vvvbu+rOfqt7r77bunYsWOjz3HwwQd7tP1A63D55ZfX/V1ZWSmvvfZao4/R+iPW367WhwjV329L0foNV111lc9/5877loceesjvtar8oaX2xa7cfvvtHh3PT5061dTb8eRz8QXn3+K//vUvt/d1vu3111+XYOX8nX388cdNznxP9OrVy9Q0stPzJc1f7ylPv1OhUmfQqim5+a31nFyts7n69etnanBonYTGbNmyxRy3nHvuuWYbpvsV5xpl/vLMM8841LV7+OGH69XZcUePu2+44QaH7Y83tSs8PTabNm2aqb9mp/vgd999t9WdAwNAa0LQAV4rLCx0uOxt0THt4NWTlsaWc845R/xBi2DZORfEDgTnAn3WQqq+XO9bb70loUS/V1rcrqmflXOHRUO06LiVL4vIuTNnzhxzgmldtBNbC+bqiYl2mjif+GixwSeffNLj1+9pp/iPP/5Y7zfqXLjXHT2Q15Mj58CiM+3Isxal02DPlVdeKb6QnZ0tv/32W93l4447zqFTxBNaWNFO33dX3zfrb0oL2WnxaV+wrlffO+dAU6jQ4tLWotLeFLjUwI8WsfSEFpy1FgJv6Pdq/c7pd1WDmZ664IILPL4vQpt2VlqPZbQ49O6BuJ4VkFbWTkC4pvsWTzt/PN0vJyUlORSv1wKk/jp+9LeW2Be7oscfp59+ukf31cEg1sECup0vLy8Xf51vLFq0yOE1NtROezFku40bN5pBHMFGC+cuWbLEYZCMJ4NJ3B2zqJ9++smjx2nhXC1Q3VqUlZU5XI6OjvZ6Hc4d684d1r6gA4F0MIZ2kOvn7QltxxtvvGGKKT/22GPib1988YVDYGvKlCkt8p309thMAzFWzgWvVbdu3Ry+C5999pkZ1AUAaHkEHeA155EIxcXFEmg7duwwnbSnnXaaOTjr3r27OYh07tTVxTqKLCsrSwJtwoQJEh8fX3dZT7D0pOrPP/9s1no1cGN10003mZEkaWlpEkgbNmwwHe46snzo0KF1B4auPqvU1NQmf1b77befx/e1BjdUsB2Y6nuhHfTffPONx6O4PD2pUc4n5dZRRJ7OILLSEV3OdMST9UROOwf0pMYXfv75Z4cOQm8+e+sJoZW1E8vdb0o7th599FHJy8vz+vncrVe/e4cffrh88MEHJlATaPraXnnlFbnwwgtl/PjxZuSdds66+r06zy7x5jfrzWemJ6jWjjh3v1cdaZeRkVF3eZ999jEzMjx12GGHeXxfhDYd4W3t9NDRpq46Mqzbs7Vr1zp8V4YNG+b3dg4cONBs6zxd9NgomPhjv+zcsaX7L09H5wabltgXuxuMo8GaYDtm0v1gSUlJ3WWdTdfYNtw54OTJrKWWpsf31vfMX8cszT02DAXOx8QVFRVer8M5aNaU2RKe0OMk3c/o7/yff/4xs100sNjYrCxtn57D3XzzzeIvubm5sn79+rrLerzn7Wzepn4nvT02mzx5ssNl66Aj63Gi9RhOZ4vosfVXX31lgn4AgJZD0AFecz4wCGQHrXYqaQqMIUOGmHQ0H374oRnZpJ1dnhx4Nrez0Bf04Fbb7nyipQdhOmLr+uuvl8WLF3sdLNBR8dbOTJ0ye//995v0BocccojMnj1bvvvuu3ozV/x5kqUHgDpKWZ/7k08+MR07Okrdk85Vbz8r55PihjjP1vHHKCdv6cH+qFGjTMoEPXDXac86fbkpo849+R1Z6XfPG2PGjHG47Gqk/tatWx0uN+Uk2x3nExtN2+SqU7yh5dhjj20wjZzSoKZ1FKXOiNCTwJ49e5oRXg8++KAJgDiPvGvMLbfc4vDZbtu2zQQe9TusJ6jPP/+8/PHHHy16oqTBZH0f7WkctONGZ3/odsjaCeSr36w3v1fn36y736tzSgLd9nhDAxvWdCVo3f7v//7P4fJLL73k9r7Ot3mayq6t88d+2Z/7lpbWEvtiX29//XnM5E1qJWvQwZqm6P333/fbTAxfHbM8++yzXh+z6PFhY8cszT02DAUdOnRwuOzt8Zer76/zOv1BB11dd911snDhQklJSTGzQzW1px7z6zmtKzrIRc9z/WHTpk0Og3d01oO330nn7YKn30lvj800uGEdrKfbTVczE++55x6HwIkewx599NFmdrHOZNXti75uAIB/EXRAs4MO2mnsDQ0IuBqR9/3333u1Hh3ZoCdYepDWWBoEd5oyIsYf7rjjDrnssstcnpjMmzevbiTMiBEjzEGqp9PF33nnnXo5RLXjUjtGdbaBTp3Vz3PixIkmIOGvUZE6rVU7Ajyd6u+KtyeNzRmp1NTvkzc0DYLOXLAuWstEZ+JoXmdN36OjjvQkw9tp98p6QO7JCCcrnX3iDf0OWQ/sndfn6uTD206Ohni7DfKEq2CqjpzSlErOn4duR/Tz09+xBvS0s1pHVOkoNk/yLOvJp55IOn9m2mmvHSZXXHGF2dbp53LKKaeY6fn+DIzpNlq3CZrPtzmdNd481h+/V+eghzcjeZvzGIQm3Ufp6E7rrENXnSYakLPmkNZUbvq7RGB+5/7ct7S0ltgXu9Lckd3+OGbS4Lseq1pnIx1//PEezQbSmjzW90DPE4JJSx2zNPfYMBQ4BwiaMvve+TEtEXRwph3hJ5xwgjnm10Cqjsh31RmvM9b9MQAlkN/JphxnWfsi9P3QcyZnejyuaRCdU27pcbnWe9EZvHperefXGtDU4/tgmGEMAK3N/5IiAx5yTiGgBxXaWW2tleBvenB0zDHH1Dug0VFhepCh+Ur1AE6Lg+nJlHXUk05R1VHDwUTb98ILL5jgwn333edwomWlIzJ00bz+Bx10kOnUbGhUn56wake/HnRpPlCdWeBMZ0DoFHxdZs6cKWeffbY88sgjZvS2L+gU4lNPPdUhwKOvV6d462wMHdGjo6n1c3I+8dWDQG+K44USfd3e5kv1hjdF05xrR3hbp0U/T/2t2U/cXM2ecb7Olyd1/pix5O6kTj83LZypvz0dnaij01x1tmtqFl10FpMWqdWgXkNF8jRthKYe0wKPGix0VchQO0+0I1QXTSGno7iuvvpqj4tPekpndDind9MR/xpI0ZkeWhRRPz/9zK0dXJp7+M0335Rg4fweejNTqKm/BW9Za1Oo5gR5nEd4UjixabMd7AMA9LPQ77MG+q00EGjdnmkA2V/pONA4f+5bWlpL7ItDhXYKWoMZWuPD3bGxM50FYE27pevytGZFS2jJYxZnrW2/4Bxk1Bov3nI+jguGwKXOntXBdXqua031p+eAK1euNKl5W8t30hfHZrrt1MCkMw0s6CAaHWinx86uBhvqDF6tp6iL9mX8+9//lrPOOsvrNgEAXCPoAK/pzlsLyVVXVzvkN27JoIN24FlHZehIYS2U7Emu0qYc3LSUadOmmUXTgnz99dfmQFODBjr11tmyZctM4EFft3YSNnSCoSOlddHPSVMq6Xq1oJnzyBA9QNT16Yh7vU9TRtg7u/322x060vQz0hNAHV3SGF93psLzkWLejDzSjgHryHtXnevO17nqVG8q59+0piRzTpfkLXfT2+3Pd+edd5rvtv4O//vf/5rfi85Acp6BoCc4Tz31lPk962+5oZNZ7czX4KMGCO2/QX3MunXrHLa39uKd1157rSk8qp2guk32BU17Zj3B1c/tueeeMydgjeX31W1LMHE+KfU0LVRL1ixyLhLbnN+FcwejpwVo8T/6PdeBCfb3UtMoOQcdSK0UXPy5b2mN++JQoK9Dg9hWuq9zLrTtKR01rh2Lvqoj5etjFt3uXHTRRc1apw60aoucz1PcFZxviPNjPDk/aQkaQHz11VfNOa71GFADar4OOjh/J3WQiR7nNofOAvSEL47NGgo26+epsxN14I713FqzCTjP0tJBlDr4TgM+jz/+uNftAgDUR9ABXtMduxaB1Q5sa+5HHc3eUrSTzU5HGC5ZsqTBTsKm5Jhsbqd3Uw6irDlXdXS0LvZp5tqhp6M09IDJPnpEOzTPO+88c/DpXMDLFZ0VoYuOvtZ1aGemvnf6furfdnpypp+nXudtITHnk//PP/+87rLOntDn8/RA1NPUAGge589DA3qefJ+svynriCZXn69zWjZrgd/mck5BoVOl/TmLxE5/GzqzSpdZs2aZadm6XdTvuM5WsM4s0tFpmkNWt5WedJZro
gitextract_x9fosayw/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── bug_report.md
│ │ └── feature_request.md
│ └── workflows/
│ ├── build_docs.yml
│ ├── format_and_lint.yml
│ ├── python-publish.yml
│ └── run_tests.yml
├── .gitignore
├── .pre-commit-config.yaml
├── LICENSE
├── README.md
├── bayes_opt/
│ ├── __init__.py
│ ├── acquisition.py
│ ├── bayesian_optimization.py
│ ├── constraint.py
│ ├── domain_reduction.py
│ ├── exception.py
│ ├── logger.py
│ ├── parameter.py
│ ├── py.typed
│ ├── target_space.py
│ └── util.py
├── docsrc/
│ ├── Makefile
│ ├── conf.py
│ ├── index.rst
│ ├── make.bat
│ ├── reference/
│ │ ├── acquisition/
│ │ │ ├── ConstantLiar.rst
│ │ │ ├── ExpectedImprovement.rst
│ │ │ ├── GPHedge.rst
│ │ │ ├── ProbabilityOfImprovement.rst
│ │ │ └── UpperConfidenceBound.rst
│ │ ├── acquisition.rst
│ │ ├── bayes_opt.rst
│ │ ├── constraint.rst
│ │ ├── domain_reduction.rst
│ │ ├── exception.rst
│ │ ├── other.rst
│ │ ├── parameter.rst
│ │ └── target_space.rst
│ └── requirements.txt
├── examples/
│ ├── acquisition_functions.ipynb
│ ├── advanced-tour.ipynb
│ ├── async_optimization.py
│ ├── async_optimization_dummies.py
│ ├── basic-tour.ipynb
│ ├── constraints.ipynb
│ ├── domain_reduction.ipynb
│ ├── duplicate_point.py
│ ├── exploitation_vs_exploration.ipynb
│ ├── parameter_types.ipynb
│ ├── sklearn_example.py
│ ├── typed_hyperparameter_tuning.py
│ └── visualization.ipynb
├── pyproject.toml
├── ruff.toml
├── scripts/
│ ├── check.sh
│ ├── check_precommit.sh
│ └── format.sh
└── tests/
├── test_acquisition.py
├── test_bayesian_optimization.py
├── test_constraint.py
├── test_logger.py
├── test_notebooks_run.py
├── test_parameter.py
├── test_seq_domain_red.py
├── test_target_space.py
└── test_util.py
SYMBOL INDEX (357 symbols across 23 files)
FILE: bayes_opt/acquisition.py
class AcquisitionFunction (line 56) | class AcquisitionFunction(abc.ABC):
method __init__ (line 65) | def __init__(self, random_state: int | RandomState | None = None) -> N...
method base_acq (line 76) | def base_acq(self, *args: Any, **kwargs: Any) -> NDArray[Float]:
method _fit_gp (line 79) | def _fit_gp(self, gp: GaussianProcessRegressor, target_space: TargetSp...
method get_acquisition_params (line 88) | def get_acquisition_params(self) -> dict[str, Any]:
method set_acquisition_params (line 102) | def set_acquisition_params(self, params: dict[str, Any]) -> None:
method suggest (line 116) | def suggest(
method _get_acq (line 171) | def _get_acq(
method _acq_min (line 221) | def _acq_min(
method _random_sample_minimize (line 274) | def _random_sample_minimize(
method _smart_minimize (line 322) | def _smart_minimize(
class UpperConfidenceBound (line 423) | class UpperConfidenceBound(AcquisitionFunction):
method __init__ (line 445) | def __init__(
method base_acq (line 469) | def base_acq(self, mean: NDArray[Float], std: NDArray[Float]) -> NDArr...
method suggest (line 487) | def suggest(
method decay_exploration (line 541) | def decay_exploration(self) -> None:
method get_acquisition_params (line 556) | def get_acquisition_params(self) -> dict[str, Any]:
method set_acquisition_params (line 570) | def set_acquisition_params(self, params: dict[str, Any]) -> None:
class ProbabilityOfImprovement (line 583) | class ProbabilityOfImprovement(AcquisitionFunction):
method __init__ (line 608) | def __init__(
method base_acq (line 633) | def base_acq(self, mean: NDArray[Float], std: NDArray[Float]) -> NDArr...
method suggest (line 663) | def suggest(
method decay_exploration (line 721) | def decay_exploration(self) -> None:
method get_acquisition_params (line 736) | def get_acquisition_params(self) -> dict[str, Any]:
method set_acquisition_params (line 750) | def set_acquisition_params(self, params: dict[str, Any]) -> None:
class ExpectedImprovement (line 763) | class ExpectedImprovement(AcquisitionFunction):
method __init__ (line 795) | def __init__(
method base_acq (line 820) | def base_acq(self, mean: NDArray[Float], std: NDArray[Float]) -> NDArr...
method suggest (line 851) | def suggest(
method decay_exploration (line 910) | def decay_exploration(self) -> None:
method get_acquisition_params (line 925) | def get_acquisition_params(self) -> dict[str, Any]:
method set_acquisition_params (line 939) | def set_acquisition_params(self, params: dict[str, Any]) -> None:
class ConstantLiar (line 952) | class ConstantLiar(AcquisitionFunction):
method __init__ (line 983) | def __init__(
method base_acq (line 1001) | def base_acq(self, *args: Any, **kwargs: Any) -> NDArray[Float]:
method _copy_target_space (line 1013) | def _copy_target_space(self, target_space: TargetSpace) -> TargetSpace:
method _remove_expired_dummies (line 1039) | def _remove_expired_dummies(self, target_space: TargetSpace) -> None:
method suggest (line 1058) | def suggest(
method get_acquisition_params (line 1150) | def get_acquisition_params(self) -> dict[str, Any]:
method set_acquisition_params (line 1166) | def set_acquisition_params(self, params: dict[str, Any]) -> None:
class GPHedge (line 1181) | class GPHedge(AcquisitionFunction):
method __init__ (line 1203) | def __init__(
method base_acq (line 1212) | def base_acq(self, *args: Any, **kwargs: Any) -> NoReturn:
method _sample_idx_from_softmax_gains (line 1221) | def _sample_idx_from_softmax_gains(self, random_state: RandomState) ->...
method _update_gains (line 1238) | def _update_gains(self, gp: GaussianProcessRegressor) -> None:
method suggest (line 1252) | def suggest(
method get_acquisition_params (line 1321) | def get_acquisition_params(self) -> dict[str, Any]:
method set_acquisition_params (line 1337) | def set_acquisition_params(self, params: dict[str, Any]) -> None:
FILE: bayes_opt/bayesian_optimization.py
class BayesianOptimization (line 43) | class BayesianOptimization:
method __init__ (line 86) | def __init__(
method space (line 147) | def space(self) -> TargetSpace:
method acquisition_function (line 152) | def acquisition_function(self) -> AcquisitionFunction:
method constraint (line 157) | def constraint(self) -> ConstraintModel | None:
method max (line 164) | def max(self) -> dict[str, Any] | None:
method res (line 172) | def res(self) -> list[dict[str, Any]]:
method predict (line 179) | def predict(
method register (line 265) | def register(
method probe (line 297) | def probe(self, params: ParamsType, lazy: bool = True) -> None:
method random_sample (line 331) | def random_sample(self, n: int = 1) -> list[dict[str, float | NDArray[...
method suggest (line 349) | def suggest(self) -> dict[str, float | NDArray[Float]]:
method _prime_queue (line 361) | def _prime_queue(self, init_points: int) -> None:
method maximize (line 374) | def maximize(self, init_points: int = 5, n_iter: int = 25) -> None:
method set_bounds (line 419) | def set_bounds(self, new_bounds: BoundsMapping) -> None:
method set_gp_params (line 429) | def set_gp_params(self, **params: Any) -> None:
method save_state (line 435) | def save_state(self, path: str | PathLike[str]) -> None:
method load_state (line 480) | def load_state(self, path: str | PathLike[str]) -> None:
FILE: bayes_opt/constraint.py
class ConstraintModel (line 23) | class ConstraintModel:
method __init__ (line 55) | def __init__(
method lb (line 84) | def lb(self) -> NDArray[Float]:
method ub (line 89) | def ub(self) -> NDArray[Float]:
method model (line 94) | def model(self) -> list[GaussianProcessRegressor]:
method eval (line 98) | def eval(self, **kwargs: Any) -> float | NDArray[Float]: # noqa: D417
method fit (line 132) | def fit(self, X: NDArray[Float], Y: NDArray[Float]) -> None:
method predict (line 153) | def predict(self, X: NDArray[Float]) -> NDArray[Float]:
method approx (line 223) | def approx(self, X: NDArray[Float]) -> NDArray[Float]:
method allowed (line 245) | def allowed(self, constraint_values: NDArray[Float]) -> NDArray[np.boo...
FILE: bayes_opt/domain_reduction.py
class DomainTransformer (line 26) | class DomainTransformer(ABC):
method __init__ (line 30) | def __init__(self, **kwargs: Any) -> None:
method initialize (line 34) | def initialize(self, target_space: TargetSpace) -> None:
method transform (line 38) | def transform(self, target_space: TargetSpace) -> dict[str, NDArray[Fl...
class SequentialDomainReductionTransformer (line 42) | class SequentialDomainReductionTransformer(DomainTransformer):
method __init__ (line 64) | def __init__(
method initialize (line 80) | def initialize(self, target_space: TargetSpace) -> None:
method _update (line 132) | def _update(self, target_space: TargetSpace) -> None:
method _trim (line 158) | def _trim(self, new_bounds: NDArray[Float], global_bounds: NDArray[Flo...
method _window_bounds_compatibility (line 243) | def _window_bounds_compatibility(self, global_bounds: NDArray[Float]) ...
method _create_bounds (line 263) | def _create_bounds(self, parameters: Iterable[str], bounds: NDArray[Fl...
method transform (line 276) | def transform(self, target_space: TargetSpace) -> dict[str, NDArray[Fl...
FILE: bayes_opt/exception.py
class BayesianOptimizationError (line 14) | class BayesianOptimizationError(Exception):
class NotUniqueError (line 18) | class NotUniqueError(BayesianOptimizationError):
class ConstraintNotSupportedError (line 22) | class ConstraintNotSupportedError(BayesianOptimizationError):
class NoValidPointRegisteredError (line 26) | class NoValidPointRegisteredError(BayesianOptimizationError):
class TargetSpaceEmptyError (line 30) | class TargetSpaceEmptyError(BayesianOptimizationError):
FILE: bayes_opt/logger.py
class ScreenLogger (line 16) | class ScreenLogger:
method __init__ (line 35) | def __init__(self, verbose: int = 2, is_constrained: bool = False) -> ...
method verbose (line 46) | def verbose(self) -> int:
method verbose (line 51) | def verbose(self, v: int) -> None:
method is_constrained (line 62) | def is_constrained(self) -> bool:
method _format_number (line 66) | def _format_number(self, x: float) -> str:
method _format_bool (line 106) | def _format_bool(self, x: bool) -> str:
method _format_str (line 121) | def _format_str(self, str_: str) -> str:
method _print_step (line 138) | def _print_step(
method _print_header (line 181) | def _print_header(self, keys: list[str]) -> str:
method _is_new_max (line 205) | def _is_new_max(self, current_max: dict[str, Any] | None) -> bool:
method _update_tracker (line 226) | def _update_tracker(self, current_max: dict[str, Any] | None) -> None:
method log_optimization_start (line 243) | def log_optimization_start(self, keys: list[str]) -> None:
method log_optimization_step (line 255) | def log_optimization_step(
method log_optimization_end (line 289) | def log_optimization_end(self) -> None:
FILE: bayes_opt/parameter.py
function is_numeric (line 37) | def is_numeric(value: Any) -> bool:
class BayesParameter (line 45) | class BayesParameter(abc.ABC):
method __init__ (line 54) | def __init__(self, name: str, bounds: NDArray[Any]) -> None:
method bounds (line 59) | def bounds(self) -> NDArray[Any]:
method is_continuous (line 65) | def is_continuous(self) -> bool:
method random_sample (line 68) | def random_sample(
method to_float (line 90) | def to_float(self, value: Any) -> float | NDArray[Float]:
method to_param (line 100) | def to_param(self, value: float | NDArray[Float]) -> Any:
method kernel_transform (line 115) | def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
method to_string (line 128) | def to_string(self, value: Any, str_len: int) -> str:
method dim (line 151) | def dim(self) -> int:
class FloatParameter (line 155) | class FloatParameter(BayesParameter):
method __init__ (line 167) | def __init__(self, name: str, bounds: tuple[float, float]) -> None:
method is_continuous (line 171) | def is_continuous(self) -> bool:
method to_float (line 175) | def to_float(self, value: float) -> float:
method to_param (line 185) | def to_param(self, value: float | NDArray[Float]) -> float:
method to_string (line 200) | def to_string(self, value: float, str_len: int) -> str:
method kernel_transform (line 222) | def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
method dim (line 237) | def dim(self) -> int:
class IntParameter (line 242) | class IntParameter(BayesParameter):
method __init__ (line 254) | def __init__(self, name: str, bounds: tuple[int, int]) -> None:
method is_continuous (line 258) | def is_continuous(self) -> bool:
method random_sample (line 262) | def random_sample(
method to_float (line 283) | def to_float(self, value: int | float) -> float:
method to_param (line 293) | def to_param(self, value: int | float | NDArray[Int] | NDArray[Float])...
method kernel_transform (line 308) | def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
method dim (line 323) | def dim(self) -> int:
class CategoricalParameter (line 328) | class CategoricalParameter(BayesParameter):
method __init__ (line 340) | def __init__(self, name: str, categories: Sequence[Any]) -> None:
method is_continuous (line 355) | def is_continuous(self) -> bool:
method random_sample (line 359) | def random_sample(
method to_float (line 383) | def to_float(self, value: Any) -> NDArray[Float]:
method to_param (line 396) | def to_param(self, value: float | NDArray[Float]) -> Any:
method to_string (line 411) | def to_string(self, value: Any, str_len: int) -> str:
method kernel_transform (line 434) | def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
method dim (line 452) | def dim(self) -> int:
function wrap_kernel (line 457) | def wrap_kernel(kernel: kernels.Kernel, transform: Callable[[Any], Any])...
function _copy_signature (line 498) | def _copy_signature(source_fct: Callable[..., Any]) -> Callable[[Callabl...
FILE: bayes_opt/target_space.py
function _hashable (line 30) | def _hashable(x: NDArray[Float]) -> tuple[float, ...]:
class TargetSpace (line 35) | class TargetSpace:
method __init__ (line 71) | def __init__(
method __contains__ (line 120) | def __contains__(self, x: NDArray[Float]) -> bool:
method __len__ (line 129) | def __len__(self) -> int:
method empty (line 139) | def empty(self) -> bool:
method params (line 149) | def params(self) -> NDArray[Float]:
method target (line 159) | def target(self) -> NDArray[Float]:
method dim (line 169) | def dim(self) -> int:
method keys (line 179) | def keys(self) -> list[str]:
method params_config (line 189) | def params_config(self) -> dict[str, BayesParameter]:
method bounds (line 194) | def bounds(self) -> NDArray[Float]:
method constraint (line 204) | def constraint(self) -> ConstraintModel | None:
method masks (line 214) | def masks(self) -> dict[str, NDArray[np.bool_]]:
method continuous_dimensions (line 224) | def continuous_dimensions(self) -> NDArray[np.bool_]:
method make_params (line 237) | def make_params(self, pbounds: BoundsMapping) -> dict[str, BayesParame...
method make_masks (line 281) | def make_masks(self) -> dict[str, NDArray[np.bool_]]:
method calculate_bounds (line 301) | def calculate_bounds(self) -> NDArray[Float]:
method params_to_array (line 308) | def params_to_array(self, params: Mapping[str, float | NDArray[Float]]...
method constraint_values (line 327) | def constraint_values(self) -> NDArray[Float]:
method kernel_transform (line 340) | def kernel_transform(self, value: NDArray[Float]) -> NDArray[Float]:
method array_to_params (line 349) | def array_to_params(self, x: NDArray[Float]) -> dict[str, float | NDAr...
method _to_float (line 369) | def _to_float(self, value: Mapping[str, float | NDArray[Float]]) -> ND...
method _to_params (line 379) | def _to_params(self, value: NDArray[Float]) -> dict[str, float | NDArr...
method mask (line 388) | def mask(self) -> NDArray[np.bool_]:
method _as_array (line 412) | def _as_array(self, x: Any) -> NDArray[Float]:
method register (line 424) | def register(
method probe (line 520) | def probe(self, params: ParamsType) -> float | tuple[float, float | ND...
method random_sample (line 565) | def random_sample(
method _target_max (line 605) | def _target_max(self) -> float | None:
method max (line 624) | def max(self) -> dict[str, Any] | None:
method res (line 654) | def res(self) -> list[dict[str, Any]]:
method set_bounds (line 687) | def set_bounds(self, new_bounds: BoundsMapping) -> None:
FILE: bayes_opt/util.py
function ensure_rng (line 8) | def ensure_rng(random_state: int | np.random.RandomState | None = None) ...
FILE: examples/async_optimization.py
function black_box_function (line 24) | def black_box_function(x, y):
class BayesianOptimizationHandler (line 35) | class BayesianOptimizationHandler(RequestHandler):
method post (line 43) | def post(self):
function run_optimization_app (line 61) | def run_optimization_app():
function run_optimizer (line 73) | def run_optimizer():
FILE: examples/async_optimization_dummies.py
function _closest_distance (line 15) | def _closest_distance(point, points):
function optimize (line 19) | def optimize(
FILE: examples/duplicate_point.py
function f (line 6) | def f(x):
FILE: examples/sklearn_example.py
function get_data (line 9) | def get_data():
function svc_cv (line 21) | def svc_cv(C, gamma, data, targets):
function rfc_cv (line 36) | def rfc_cv(n_estimators, min_samples_split, max_features, data, targets):
function optimize_svc (line 58) | def optimize_svc(data, targets):
function optimize_rfc (line 82) | def optimize_rfc(data, targets):
FILE: examples/typed_hyperparameter_tuning.py
function gboost (line 38) | def gboost(log_learning_rate, max_depth, min_samples_split):
FILE: tests/test_acquisition.py
function target_func_x_and_y (line 25) | def target_func_x_and_y():
function pbounds (line 30) | def pbounds():
function constraint (line 35) | def constraint(constraint_func):
function target_func (line 40) | def target_func():
function random_state (line 45) | def random_state():
function gp (line 50) | def gp(random_state):
function target_space (line 55) | def target_space(target_func):
function constraint_func (line 60) | def constraint_func():
function constrained_target_space (line 65) | def constrained_target_space(target_func):
class MockAcquisition (line 72) | class MockAcquisition(acquisition.AcquisitionFunction):
method __init__ (line 73) | def __init__(self):
method _get_acq (line 76) | def _get_acq(self, gp, constraint=None):
method base_acq (line 82) | def base_acq(self, mean, std):
method get_acquisition_params (line 85) | def get_acquisition_params(self) -> dict:
method set_acquisition_params (line 88) | def set_acquisition_params(self, params: dict) -> None:
function test_acquisition_optimization (line 92) | def test_acquisition_optimization(gp, target_space):
function test_acquisition_optimization_only_random (line 102) | def test_acquisition_optimization_only_random(gp, target_space, random_s...
function test_acquisition_optimization_only_l_bfgs_b (line 117) | def test_acquisition_optimization_only_l_bfgs_b(gp, target_space):
function test_upper_confidence_bound (line 124) | def test_upper_confidence_bound(gp, target_space, random_state):
function test_smart_minimize_fails (line 141) | def test_smart_minimize_fails(target_space, random_state):
function test_upper_confidence_bound_with_constraints (line 156) | def test_upper_confidence_bound_with_constraints(gp, constrained_target_...
function test_probability_of_improvement (line 164) | def test_probability_of_improvement(gp, target_space, random_state):
function test_probability_of_improvement_with_constraints (line 185) | def test_probability_of_improvement_with_constraints(gp, constrained_tar...
function test_expected_improvement (line 202) | def test_expected_improvement(gp, target_space, random_state):
function test_expected_improvement_with_constraints (line 223) | def test_expected_improvement_with_constraints(gp, constrained_target_sp...
function test_constant_liar (line 241) | def test_constant_liar(gp, target_space, target_func, random_state, stra...
function test_constant_liar_invalid_strategy (line 271) | def test_constant_liar_invalid_strategy():
function test_constant_liar_with_constraints (line 276) | def test_constant_liar_with_constraints(gp, constrained_target_space, ra...
function test_gp_hedge (line 292) | def test_gp_hedge(random_state):
function test_gphedge_update_gains (line 310) | def test_gphedge_update_gains(random_state):
function test_gphedge_softmax_sampling (line 334) | def test_gphedge_softmax_sampling(random_state):
function test_gphedge_integration (line 358) | def test_gphedge_integration(gp, target_space, random_state):
function test_upper_confidence_bound_invalid_kappa_error (line 375) | def test_upper_confidence_bound_invalid_kappa_error(kappa: float):
function test_upper_confidence_bound_invalid_exploration_decay_error (line 381) | def test_upper_confidence_bound_invalid_exploration_decay_error(explorat...
function test_upper_confidence_bound_invalid_exploration_decay_delay_error (line 389) | def test_upper_confidence_bound_invalid_exploration_decay_delay_error(ex...
function test_probability_of_improvement_invalid_xi_error (line 397) | def test_probability_of_improvement_invalid_xi_error(xi: float):
function test_probability_of_improvement_invalid_exploration_decay_error (line 403) | def test_probability_of_improvement_invalid_exploration_decay_error(expl...
function test_probability_of_improvement_invalid_exploration_decay_delay_error (line 411) | def test_probability_of_improvement_invalid_exploration_decay_delay_erro...
function test_expected_improvement_invalid_xi_error (line 419) | def test_expected_improvement_invalid_xi_error(xi: float):
function test_expected_improvement_invalid_exploration_decay_error (line 425) | def test_expected_improvement_invalid_exploration_decay_error(exploratio...
function test_expected_improvement_invalid_exploration_decay_delay_error (line 433) | def test_expected_improvement_invalid_exploration_decay_delay_error(expl...
function verify_optimizers_match (line 440) | def verify_optimizers_match(optimizer1, optimizer2):
function test_integration_acquisition_functions (line 488) | def test_integration_acquisition_functions(
function test_integration_constrained (line 518) | def test_integration_constrained(target_func_x_and_y, pbounds, constrain...
function test_custom_acquisition_without_get_params (line 548) | def test_custom_acquisition_without_get_params():
function test_custom_acquisition_without_set_params (line 569) | def test_custom_acquisition_without_set_params():
FILE: tests/test_bayesian_optimization.py
class FixedPerimeterTriangleParameter (line 19) | class FixedPerimeterTriangleParameter(BayesParameter):
method __init__ (line 20) | def __init__(self, name: str, bounds, perimeter) -> None:
method is_continuous (line 25) | def is_continuous(self):
method random_sample (line 28) | def random_sample(self, n_samples: int, random_state):
method to_float (line 41) | def to_float(self, value):
method to_param (line 44) | def to_param(self, value):
method kernel_transform (line 47) | def kernel_transform(self, value):
method to_string (line 50) | def to_string(self, value, str_len: int) -> str:
method dim (line 56) | def dim(self):
function area_of_triangle (line 60) | def area_of_triangle(sides):
function target_func (line 66) | def target_func(**kwargs):
function test_properties (line 74) | def test_properties():
function test_register (line 82) | def test_register():
function test_probe_lazy (line 100) | def test_probe_lazy():
function test_probe_eager (line 116) | def test_probe_eager():
function test_suggest_at_random (line 138) | def test_suggest_at_random():
function test_suggest_with_one_observation (line 149) | def test_suggest_with_one_observation():
function test_prime_queue_all_empty (line 167) | def test_prime_queue_all_empty():
function test_prime_queue_empty_with_init (line 177) | def test_prime_queue_empty_with_init():
function test_prime_queue_with_register (line 187) | def test_prime_queue_with_register():
function test_prime_queue_with_register_and_init (line 198) | def test_prime_queue_with_register_and_init():
function test_set_bounds (line 209) | def test_set_bounds():
function test_set_gp_params (line 224) | def test_set_gp_params():
function test_maximize (line 238) | def test_maximize():
function test_define_wrong_transformer (line 262) | def test_define_wrong_transformer():
function test_single_value_objective (line 269) | def test_single_value_objective():
function test_pickle (line 282) | def test_pickle():
function test_duplicate_points (line 294) | def test_duplicate_points():
function test_save_load_state (line 326) | def test_save_load_state(tmp_path):
function test_save_load_w_categorical_params (line 348) | def test_save_load_w_categorical_params(tmp_path):
function test_suggest_point_returns_same_point (line 379) | def test_suggest_point_returns_same_point(tmp_path):
function test_save_load_random_state (line 396) | def test_save_load_random_state(tmp_path):
function test_save_load_unused_optimizer (line 418) | def test_save_load_unused_optimizer(tmp_path):
function test_save_load_w_domain_reduction (line 500) | def test_save_load_w_domain_reduction(tmp_path):
function test_save_load_w_custom_parameter (line 537) | def test_save_load_w_custom_parameter(tmp_path):
function test_predict (line 592) | def test_predict():
function test_predict_example (line 616) | def test_predict_example():
function test_predict_no_fit (line 645) | def test_predict_no_fit():
function test_predict_return_cov (line 670) | def test_predict_return_cov():
function test_predict_integer_params (line 685) | def test_predict_integer_params():
function test_predict_categorical_params (line 722) | def test_predict_categorical_params():
function test_predict_no_points_registered (line 752) | def test_predict_no_points_registered():
function test_predict_custom_parameter (line 772) | def test_predict_custom_parameter():
function test_predict_invalid_params_type (line 806) | def test_predict_invalid_params_type():
function test_predict_with_tuple (line 816) | def test_predict_with_tuple():
function test_predict_return_std_and_cov_mutually_exclusive (line 826) | def test_predict_return_std_and_cov_mutually_exclusive():
function test_predict_shape_semantics_dict_vs_list (line 841) | def test_predict_shape_semantics_dict_vs_list():
function test_predict_shape_semantics_with_std (line 862) | def test_predict_shape_semantics_with_std():
function test_predict_shape_semantics_with_cov (line 890) | def test_predict_shape_semantics_with_cov():
FILE: tests/test_constraint.py
function target_function (line 11) | def target_function():
function constraint_function (line 16) | def constraint_function():
function test_constraint_property (line 20) | def test_constraint_property(target_function, constraint_function):
function test_single_constraint_upper (line 32) | def test_single_constraint_upper(target_function, constraint_function):
function test_single_constraint_lower (line 47) | def test_single_constraint_lower(target_function, constraint_function):
function test_single_constraint_lower_upper (line 62) | def test_single_constraint_lower_upper(target_function, constraint_funct...
function test_multiple_constraints (line 95) | def test_multiple_constraints(target_function):
function test_kwargs_not_the_same (line 126) | def test_kwargs_not_the_same(target_function):
function test_lower_less_than_upper (line 145) | def test_lower_less_than_upper(target_function):
function test_null_constraint_function (line 164) | def test_null_constraint_function():
FILE: tests/test_logger.py
function target_func (line 12) | def target_func(**kwargs):
function test_initialization (line 20) | def test_initialization():
function test_verbose_property (line 33) | def test_verbose_property():
function test_is_constrained_property (line 45) | def test_is_constrained_property():
function test_format_number (line 54) | def test_format_number():
function test_format_bool (line 110) | def test_format_bool():
function test_format_str (line 131) | def test_format_str():
function test_step (line 147) | def test_step():
function test_print_header (line 172) | def test_print_header():
function test_is_new_max (line 194) | def test_is_new_max():
function test_update_tracker (line 218) | def test_update_tracker():
function test_log_optimization_start (line 250) | def test_log_optimization_start(mock_stdout):
function test_log_optimization_step (line 270) | def test_log_optimization_step(mock_stdout):
function test_log_optimization_end (line 323) | def test_log_optimization_end(mock_stdout):
FILE: tests/test_notebooks_run.py
function test_all_notebooks_run (line 20) | def test_all_notebooks_run(notebook: Path):
FILE: tests/test_parameter.py
function test_float_parameters (line 13) | def test_float_parameters():
function test_int_parameters (line 51) | def test_int_parameters():
function test_cat_parameters (line 90) | def test_cat_parameters():
function test_cateogrical_valid_bounds (line 131) | def test_cateogrical_valid_bounds():
function test_to_string (line 141) | def test_to_string():
function test_preconstructed_parameter (line 173) | def test_preconstructed_parameter():
function test_integration_mixed_optimization (line 189) | def test_integration_mixed_optimization():
function test_integration_mixed_optimization_with_constraints (line 206) | def test_integration_mixed_optimization_with_constraints():
function test_wrapped_kernel_fit (line 234) | def test_wrapped_kernel_fit():
function test_combined_wrapped_kernel_fit (line 249) | def test_combined_wrapped_kernel_fit():
FILE: tests/test_seq_domain_red.py
function black_box_function (line 10) | def black_box_function(x, y):
function test_bound_x_maximize (line 20) | def test_bound_x_maximize():
function test_minimum_window_is_kept (line 63) | def test_minimum_window_is_kept():
function test_minimum_window_array_is_kept (line 79) | def test_minimum_window_array_is_kept():
function test_trimming_bounds (line 96) | def test_trimming_bounds():
function test_exceeded_bounds (line 116) | def test_exceeded_bounds():
function test_trim_when_both_new_bounds_exceed_global_bounds (line 131) | def test_trim_when_both_new_bounds_exceed_global_bounds():
function test_minimum_window_dict_ordering (line 171) | def test_minimum_window_dict_ordering():
function test_mixed_parameters (line 182) | def test_mixed_parameters():
FILE: tests/test_target_space.py
function target_func (line 11) | def target_func(**kwargs):
function test_keys_and_bounds_in_same_order (line 19) | def test_keys_and_bounds_in_same_order():
function test_params_to_array (line 30) | def test_params_to_array():
function test_array_to_params (line 43) | def test_array_to_params():
function test_to_float (line 53) | def test_to_float():
function test_register (line 72) | def test_register():
function test_register_with_constraint (line 101) | def test_register_with_constraint():
function test_register_point_beyond_bounds (line 124) | def test_register_point_beyond_bounds():
function test_probe (line 132) | def test_probe():
function test_random_sample (line 175) | def test_random_sample():
function test_y_max (line 186) | def test_y_max():
function test_y_max_with_constraint (line 195) | def test_y_max_with_constraint():
function test_y_max_within_pbounds (line 206) | def test_y_max_within_pbounds():
function test_max (line 217) | def test_max():
function test_max_with_constraint (line 229) | def test_max_with_constraint():
function test_max_with_constraint_identical_target_value (line 242) | def test_max_with_constraint_identical_target_value():
function test_max_categorical (line 256) | def test_max_categorical() -> None:
function test_res (line 276) | def test_res():
function test_res_categorical (line 296) | def test_res_categorical() -> None:
function test_res_categorical_with_constraints (line 320) | def test_res_categorical_with_constraints() -> None:
function test_set_bounds (line 344) | def test_set_bounds():
function test_no_target_func (line 359) | def test_no_target_func():
function test_change_typed_bounds (line 365) | def test_change_typed_bounds():
FILE: tests/test_util.py
function test_ensure_rng (line 8) | def test_ensure_rng():
Condensed preview — 66 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (3,725K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.md",
"chars": 1171,
"preview": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: ''\nlabels: bug, enhancement\nassignees: ''\n\n---\n\n**"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md",
"chars": 778,
"preview": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: ''\nlabels: enhancement\nassignees: ''\n\n---\n\n**Is"
},
{
"path": ".github/workflows/build_docs.yml",
"chars": 3675,
"preview": "name: docs\n\non:\n release:\n types: [published]\n push:\n branches:\n - master\n pull_request:\n\nconcurrency:\n g"
},
{
"path": ".github/workflows/format_and_lint.yml",
"chars": 473,
"preview": "name: Code format and lint\n\non:\n push:\n branches: [ \"master\" ]\n pull_request:\n\npermissions:\n contents: read\n\njobs:"
},
{
"path": ".github/workflows/python-publish.yml",
"chars": 559,
"preview": "# This workflow will upload a Python Package using uv when a release is created\n# Note that you must manually update the"
},
{
"path": ".github/workflows/run_tests.yml",
"chars": 1343,
"preview": "# This workflow will install Python dependencies and run tests in multiple versions of Python\n# For more information see"
},
{
"path": ".gitignore",
"chars": 494,
"preview": ".ipynb_checkpoints\n*.pyc\n*.egg-info/\nbuild/\ndist/\nscratch/\n.idea/\n.DS_Store\nbo_eg*.png\ngif/\n\n# Unit test / coverage repo"
},
{
"path": ".pre-commit-config.yaml",
"chars": 204,
"preview": "repos:\n - hooks:\n - id: ruff\n name: ruff-lint\n - id: ruff-format\n name: ruff-format\n arg"
},
{
"path": "LICENSE",
"chars": 1089,
"preview": "The MIT License (MIT)\n\nCopyright (c) 2014 Fernando M. F. Nogueira\n\nPermission is hereby granted, free of charge, to any "
},
{
"path": "README.md",
"chars": 8581,
"preview": "<div align=\"center\">\n <img src=\"https://raw.githubusercontent.com/bayesian-optimization/BayesianOptimization/master/doc"
},
{
"path": "bayes_opt/__init__.py",
"chars": 703,
"preview": "\"\"\"Pure Python implementation of bayesian global optimization with gaussian processes.\"\"\"\n\nfrom __future__ import annota"
},
{
"path": "bayes_opt/acquisition.py",
"chars": 47581,
"preview": "\"\"\"Acquisition functions for Bayesian Optimization.\n\nThe acquisition functions in this module can be grouped the followi"
},
{
"path": "bayes_opt/bayesian_optimization.py",
"chars": 20953,
"preview": "\"\"\"Main module.\n\nHolds the `BayesianOptimization` class, which handles the maximization of a\nfunction over a specific ta"
},
{
"path": "bayes_opt/constraint.py",
"chars": 8828,
"preview": "\"\"\"Constraint handling.\"\"\"\n\nfrom __future__ import annotations\n\nfrom typing import TYPE_CHECKING, Any\n\nimport numpy as n"
},
{
"path": "bayes_opt/domain_reduction.py",
"chars": 11546,
"preview": "\"\"\"Implement domain transformation.\n\nIn particular, this provides a base transformer class and a sequential domain\nreduc"
},
{
"path": "bayes_opt/exception.py",
"chars": 871,
"preview": "\"\"\"This module contains custom exceptions for Bayesian Optimization.\"\"\"\n\nfrom __future__ import annotations\n\n__all__ = ["
},
{
"path": "bayes_opt/logger.py",
"chars": 8962,
"preview": "\"\"\"Contains classes and functions for logging.\"\"\"\n\nfrom __future__ import annotations\n\nfrom collections.abc import Mappi"
},
{
"path": "bayes_opt/parameter.py",
"chars": 13932,
"preview": "\"\"\"Parameter classes for Bayesian optimization.\"\"\"\n\nfrom __future__ import annotations\n\nimport abc\nfrom collections.abc "
},
{
"path": "bayes_opt/py.typed",
"chars": 0,
"preview": ""
},
{
"path": "bayes_opt/target_space.py",
"chars": 24092,
"preview": "\"\"\"Manages the optimization domain and holds points.\"\"\"\n\nfrom __future__ import annotations\n\nfrom copy import deepcopy\nf"
},
{
"path": "bayes_opt/util.py",
"chars": 1020,
"preview": "\"\"\"Contains utility functions.\"\"\"\n\nfrom __future__ import annotations\n\nimport numpy as np\n\n\ndef ensure_rng(random_state:"
},
{
"path": "docsrc/Makefile",
"chars": 739,
"preview": "# Minimal makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line, and also\n# from the "
},
{
"path": "docsrc/conf.py",
"chars": 6930,
"preview": "# Configuration file for the Sphinx documentation builder.\n#\n# This file only contains a selection of the most common op"
},
{
"path": "docsrc/index.rst",
"chars": 7761,
"preview": ".. toctree::\n :hidden:\n\n Quickstart <self>\n\n.. toctree::\n :hidden:\n :maxdepth: 3\n :caption: Example Notebooks:"
},
{
"path": "docsrc/make.bat",
"chars": 956,
"preview": "@ECHO OFF\n\npushd %~dp0\n\nREM Command file for Sphinx documentation\n\nif \"%SPHINXBUILD%\" == \"\" (\n\tset SPHINXBUILD=sphinx-bu"
},
{
"path": "docsrc/reference/acquisition/ConstantLiar.rst",
"chars": 158,
"preview": ":py:class:`bayes_opt.acquisition.ConstantLiar`\n----------------------------------------------\n\n.. autoclass:: bayes_opt."
},
{
"path": "docsrc/reference/acquisition/ExpectedImprovement.rst",
"chars": 179,
"preview": ":py:class:`bayes_opt.acquisition.ExpectedImprovement`\n-----------------------------------------------------\n\n.. autoclas"
},
{
"path": "docsrc/reference/acquisition/GPHedge.rst",
"chars": 143,
"preview": ":py:class:`bayes_opt.acquisition.GPHedge`\n-----------------------------------------\n\n.. autoclass:: bayes_opt.acquisitio"
},
{
"path": "docsrc/reference/acquisition/ProbabilityOfImprovement.rst",
"chars": 194,
"preview": ":py:class:`bayes_opt.acquisition.ProbabilityOfImprovement`\n----------------------------------------------------------\n\n."
},
{
"path": "docsrc/reference/acquisition/UpperConfidenceBound.rst",
"chars": 182,
"preview": ":py:class:`bayes_opt.acquisition.UpperConfidenceBound`\n------------------------------------------------------\n\n.. autocl"
},
{
"path": "docsrc/reference/acquisition.rst",
"chars": 325,
"preview": ":py:mod:`bayes_opt.acquisition`\n-------------------------------\n\n.. automodule:: bayes_opt.acquisition\n :members: Acqu"
},
{
"path": "docsrc/reference/bayes_opt.rst",
"chars": 146,
"preview": ":py:class:`bayes_opt.BayesianOptimization`\n------------------------------------------\n\n.. autoclass:: bayes_opt.Bayesian"
},
{
"path": "docsrc/reference/constraint.rst",
"chars": 283,
"preview": ":py:class:`bayes_opt.ConstraintModel`\n------------------------------------------------\n\nSee the `Constrained Optimizatio"
},
{
"path": "docsrc/reference/domain_reduction.rst",
"chars": 294,
"preview": ":py:class:`bayes_opt.SequentialDomainReductionTransformer`\n----------------------------------------------------------\n\nS"
},
{
"path": "docsrc/reference/exception.rst",
"chars": 113,
"preview": ":py:mod:`bayes_opt.exception`\n-------------------------------\n\n.. automodule:: bayes_opt.exception\n :members: \n"
},
{
"path": "docsrc/reference/other.rst",
"chars": 65,
"preview": "Other\n-----\n\n.. autoclass:: bayes_opt.ScreenLogger\n :members:\n"
},
{
"path": "docsrc/reference/parameter.rst",
"chars": 114,
"preview": ":py:mod:`bayes_opt.parameter`\n--------------------------------\n\n.. automodule:: bayes_opt.parameter\n :members: \n"
},
{
"path": "docsrc/reference/target_space.rst",
"chars": 119,
"preview": ":py:class:`bayes_opt.TargetSpace`\n---------------------------------\n\n.. autoclass:: bayes_opt.TargetSpace\n :members:\n"
},
{
"path": "docsrc/requirements.txt",
"chars": 32,
"preview": "sphinx\nnbsphinx\nsphinx_rtd_theme"
},
{
"path": "examples/acquisition_functions.ipynb",
"chars": 687384,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Acquisition functions\\n\",\n \"\\n"
},
{
"path": "examples/advanced-tour.ipynb",
"chars": 11487,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Advanced tour of the Bayesian Opt"
},
{
"path": "examples/async_optimization.py",
"chars": 3817,
"preview": "import time\nimport random\n\nfrom bayes_opt import BayesianOptimization\nfrom bayes_opt.util import UtilityFunction\n\nimport"
},
{
"path": "examples/async_optimization_dummies.py",
"chars": 3068,
"preview": "\"\"\"Originally by @rhizhiy\nhttps://github.com/bayesian-optimization/BayesianOptimization/issues/347#issuecomment-12734650"
},
{
"path": "examples/basic-tour.ipynb",
"chars": 15413,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Basic tour of the Bayesian Optimi"
},
{
"path": "examples/constraints.ipynb",
"chars": 323360,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Constrained Optimization\\n\",\n "
},
{
"path": "examples/domain_reduction.ipynb",
"chars": 114868,
"preview": "{\n \"cells\": [\n {\n \"attachments\": {},\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Sequential "
},
{
"path": "examples/duplicate_point.py",
"chars": 952,
"preview": "import numpy as np\nfrom bayes_opt import BayesianOptimization\nfrom bayes_opt import acquisition\n\n\ndef f(x):\n return n"
},
{
"path": "examples/exploitation_vs_exploration.ipynb",
"chars": 618250,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Exploitation vs Exploration\"\n ]"
},
{
"path": "examples/parameter_types.ipynb",
"chars": 196954,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Optimizing over non-float Paramet"
},
{
"path": "examples/sklearn_example.py",
"chars": 3925,
"preview": "from sklearn.datasets import make_classification\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.ensemb"
},
{
"path": "examples/typed_hyperparameter_tuning.py",
"chars": 3257,
"preview": "import numpy as np\nfrom bayes_opt import BayesianOptimization, acquisition\nfrom sklearn.ensemble import GradientBoosting"
},
{
"path": "examples/visualization.ipynb",
"chars": 1424722,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": [\n "
},
{
"path": "pyproject.toml",
"chars": 1708,
"preview": "[project]\nname = \"bayesian-optimization\"\nversion = \"3.2.1\"\ndescription = \"Bayesian Optimization package\"\nauthors = [{ na"
},
{
"path": "ruff.toml",
"chars": 2791,
"preview": "line-length = 110\nunsafe-fixes = false\ntarget-version = \"py39\"\nextend = \"./pyproject.toml\"\nexclude = [\n # docs\n \"d"
},
{
"path": "scripts/check.sh",
"chars": 104,
"preview": "#!/usr/bin/env sh\nset -ex\n\nuv run ruff format --check bayes_opt tests\nuv run ruff check bayes_opt tests\n"
},
{
"path": "scripts/check_precommit.sh",
"chars": 110,
"preview": "#!/usr/bin/env sh\nset -ex\n\nuv run pre-commit install\nuv run pre-commit run --all-files --show-diff-on-failure\n"
},
{
"path": "scripts/format.sh",
"chars": 96,
"preview": "#!/usr/bin/env sh\nset -ex\n\nuv run ruff format bayes_opt tests\nuv run ruff check bayes_opt --fix\n"
},
{
"path": "tests/test_acquisition.py",
"chars": 22144,
"preview": "from __future__ import annotations\n\nimport sys\n\nimport numpy as np\nimport pytest\nfrom scipy.optimize import NonlinearCon"
},
{
"path": "tests/test_bayesian_optimization.py",
"chars": 36008,
"preview": "from __future__ import annotations\n\nimport pickle\nfrom pathlib import Path\n\nimport numpy as np\nimport pytest\nfrom scipy."
},
{
"path": "tests/test_constraint.py",
"chars": 5862,
"preview": "from __future__ import annotations\n\nimport numpy as np\nimport pytest\nfrom scipy.optimize import NonlinearConstraint\n\nfro"
},
{
"path": "tests/test_logger.py",
"chars": 11328,
"preview": "from __future__ import annotations\n\nimport io\nfrom unittest.mock import patch\n\nfrom colorama import Fore\n\nfrom bayes_opt"
},
{
"path": "tests/test_notebooks_run.py",
"chars": 1139,
"preview": "\"\"\"\ncollect all notebooks in examples, and check that they run without error\n\"\"\"\n\nfrom __future__ import annotations\n\nfr"
},
{
"path": "tests/test_parameter.py",
"chars": 9214,
"preview": "from __future__ import annotations\n\nimport numpy as np\nimport pytest\nfrom scipy.optimize import NonlinearConstraint\nfrom"
},
{
"path": "tests/test_seq_domain_red.py",
"chars": 7147,
"preview": "from __future__ import annotations\n\nimport numpy as np\nimport pytest\n\nfrom bayes_opt import BayesianOptimization, Sequen"
},
{
"path": "tests/test_target_space.py",
"chars": 13217,
"preview": "from __future__ import annotations\n\nimport numpy as np\nimport pytest\nfrom scipy.optimize import NonlinearConstraint\n\nfro"
},
{
"path": "tests/test_util.py",
"chars": 788,
"preview": "from __future__ import annotations\n\nimport numpy as np\n\nfrom bayes_opt.util import ensure_rng\n\n\ndef test_ensure_rng():\n "
}
]
About this extraction
This page contains the full source code of the bayesian-optimization/BayesianOptimization GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 66 files (3.5 MB), approximately 926.7k tokens, and a symbol index with 357 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.