Showing preview only (555K chars total). Download the full file or copy to clipboard to get everything.
Repository: uber/bayesmark
Branch: master
Commit: 8c420e935718
Files: 102
Total size: 525.6 KB
Directory structure:
gitextract_vn3wrx6j/
├── .coveragerc
├── .gitignore
├── .pre-commit-config.yaml
├── .readthedocs.yml
├── .secrets.baseline
├── .travis.yml
├── LICENSE
├── MANIFEST.in
├── README.rst
├── bayesmark/
│ ├── __init__.py
│ ├── abstract_optimizer.py
│ ├── builtin_opt/
│ │ ├── __init__.py
│ │ ├── config.py
│ │ ├── hyperopt_optimizer.py
│ │ ├── nevergrad_optimizer.py
│ │ ├── opentuner_optimizer.py
│ │ ├── pysot_optimizer.py
│ │ ├── random_optimizer.py
│ │ └── scikit_optimizer.py
│ ├── cmd_parse.py
│ ├── constants.py
│ ├── data.py
│ ├── expected_max.py
│ ├── experiment.py
│ ├── experiment_aggregate.py
│ ├── experiment_analysis.py
│ ├── experiment_baseline.py
│ ├── experiment_db_init.py
│ ├── experiment_launcher.py
│ ├── np_util.py
│ ├── path_util.py
│ ├── quantiles.py
│ ├── random_search.py
│ ├── serialize.py
│ ├── signatures.py
│ ├── sklearn_funcs.py
│ ├── space.py
│ ├── stats.py
│ ├── util.py
│ └── xr_util.py
├── build_wheel.sh
├── docs/
│ ├── .gitignore
│ ├── Makefile
│ ├── authors.rst
│ ├── code.rst
│ ├── conf.py
│ ├── dummy.py
│ ├── index.rst
│ ├── readme.rst
│ └── scoring.rst
├── example_opt_root/
│ ├── config.json
│ ├── flaky_optimizer.py
│ ├── hyperopt_optimizer.py
│ ├── nevergrad_optimizer.py
│ ├── opentuner_optimizer.py
│ ├── pysot_optimizer.py
│ ├── random_optimizer.py
│ └── scikit_optimizer.py
├── integration_test.sh
├── integration_test_with_setup.sh
├── notebooks/
│ ├── dummy.py
│ ├── plot_mean_score.ipynb
│ └── plot_test_case.ipynb
├── requirements/
│ ├── base.in
│ ├── base.txt
│ ├── docs.in
│ ├── docs.txt
│ ├── ipynb.in
│ ├── ipynb.txt
│ ├── optimizers.in
│ ├── optimizers.txt
│ ├── pipreqs_edits.sed
│ ├── self.txt
│ ├── test.in
│ ├── test.txt
│ ├── tools.in
│ └── tools.txt
├── setup.py
├── test/
│ ├── data_test.py
│ ├── dummy.py
│ ├── expected_max_test.py
│ ├── experiment_aggregate_test.py
│ ├── experiment_analysis_test.py
│ ├── experiment_baseline_test.py
│ ├── experiment_db_init_test.py
│ ├── experiment_launcher_test.py
│ ├── experiment_test.py
│ ├── hypothesis_util.py
│ ├── np_util_test.py
│ ├── quantiles_test.py
│ ├── random_search_test.py
│ ├── serialize_test.py
│ ├── signatures_test.py
│ ├── sklearn_funcs_test.py
│ ├── space_test.py
│ ├── stats_test.py
│ ├── util.py
│ ├── util_test.py
│ └── xr_util_test.py
├── test.sh
└── tools/
├── archive_branch.sh
└── deploy.sh
================================================
FILE CONTENTS
================================================
================================================
FILE: .coveragerc
================================================
[report]
exclude_lines =
pragma: no cover
@abstract
ValueError
NotImplementedError
assert
_error
def main()
pragma: io
pragma: main
pragma: validator
================================================
FILE: .gitignore
================================================
.*
!.gitignore
!.gitmodules
!.flake8
!.coveragerc
!.pre-commit-config.yaml
!.secrets.baseline
!.travis.yml
!.readthedocs.yml
# For wheels
bayesmark/version.py
dist/
# Java
*.class
# Intellij
*.iml
*.iws
# Gradle
build/
classes/
log/
tmp/
/out/
ins.xml
*.log
# Python
*.py[co]
*.egg*
.cache
.DS_Store
# env
env/
# Emacs
*~
.\#*
\#*\#
# *ipynb
.ipynb_checkpoints
*.png
*.aux
# Hypothesis
tests/src
src/
# Coverage
htmlcov/
# for the test.sh pip compile check
requirements/*.chk
requirement_chk.in
================================================
FILE: .pre-commit-config.yaml
================================================
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v1.2.3
hooks:
- id: flake8
exclude: ^(docs/*)
args: [--max-line-length=120, --ignore=E203]
- id: check-byte-order-marker
- id: check-case-conflict
- id: check-merge-conflict
- id: end-of-file-fixer
- id: forbid-new-submodules
- id: mixed-line-ending
args: [--fix=lf]
- id: trailing-whitespace
- id: debug-statements
- id: check-json
- id: pretty-format-json
args: [--autofix, --indent=4]
- id: check-yaml
- id: sort-simple-yaml
- repo: https://github.com/ambv/black
rev: 19.3b0
hooks:
- id: black
args: [-l 120, --target-version=py36]
- repo: https://github.com/asottile/seed-isort-config
rev: v1.2.0
hooks:
- id: seed-isort-config
args: [--application-directories=test]
- repo: https://github.com/pre-commit/mirrors-isort
rev: v4.3.4
hooks:
- id: isort
language_version: python3
args: [-w 120, -m 3, -tc, --project=bayesmark]
- repo: https://github.com/jumanjihouse/pre-commit-hooks
rev: 1.11.0
hooks:
- id: require-ascii
- id: script-must-have-extension
- id: forbid-binary
- repo: https://github.com/Lucas-C/pre-commit-hooks
rev: v1.1.6
hooks:
- id: forbid-crlf
- id: forbid-tabs
- repo: https://github.com/kynan/nbstripout
rev: fe155a55548c61e4eb53522e57921077acf82c00 # pragma: allowlist secret
hooks:
- id: nbstripout
exclude: ^notebooks/.*\.out\.ipynb$
- repo: https://github.com/Yelp/detect-secrets
rev: v0.12.5
hooks:
- id: detect-secrets
args: ['--baseline', '.secrets.baseline']
- repo: https://github.com/pre-commit/pygrep-hooks
rev: v1.4.1 # Use the ref you want to point at
hooks:
- id: python-no-eval
- id: python-check-blanket-noqa
- repo: https://github.com/asottile/yesqa
rev: v0.0.11
hooks:
- id: yesqa
- repo: https://github.com/myint/eradicate
rev: 522ed7ce2da82d33b3e2331bf50d4671c5a5af9a # pragma: allowlist secret
hooks:
- id: eradicate
exclude: docs/conf.py
================================================
FILE: .readthedocs.yml
================================================
# .readthedocs.yml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py
# Build documentation with MkDocs
#mkdocs:
# configuration: mkdocs.yml
# Optionally build your docs in additional formats such as PDF and ePub
formats: all
# Optionally set the version of Python and requirements required to build your docs
python:
version: 3.6
install:
- requirements: requirements/docs.txt
================================================
FILE: .secrets.baseline
================================================
{
"exclude": {
"files": null,
"lines": null
},
"generated_at": "2019-09-18T01:04:54Z",
"plugins_used": [
{
"name": "AWSKeyDetector"
},
{
"name": "ArtifactoryDetector"
},
{
"base64_limit": 4.5,
"name": "Base64HighEntropyString"
},
{
"name": "BasicAuthDetector"
},
{
"hex_limit": 3,
"name": "HexHighEntropyString"
},
{
"name": "KeywordDetector"
},
{
"name": "PrivateKeyDetector"
},
{
"name": "SlackDetector"
},
{
"name": "StripeDetector"
}
],
"results": {},
"version": "0.12.5"
}
================================================
FILE: .travis.yml
================================================
language: python
python:
- "3.6"
before_script:
- "curl -H 'Cache-Control: no-cache' https://raw.githubusercontent.com/fossas/fossa-cli/master/install.sh | sudo bash"
script:
- ./integration_test_with_setup.sh
- ./test.sh
- cat requirements/*.txt >requirements.txt
- '[ ! -z "$FOSSA_API_KEY" ] && (fossa init && fossa analyze) || true'
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: MANIFEST.in
================================================
include requirements/base.in
include requirements/optimizers.in
include requirements/ipynb.in
include LICENSE
include README.rst
================================================
FILE: README.rst
================================================
Installation
============
This project provides a benchmark framework to easily compare Bayesian optimization methods on real machine learning tasks.
This project is experimental and the APIs are not considered stable.
This Bayesian optimization (BO) benchmark framework requires a few easy steps for setup. It can be run either on a local machine (in serial) or prepare a *commands file* to run on a cluster as parallel experiments (dry run mode).
Only ``Python>=3.6`` is officially supported, but older versions of Python likely work as well.
The core package itself can be installed with:
.. code-block:: bash
pip install bayesmark
However, to also require installation of all the "built in" optimizers for evaluation, run:
.. code-block:: bash
pip install bayesmark[optimizers]
It is also possible to use the same pinned dependencies we used in testing by `installing from the repo <#install-in-editable-mode>`_.
Building an environment to run the included notebooks can be done with:
.. code-block:: bash
pip install bayesmark[notebooks]
Or, ``bayesmark[optimizers,notebooks]`` can be used.
A quick example of running the benchmark is `here <#example>`_. The instructions are used to generate results as below:
.. image:: https://user-images.githubusercontent.com/28273671/66338456-02516b80-e8f6-11e9-8156-2e84e04cf6fe.png
:width: 95 %
Non-pip dependencies
--------------------
To be able to install ``opentuner`` some system level (non-pip) dependencies must be installed. This can be done with:
.. code-block:: bash
sudo apt-get install libsqlite3-0
sudo apt-get install libsqlite3-dev
On Ubuntu, this results in:
.. code-block:: console
> dpkg -l | grep libsqlite
ii libsqlite3-0:amd64 3.11.0-1ubuntu1 amd64 SQLite 3 shared library
ii libsqlite3-dev:amd64 3.11.0-1ubuntu1 amd64 SQLite 3 development files
The environment should now all be setup to run the BO benchmark.
Running
=======
Now we can run each step of the experiments. First, we run all combinations and then run some quick commands to analyze the output.
Launch the experiments
----------------------
The experiments are run using the experiment launcher, which has the following interface:
.. code-block::
usage: bayesmark-launch [-h] [-dir DB_ROOT] [-odir OPTIMIZER_ROOT] [-v] [-u UUID]
[-dr DATA_ROOT] [-b DB] [-o OPTIMIZER [OPTIMIZER ...]]
[-d DATA [DATA ...]]
[-c [{DT,MLP-adam,MLP-sgd,RF,SVM,ada,kNN,lasso,linear} ...]]
[-m [{acc,mae,mse,nll} ...]] [-n N_CALLS]
[-p N_SUGGEST] [-r N_REPEAT] [-nj N_JOBS] [-ofile JOBS_FILE]
The arguments are:
.. code-block::
-h, --help show this help message and exit
-dir DB_ROOT, -db-root DB_ROOT
root directory for all benchmark experiments output
-odir OPTIMIZER_ROOT, --opt-root OPTIMIZER_ROOT
Directory with optimization wrappers
-v, --verbose print the study logs to console
-u UUID, --uuid UUID length 32 hex UUID for this experiment
-dr DATA_ROOT, --data-root DATA_ROOT
root directory for all custom csv files
-b DB, --db DB database ID of this benchmark experiment
-o OPTIMIZER [OPTIMIZER ...], --opt OPTIMIZER [OPTIMIZER ...]
optimizers to use
-d DATA [DATA ...], --data DATA [DATA ...]
data sets to use
-c, --classifier [{DT,MLP-adam,MLP-sgd,RF,SVM,ada,kNN,lasso,linear} ...]
classifiers to use
-m, --metric [{acc,mae,mse,nll} ...]
scoring metric to use
-n N_CALLS, --calls N_CALLS
number of function evaluations
-p N_SUGGEST, --suggestions N_SUGGEST
number of suggestions to provide in parallel
-r N_REPEAT, --repeat N_REPEAT
number of repetitions of each study
-nj N_JOBS, --num-jobs N_JOBS
number of jobs to put in the dry run file, the default
0 value disables dry run (real run)
-ofile JOBS_FILE, --jobs-file JOBS_FILE
a jobs file with all commands to be run
The output files will be placed in ``[DB_ROOT]/[DBID]``. If ``DBID`` is not specified, it will be a randomly created subdirectory with a new name to avoid overwriting previous experiments. The path to ``DBID`` is shown at the beginning of ``stdout`` when running ``bayesmark-launch``. In general, let the launcher create and setup ``DBID`` unless you are appending to a previous experiment, in which case, specify the existing ``DBID``.
The launcher's sequence of commands can be accessed programmatically via :func:`.experiment_launcher.gen_commands`. The individual experiments can be launched programmatically via :func:`.experiment.run_sklearn_study`.
Selecting the experiments
^^^^^^^^^^^^^^^^^^^^^^^^^
A list of optimizers, classifiers, data sets, and metrics can be listed using the ``-o``/``-c``/``-d``/``-m`` commands, respectively. If not specified, the program launches all possible options.
Selecting the optimizer
^^^^^^^^^^^^^^^^^^^^^^^
A few different open source optimizers have been included as an example and are considered the "built-in" optimizers. The original repos are shown in the `Links <#links>`_.
The data argument ``-o`` allows a list containing the "built-in" optimizers:
.. code-block::
"HyperOpt", "Nevergrad-OnePlusOne", "OpenTuner-BanditA", "OpenTuner-GA", "OpenTuner-GA-DE", "PySOT", "RandomSearch", "Scikit-GBRT-Hedge", "Scikit-GP-Hedge", "Scikit-GP-LCB"
or, one can specify a user-defined optimizer. The class containing an optimizer conforming to the API must be found in in the folder specified by ``--opt-root``. Additionally, a configuration defining each optimizer must be defined in ``[OPT_ROOT]/config.json``. The ``--opt-root`` and ``config.json`` may be omitted if only built-in optimizers are used.
Additional details for providing a new optimizer are found in `adding a new optimizer <#adding-a-new-optimizer>`_.
Selecting the data set
^^^^^^^^^^^^^^^^^^^^^^
By default, this benchmark uses the `sklearn example data sets <https://scikit-learn.org/stable/datasets/index.html#toy-datasets>`_ as the "built-in" data sets for use in ML model tuning problems.
The data argument ``-d`` allows a list containing the "built-in" data sets:
.. code-block::
"breast", "digits", "iris", "wine", "boston", "diabetes"
or, it can refer to a custom ``csv`` file, which is the name of file in the folder specified by ``--data-root``. It also follows the convention that regression data sets start with ``reg-`` and classification data sets start with ``clf-``. For example, the classification data set in ``[DATA_ROOT]/clf-foo.csv`` is specified with ``-d clf-foo``.
The ``csv`` file can be anything readable by pandas, but we assume the final column is the target and all other columns are features. The target column should be integer for classification data and float for regression. The features should float (or ``str`` for categorical variable columns). See ``bayesmark.data.load_data`` for more information.
Dry run for cluster jobs
^^^^^^^^^^^^^^^^^^^^^^^^
It is also possible to do a "dry run" of the launcher by specifying a value for ``--num-jobs`` greater than zero. For example, if ``--num-jobs 50`` is provided, a text file listing 50 commands to run is produced, with one command (job) per line. This is useful when preparing a list of commands to run later on a cluster.
A dry run will generate a command file (e.g., ``jobs.txt``) like the following (with a meta-data header). Each line corresponds to a command that can be used as a job on a different worker:
.. code-block::
# running: {'--uuid': None, '-db-root': '/foo', '--opt-root': '/example_opt_root', '--data-root': None, '--db': 'bo_example_folder', '--opt': ['RandomSearch', 'PySOT'], '--data': None, '--classifier': ['SVM', 'DT'], '--metric': None, '--calls': 15, '--suggestions': 1, '--repeat': 3, '--num-jobs': 50, '--jobs-file': '/jobs.txt', '--verbose': False, 'dry_run': True, 'rev': '9a14ef2', 'opt_rev': None}
# cmd: python bayesmark-launch -n 15 -r 3 -dir foo -o RandomSearch PySOT -c SVM DT -nj 50 -b bo_example_folder
job_e2b63a9_00 bayesmark-exp -c SVM -d diabetes -o PySOT -u 079a155f03095d2ba414a5d2cedde08c -m mse -n 15 -p 1 -dir foo -b bo_example_folder && bayesmark-exp -c SVM -d boston -o RandomSearch -u 400e4c0be8295ad59db22d9b5f31d153 -m mse -n 15 -p 1 -dir foo -b bo_example_folder && bayesmark-exp -c SVM -d digits -o RandomSearch -u fe73a2aa960a5e3f8d78bfc4bcf51428 -m acc -n 15 -p 1 -dir foo -b bo_example_folder
job_e2b63a9_01 bayesmark-exp -c DT -d diabetes -o PySOT -u db1d9297948554e096006c172a0486fb -m mse -n 15 -p 1 -dir foo -b bo_example_folder && bayesmark-exp -c SVM -d boston -o RandomSearch -u 7148f690ed6a543890639cc59db8320b -m mse -n 15 -p 1 -dir foo -b bo_example_folder && bayesmark-exp -c SVM -d breast -o PySOT -u 72c104ba1b6d5bb8a546b0064a7c52b1 -m nll -n 15 -p 1 -dir foo -b bo_example_folder
job_e2b63a9_02 bayesmark-exp -c SVM -d iris -o PySOT -u cc63b2c1e4315a9aac0f5f7b496bfb0f -m nll -n 15 -p 1 -dir foo -b bo_example_folder && bayesmark-exp -c DT -d breast -o RandomSearch -u aec62e1c8b5552e6b12836f0c59c1681 -m nll -n 15 -p 1 -dir foo -b bo_example_folder && bayesmark-exp -c DT -d digits -o RandomSearch -u 4d0a175d56105b6bb3055c3b62937b2d -m acc -n 15 -p 1 -dir foo -b bo_example_folder
...
This package does not have built in support for deploying these jobs on a cluster or cloud environment (.e.g., AWS).
The UUID argument
^^^^^^^^^^^^^^^^^
The ``UUID`` is a 32-char hex string used as a master random seed which we use to draw random seeds for the experiments. If ``UUID`` is not specified a version 4 UUID is generated. The used UUID is displayed at the beginning of ``stdout``. In general, the ``UUID`` should not specified/re-used except for debugging because it violates the assumption that the experiment UUIDs are unique.
Aggregate results
-----------------
Next to aggregate all the experiment files into combined (json) files we need to run the aggregation command:
.. code-block::
usage: bayesmark-agg [-h] [-dir DB_ROOT] [-odir OPTIMIZER_ROOT] [-v] -b DB [-rv]
The arguments are:
.. code-block::
-h, --help show this help message and exit
-dir DB_ROOT, -db-root DB_ROOT
root directory for all benchmark experiments output
-odir OPTIMIZER_ROOT, --opt-root OPTIMIZER_ROOT
Directory with optimization wrappers
-v, --verbose print the study logs to console
-b DB, --db DB database ID of this benchmark experiment
-rv, --ravel ravel all studies to store batch suggestions as if
they were serial
The ``DB_ROOT`` must match the folder from the launcher ``bayesmark-launch``, and ``DBID`` must match that displayed from the launcher as well. The aggregate files are found in ``[DB_ROOT]/[DBID]/derived``.
The result aggregation can be done programmatically via :func:`.experiment_aggregate.concat_experiments`.
Analyze and summarize results
-----------------------------
Finally, to run a statistical analysis presenting a summary of the experiments we run
.. code-block::
usage: bayesmark-anal [-h] [-dir DB_ROOT] [-odir OPTIMIZER_ROOT] [-v] -b DB
The arguments are:
.. code-block::
-h, --help show this help message and exit
-dir DB_ROOT, -db-root DB_ROOT
root directory for all benchmark experiments output
-odir OPTIMIZER_ROOT, --opt-root OPTIMIZER_ROOT
Directory with optimization wrappers
-v, --verbose print the study logs to console
-b DB, --db DB database ID of this benchmark experiment
The ``DB_ROOT`` must match the folder from the launcher ``bayesmark-launch``, and ``DBID`` must match that displayed from the launcher as well. The aggregate files are found in ``[DB_ROOT]/[DBID]/derived``.
The ``bayesmark-anal`` command looks for a ``baseline.json`` file in ``[DB_ROOT]/[DBID]/derived``, which states the best possible and random search performance. If no such file is present, ``bayesmark-anal`` automatically calls ``bayesmark-baseline`` to build it. The baselines are inferred from the random search performance in the logs. The baseline values are considered fixed (not random) quantities when ``bayesmark-anal`` builds confidence intervals. Therefore, we allow the user to leave them fixed and do not rebuild them when ``bayesmark-anal`` is called if a baselines file is already present.
The result analysis can be done programmatically via :func:`.experiment_analysis.compute_aggregates`, and the baseline computation via :func:`.experiment_baseline.compute_baseline`.
See :ref:`how-scoring-works` for more information on how the scores are computed and aggregated.
Example
-------
After finishing the setup (environment) a small-scale serial can be run as follows:
.. code-block:: console
> # setup
> DB_ROOT=./notebooks # path/to/where/you/put/results
> DBID=bo_example_folder
> mkdir $DB_ROOT
> # experiments
> bayesmark-launch -n 15 -r 3 -dir $DB_ROOT -b $DBID -o RandomSearch PySOT -c SVM DT -v
Supply --uuid 3adc3182635e44ea96969d267591f034 to reproduce this run.
Supply --dbid bo_example_folder to append to this experiment or reproduce jobs file.
User must ensure equal reps of each optimizer for unbiased results
-c DT -d boston -o PySOT -u a1b287b450385ad09b2abd7582f404a2 -m mae -n 15 -p 1 -dir /notebooks -b bo_example_folder
-c DT -d boston -o PySOT -u 63746599ae3f5111a96942d930ba1898 -m mse -n 15 -p 1 -dir /notebooks -b bo_example_folder
-c DT -d boston -o RandomSearch -u 8ba16c880ef45b27ba0909199ab7aa8a -m mae -n 15 -p 1 -dir /notebooks -b bo_example_folder
...
0 failures of benchmark script after 144 studies.
done
> # aggregate
> bayesmark-agg -dir $DB_ROOT -b $DBID
> # analyze
> bayesmark-anal -dir $DB_ROOT -b $DBID -v
...
median score @ 15:
optimizer
PySOT_0.2.3_9b766b6 0.330404
RandomSearch_0.0.1_9b766b6 0.961829
mean score @ 15:
optimizer
PySOT_0.2.3_9b766b6 0.124262
RandomSearch_0.0.1_9b766b6 0.256422
normed mean score @ 15:
optimizer
PySOT_0.2.3_9b766b6 0.475775
RandomSearch_0.0.1_9b766b6 0.981787
done
The aggregate result files (i.e., ``summary.json``) will now be available in ``$DB_ROOT/$DBID/derived``. However, this will be high variance since it was from only 3 trials and only to 15 function evaluations.
Plotting and notebooks
----------------------
Plotting the quantitative results found in ``$DB_ROOT/$DBID/derived`` can be done using the notebooks found in the ``notebooks/`` folder of the git repository. The notebook ``plot_mean_score.ipynb`` generates plots for aggregate scores averaging over all problems. The notebook ``plot_test_case.ipynb`` generates plots for each test problem.
To use the notebooks, first copy over the ``notebooks/`` folder from git repository.
To setup the kernel for running the notebooks use:
.. code-block:: bash
virtualenv bobm_ipynb --python=python3.6
source ./bobm_ipynb/bin/activate
pip install bayesmark[notebooks]
python -m ipykernel install --name=bobm_ipynb --user
Now, the notebooks for plotting can be run with the command ``jupyter notebook`` and selecting the kernel ``bobm_ipynb``.
It is also possible to convert the notebooks to an HTML report at the command line using ``nbconvert``. For example, use the command:
.. code-block:: bash
jupyter nbconvert --to html --execute notebooks/plot_mean_score.ipynb
The output file will be in ``./notebooks/plot_mean_score.html``. Here is an example `export <https://github.com/uber/bayesmark/files/3699241/plot_mean_score.pdf>`_. See the ``nbconvert`` `documentation page <https://nbconvert.readthedocs.io/en/latest/usage.html#supported-output-formats>`_ for more output formats. By default, the notebooks look in ``./notebooks/bo_example_folder/`` for the ``summary.json`` from ``bayesmark-anal``.
To run ``plot_test_case.ipynb`` use the command:
.. code-block:: bash
jupyter nbconvert --to html --execute notebooks/plot_test_case.ipynb --ExecutePreprocessor.timeout=600
The ``--ExecutePreprocessor.timeout=600`` timeout increase is needed due to the large number of plots being generated. The output will be in ``./notebooks/plot_test_case.html``.
Adding a new optimizer
======================
All optimizers in this benchmark are required to follow the interface specified of the ``AbstractOptimizer`` class in ``bayesmark.abstract_optimizer``. In general, this requires creating a wrapper class around the new optimizer. The wrapper classes must all be placed in a folder referred to by the ``--opt-root`` argument. This folder must also contain the ``config.json`` folder.
The interface is simple, one must merely implement the ``suggest`` and ``observe`` functions. The ``suggest`` function generates new guesses for evaluating the function. Once evaluated, the function evaluations are passed to the ``observe`` function. The objective function is *not* evaluated by the optimizer class. The objective function is evaluated on outside and results are passed to ``observe``. This is the correct setup for Bayesian optimization because:
* We can observe/try inputs that were never suggested
* We can ignore suggestions
* The objective function may not be something as simple as a Python function
So passing the function as an argument as is done in ``scipy.optimization`` is artificially restrictive.
The implementation of the wrapper will look like the following:
.. code-block:: python
from bayesmark.abstract_optimizer import AbstractOptimizer
from bayesmark.experiment import experiment_main
class NewOptimizerName(AbstractOptimizer):
# Used for determining the version number of package used
primary_import = "name of import used e.g, opentuner"
def __init__(self, api_config, optional_arg_foo=None, optional_arg_bar=None):
"""Build wrapper class to use optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
"""
AbstractOptimizer.__init__(self, api_config)
# Do whatever other setup is needed
# ...
def suggest(self, n_suggestions=1):
"""Get suggestion from the optimizer.
Parameters
----------
n_suggestions : int
Desired number of parallel suggestions in the output
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
# Do whatever is needed to get the parallel guesses
# ...
return x_guess
def observe(self, X, y):
"""Feed an observation back.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated
"""
# Update the model with new objective function observations
# ...
# No return statement needed
if __name__ == "__main__":
# This is the entry point for experiments, so pass the class to experiment_main to use this optimizer.
# This statement must be included in the wrapper class file:
experiment_main(NewOptimizerName)
Depending on the API of the optimizer being wrapped, building this wrapper class may only or require a few lines of code, or be a total pain.
The config file
---------------
Note: A config file is now optional. If no ``config.json`` is provided, the experiment launcher will look for all folders with an `optimizer.py` in the ``--opt-root`` directory.
Each optimizer wrapper can have multiple configurations, which is each referred to as a different optimizer in the benchmark. For example, the JSON config file will have entries as follows:
.. code-block:: json
{
"OpenTuner-BanditA-New": [
"opentuner_optimizer.py",
{"techniques": ["AUCBanditMetaTechniqueA"]}
],
"OpenTuner-GA-DE-New": [
"opentuner_optimizer.py",
{"techniques": ["PSO_GA_DE"]}
],
"OpenTuner-GA-New": [
"opentuner_optimizer.py",
{"techniques": ["PSO_GA_Bandit"]}
]
}
Basically, the entries are ``"name_of_strategy": ["file_with_class", {kwargs_for_the_constructor}]``. Here, ``OpenTuner-BanditA``, ``OpenTuner-GA-DE``, and ``OpenTuner-GA`` are all treated as different optimizers by the benchmark even though the all use the same class from ``opentuner_optimizer.py``.
This ``config.json`` must be in the same folder as the optimizer classes (e.g., ``opentuner_optimizer.py``).
Running with a new optimizer
----------------------------
To run the benchmarks using a new optimizer, simply provide its name (from ``config.json``) in the ``-o`` list. The ``--opt-root`` argument must be specified in this case. For example, the launch command from the `example <#example>`_ becomes:
.. code-block:: bash
bayesmark-launch -n 15 -r 3 -dir $DB_ROOT -b $DBID -o RandomSearch PySOT-New -c SVM DT --opt-root ./example_opt_root -v
Here, we are using the example ``PySOT-New`` wrapper from the ``example_opt_root`` folder in the git repo. It is equivalent to the builtin ``PySOT``, but gives an example of how to provide a new custom optimizer.
Contributing
============
The following instructions have been tested with Python 3.6.8 on Ubuntu (16.04.5 LTS).
Install in editable mode
------------------------
First, define the variables for the paths we will use:
.. code-block:: bash
GIT=/path/to/where/you/put/repos
ENVS=/path/to/where/you/put/virtualenvs
Then clone the repo in your git directory ``$GIT``:
.. code-block:: bash
cd $GIT
git clone https://github.com/uber/bayesmark.git
Inside your virtual environments folder ``$ENVS``, make the environment:
.. code-block:: bash
cd $ENVS
virtualenv bayesmark --python=python3.6
source $ENVS/bayesmark/bin/activate
Now we can install the pip dependencies. Move back into your git directory and run
.. code-block:: bash
cd $GIT/bayesmark
pip install -r requirements/base.txt
pip install -r requirements/optimizers.txt
pip install -e . # Install the benchmark itself
You may want to run ``pip install -U pip`` first if you have an old version of ``pip``. The file ``optimizers.txt`` contains the dependencies for all the optimizers used in the benchmark. The analysis and aggregation programs can be run using only the requirements in ``base.txt``.
Contributor tools
-----------------
First, we need to setup some needed tools:
.. code-block:: bash
cd $ENVS
virtualenv bayesmark_tools --python=python3.6
source $ENVS/bayesmark_tools/bin/activate
pip install -r $GIT/bayesmark/requirements/tools.txt
To install the pre-commit hooks for contributing run (in the ``bayesmark_tools`` environment):
.. code-block:: bash
cd $GIT/bayesmark
pre-commit install
To rebuild the requirements, we can run:
.. code-block:: bash
cd $GIT/bayesmark
# Get py files from notebooks to analyze
jupyter nbconvert --to script notebooks/*.ipynb
# Generate the .in files (but pins to latest, which we might not want)
pipreqs bayesmark/ --ignore bayesmark/builtin_opt/ --savepath requirements/base.in
pipreqs test/ --savepath requirements/test.in
pipreqs bayesmark/builtin_opt/ --savepath requirements/optimizers.in
pipreqs notebooks/ --savepath requirements/ipynb.in
pipreqs docs/ --savepath requirements/docs.in
# Regenerate the .txt files from .in files
pip-compile-multi --no-upgrade
Generating the documentation
----------------------------
First setup the environment for building with ``Sphinx``:
.. code-block:: bash
cd $ENVS
virtualenv bayesmark_docs --python=python3.6
source $ENVS/bayesmark_docs/bin/activate
pip install -r $GIT/bayesmark/requirements/docs.txt
Then we can do the build:
.. code-block:: bash
cd $GIT/bayesmark/docs
make all
open _build/html/index.html
Documentation will be available in all formats in ``Makefile``. Use ``make html`` to only generate the HTML documentation.
Running the tests
-----------------
The tests for this package can be run with:
.. code-block:: bash
cd $GIT/bayesmark
./test.sh
The script creates a conda environment using the requirements found in ``requirements/test.txt``.
The ``test.sh`` script *must* be run from a *clean* git repo.
Or if we only want to run the unit tests and not check the adequacy of the requirements files, one can use
.. code-block:: bash
# Setup environment
cd $ENVS
virtualenv bayesmark_test --python=python3.6
source $ENVS/bayesmark_test/bin/activate
pip install -r $GIT/bayesmark/requirements/test.txt
pip install -e $GIT/bayesmark
# Now run tests
cd $GIT/bayesmark/
pytest test/ -s -v --hypothesis-seed=0 --disable-pytest-warnings --cov=bayesmark --cov-report html
A code coverage report will also be produced in ``$GIT/bayesmark/htmlcov/index.html``.
Deployment
----------
The wheel (tar ball) for deployment as a pip installable package can be built using the script:
.. code-block:: bash
cd $GIT/bayesmark/
./build_wheel.sh
Links
=====
The `source <https://github.com/uber/bayesmark>`_ is hosted on GitHub.
The `documentation <https://bayesmark.readthedocs.io/en/latest/>`_ is hosted at Read the Docs.
Installable from `PyPI <https://pypi.org/project/bayesmark/>`_.
The builtin optimizers are wrappers on the following projects:
* `HyperOpt <https://github.com/hyperopt/hyperopt>`_
* `Nevergrad <https://github.com/facebookresearch/nevergrad>`_
* `OpenTuner <https://github.com/jansel/opentuner>`_
* `PySOT <https://github.com/dme65/pySOT>`_
* `Scikit-optimize <https://github.com/scikit-optimize/scikit-optimize>`_
License
=======
This project is licensed under the Apache 2 License - see the LICENSE file for details.
================================================
FILE: bayesmark/__init__.py
================================================
__version__ = "0.0.8"
__author__ = "Ryan Turner"
__license__ = "Apache v2"
================================================
FILE: bayesmark/abstract_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Abstract base class for the optimizers in the benchmark. This creates a common API across all packages.
"""
from abc import ABC, abstractmethod
from importlib_metadata import version
class AbstractOptimizer(ABC):
"""Abstract base class for the optimizers in the benchmark. This creates a common API across all packages.
"""
# Every implementation package needs to specify this static variable, e.g., "primary_import=opentuner"
primary_import = None
def __init__(self, api_config, **kwargs):
"""Build wrapper class to use an optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
"""
self.api_config = api_config
@classmethod
def get_version(cls):
"""Get the version for this optimizer.
Returns
-------
version_str : str
Version number of the optimizer. Usually, this is equivalent to ``package.__version__``.
"""
assert (cls.primary_import is None) or isinstance(cls.primary_import, str)
# Should use x.x.x as version if sub-class did not specify its primary import
version_str = "x.x.x" if cls.primary_import is None else version(cls.primary_import)
return version_str
@abstractmethod
def suggest(self, n_suggestions):
"""Get a suggestion from the optimizer.
Parameters
----------
n_suggestions : int
Desired number of parallel suggestions in the output
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
pass
@abstractmethod
def observe(self, X, y):
"""Send an observation of a suggestion back to the optimizer.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated
"""
pass
================================================
FILE: bayesmark/builtin_opt/__init__.py
================================================
================================================
FILE: bayesmark/builtin_opt/config.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from bayesmark.constants import RANDOM_SEARCH
CONFIG = {
"HyperOpt": ["hyperopt_optimizer.py", {}],
"Nevergrad-OnePlusOne": ["nevergrad_optimizer.py", {"budget": 300, "tool": "OnePlusOne"}],
"OpenTuner-BanditA": ["opentuner_optimizer.py", {"techniques": ["AUCBanditMetaTechniqueA"]}],
"OpenTuner-GA": ["opentuner_optimizer.py", {"techniques": ["PSO_GA_Bandit"]}],
"OpenTuner-GA-DE": ["opentuner_optimizer.py", {"techniques": ["PSO_GA_DE"]}],
"PySOT": ["pysot_optimizer.py", {}],
"RandomSearch": ["random_optimizer.py", {}],
"Scikit-GBRT-Hedge": [
"scikit_optimizer.py",
{"acq_func": "gp_hedge", "base_estimator": "GBRT", "n_initial_points": 5},
],
"Scikit-GP-Hedge": ["scikit_optimizer.py", {"acq_func": "gp_hedge", "base_estimator": "GP", "n_initial_points": 5}],
"Scikit-GP-LCB": ["scikit_optimizer.py", {"acq_func": "LCB", "base_estimator": "GP", "n_initial_points": 5}],
}
assert RANDOM_SEARCH in CONFIG, "%s required in settings file." % RANDOM_SEARCH
================================================
FILE: bayesmark/builtin_opt/hyperopt_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import numpy as np
from hyperopt import hp, tpe
from hyperopt.base import JOB_STATE_DONE, JOB_STATE_NEW, STATUS_OK, Domain, Trials
from scipy.interpolate import interp1d
from bayesmark.abstract_optimizer import AbstractOptimizer
from bayesmark.np_util import random as np_random
from bayesmark.np_util import random_seed
# Sklearn prefers str to unicode:
DTYPE_MAP = {"real": float, "int": int, "bool": bool, "cat": str, "ordinal": str}
def dummy_f(x):
assert False, "This is a placeholder, it should never be called."
def only(x):
y, = x
return y
class HyperoptOptimizer(AbstractOptimizer):
primary_import = "hyperopt"
def __init__(self, api_config, random=np_random):
"""Build wrapper class to use hyperopt optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
"""
AbstractOptimizer.__init__(self, api_config)
self.random = random
space, self.round_to_values = HyperoptOptimizer.get_hyperopt_dimensions(api_config)
self.domain = Domain(dummy_f, space, pass_expr_memo_ctrl=None)
self.trials = Trials()
# Some book keeping like opentuner wrapper
self.trial_id_lookup = {}
# Store just for data validation
self.param_set_chk = frozenset(api_config.keys())
@staticmethod
def hashable_dict(d):
"""A custom function for hashing dictionaries.
Parameters
----------
d : dict or dict-like
The dictionary to be converted to immutable/hashable type.
Returns
-------
hashable_object : frozenset of tuple pairs
Bijective equivalent to dict that can be hashed.
"""
hashable_object = frozenset(d.items())
return hashable_object
@staticmethod
def get_hyperopt_dimensions(api_config):
"""Help routine to setup hyperopt search space in constructor.
Take api_config as argument so this can be static.
"""
# The ordering of iteration prob makes no difference, but just to be
# safe and consistnent with space.py, I will make sorted.
param_list = sorted(api_config.keys())
space = {}
round_to_values = {}
for param_name in param_list:
param_config = api_config[param_name]
param_type = param_config["type"]
param_space = param_config.get("space", None)
param_range = param_config.get("range", None)
param_values = param_config.get("values", None)
# Some setup for case that whitelist of values is provided:
values_only_type = param_type in ("cat", "ordinal")
if (param_values is not None) and (not values_only_type):
assert param_range is None
param_values = np.unique(param_values)
param_range = (param_values[0], param_values[-1])
round_to_values[param_name] = interp1d(
param_values, param_values, kind="nearest", fill_value="extrapolate"
)
if param_type == "int":
low, high = param_range
if param_space in ("log", "logit"):
space[param_name] = hp.qloguniform(param_name, np.log(low), np.log(high), 1)
else:
space[param_name] = hp.quniform(param_name, low, high, 1)
elif param_type == "bool":
assert param_range is None
assert param_values is None
space[param_name] = hp.choice(param_name, (False, True))
elif param_type in ("cat", "ordinal"):
assert param_range is None
space[param_name] = hp.choice(param_name, param_values)
elif param_type == "real":
low, high = param_range
if param_space in ("log", "logit"):
space[param_name] = hp.loguniform(param_name, np.log(low), np.log(high))
else:
space[param_name] = hp.uniform(param_name, low, high)
else:
assert False, "type %s not handled in API" % param_type
return space, round_to_values
def get_trial(self, trial_id):
for trial in self.trials._dynamic_trials:
if trial["tid"] == trial_id:
assert isinstance(trial, dict)
# Make sure right kind of dict
assert "state" in trial and "result" in trial
assert trial["state"] == JOB_STATE_NEW
return trial
assert False, "No matching trial ID"
def cleanup_guess(self, x_guess):
assert isinstance(x_guess, dict)
# Also, check the keys are only the vars we are searching over:
assert frozenset(x_guess.keys()) == self.param_set_chk
# Do the rounding
# Make a copy to be safe, and also unpack singletons
# We may also need to consider clip_chk at some point like opentuner
x_guess = {k: only(x_guess[k]) for k in x_guess}
for param_name, round_f in self.round_to_values.items():
x_guess[param_name] = round_f(x_guess[param_name])
# Also ensure this is correct dtype so sklearn is happy
x_guess = {k: DTYPE_MAP[self.api_config[k]["type"]](x_guess[k]) for k in x_guess}
return x_guess
def _suggest(self):
"""Helper function to `suggest` that does the work of calling
`hyperopt` via its dumb API.
"""
new_ids = self.trials.new_trial_ids(1)
assert len(new_ids) == 1
self.trials.refresh()
seed = random_seed(self.random)
new_trials = tpe.suggest(new_ids, self.domain, self.trials, seed)
assert len(new_trials) == 1
self.trials.insert_trial_docs(new_trials)
self.trials.refresh()
new_trial, = new_trials # extract singleton
return new_trial
def suggest(self, n_suggestions=1):
"""Make `n_suggestions` suggestions for what to evaluate next.
This requires the user observe all previous suggestions before calling
again.
Parameters
----------
n_suggestions : int
The number of suggestions to return.
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
assert n_suggestions >= 1, "invalid value for n_suggestions"
# Get the new trials, it seems hyperopt either uses random search or
# guesses one at a time anyway, so we might as welll call serially.
new_trials = [self._suggest() for _ in range(n_suggestions)]
X = []
for trial in new_trials:
x_guess = self.cleanup_guess(trial["misc"]["vals"])
X.append(x_guess)
# Build lookup to get original trial object
x_guess_ = HyperoptOptimizer.hashable_dict(x_guess)
assert x_guess_ not in self.trial_id_lookup, "the suggestions should not already be in the trial dict"
self.trial_id_lookup[x_guess_] = trial["tid"]
assert len(X) == n_suggestions
return X
def observe(self, X, y):
"""Feed the observations back to hyperopt.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated.
"""
assert len(X) == len(y)
for x_guess, y_ in zip(X, y):
x_guess_ = HyperoptOptimizer.hashable_dict(x_guess)
assert x_guess_ in self.trial_id_lookup, "Appears to be guess that did not originate from suggest"
trial_id = self.trial_id_lookup.pop(x_guess_)
trial = self.get_trial(trial_id)
assert self.cleanup_guess(trial["misc"]["vals"]) == x_guess, "trial ID not consistent with x values stored"
# Cast to float to ensure native type
result = {"loss": float(y_), "status": STATUS_OK}
trial["state"] = JOB_STATE_DONE
trial["result"] = result
# hyperopt.fmin.FMinIter.serial_evaluate only does one refresh at end
# of loop of a bunch of evals, so we will do the same thing here.
self.trials.refresh()
opt_wrapper = HyperoptOptimizer
================================================
FILE: bayesmark/builtin_opt/nevergrad_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import nevergrad.optimization as optimization
import numpy as np
from nevergrad import instrumentation as inst
from scipy.stats import norm
from bayesmark.abstract_optimizer import AbstractOptimizer
from bayesmark.np_util import linear_rescale
from bayesmark.space import Real
class NevergradOptimizer(AbstractOptimizer):
primary_import = "nevergrad"
def __init__(self, api_config, tool, budget):
"""Build wrapper class to use nevergrad optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
budget : int
Expected number of max function evals
"""
AbstractOptimizer.__init__(self, api_config)
self.instrum, self.space = NevergradOptimizer.get_nvg_dimensions(api_config)
dimension = self.instrum.dimension
opt_class = optimization.registry[tool]
self.optim = opt_class(dimension=dimension, budget=budget)
@staticmethod
def get_nvg_dimensions(api_config):
"""Help routine to setup nevergrad search space in constructor.
Take api_config as argument so this can be static.
"""
# The ordering of iteration prob makes no difference, but just to be
# safe and consistnent with space.py, I will make sorted.
param_list = sorted(api_config.keys())
all_args = {}
all_prewarp = {}
for param_name in param_list:
param_config = api_config[param_name]
param_type = param_config["type"]
param_space = param_config.get("space", None)
param_range = param_config.get("range", None)
param_values = param_config.get("values", None)
prewarp = None
if param_type == "cat":
assert param_space is None
assert param_range is None
arg = inst.var.SoftmaxCategorical(param_values)
elif param_type == "bool":
assert param_space is None
assert param_range is None
assert param_values is None
arg = inst.var.OrderedDiscrete([False, True])
elif param_values is not None:
assert param_type in ("int", "ordinal", "real")
arg = inst.var.OrderedDiscrete(param_values)
# We are throwing away information here, but OrderedDiscrete
# appears to be invariant to monotonic transformation anyway.
elif param_type == "int":
assert param_values is None
# Need +1 since API in inclusive
choices = range(int(param_range[0]), int(param_range[-1]) + 1)
arg = inst.var.OrderedDiscrete(choices)
# We are throwing away information here, but OrderedDiscrete
# appears to be invariant to monotonic transformation anyway.
elif param_type == "real":
assert param_values is None
assert param_range is not None
# Will need to warp to this space sep.
arg = inst.var.Gaussian(mean=0, std=1)
prewarp = Real(warp=param_space, range_=param_range)
else:
assert False, "type %s not handled in API" % param_type
all_args[param_name] = arg
all_prewarp[param_name] = prewarp
instrum = inst.Instrumentation(**all_args)
return instrum, all_prewarp
def prewarp(self, xx):
"""Extra work needed to get variables into the Gaussian space
representation."""
xxw = {}
for arg_name, vv in xx.items():
assert np.isscalar(vv)
space = self.space[arg_name]
if space is not None:
# Warp so we think it is apriori uniform in [a, b]
vv = space.warp(vv)
assert vv.size == 1
# Now make uniform on [0, 1], also unpack warped to scalar
(lb, ub), = space.get_bounds()
vv = linear_rescale(vv.item(), lb, ub, 0, 1)
# Now make std Gaussian apriori
vv = norm.ppf(vv)
assert np.isscalar(vv)
xxw[arg_name] = vv
return xxw
def postwarp(self, xxw):
"""Extra work needed to undo the Gaussian space representation."""
xx = {}
for arg_name, vv in xxw.items():
assert np.isscalar(vv)
space = self.space[arg_name]
if space is not None:
# Now make std Gaussian apriori
vv = norm.cdf(vv)
# Now make uniform on [0, 1]
(lb, ub), = space.get_bounds()
vv = linear_rescale(vv, 0, 1, lb, ub)
# Warp so we think it is apriori uniform in [a, b]
vv = space.unwarp([vv])
assert np.isscalar(vv)
xx[arg_name] = vv
return xx
def suggest(self, n_suggestions=1):
"""Get suggestion from nevergrad.
Parameters
----------
n_suggestions : int
Desired number of parallel suggestions in the output
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
x_guess_data = [self.optim.ask() for _ in range(n_suggestions)]
x_guess = [None] * n_suggestions
for ii, xx in enumerate(x_guess_data):
x_pos, x_kwarg = self.instrum.data_to_arguments(xx)
assert x_pos == ()
x_guess[ii] = self.postwarp(x_kwarg)
return x_guess
def observe(self, X, y):
"""Feed an observation back to nevergrad.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated
"""
for xx, yy in zip(X, y):
xx = self.prewarp(xx)
xx = self.instrum.arguments_to_data(**xx)
self.optim.tell(xx, yy)
opt_wrapper = NevergradOptimizer
================================================
FILE: bayesmark/builtin_opt/opentuner_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
In opentuner, many search techniques are already available. All the names of
the techniques can be found as follows:
```
>>> import opentuner
>>> techniques, generators = opentuner.search.technique.all_techniques()
>>> for t in techniques:
... print t.name
```
A user can also create new search techniques
(http://opentuner.org/tutorial/techniques/).
Opentuner will create a multi-arm bandit of multiple techniques if more than
one technique is specified in `args.technique`.
Some bandits with pre-defined techniques are already registered in:
`opentuner.search.bandittechniques`
By default, we use a pre-defined bandit called `'AUCBanditMetaTechniqueA'` of 4
techniques:
```
register(AUCBanditMetaTechnique([
differentialevolution.DifferentialEvolutionAlt(),
evolutionarytechniques.UniformGreedyMutation(),
evolutionarytechniques.NormalGreedyMutation(mutation_rate=0.3),
simplextechniques.RandomNelderMead()],
name='AUCBanditMetaTechniqueA'))
```
The other two bandits used in our experiments are: PSO_GA_DE and PSO_GA_Bandit.
Specifying a list of multiple techniques will use a multi-arm bandit over them.
"""
import warnings
from argparse import Namespace
import opentuner.tuningrunmain
from opentuner.api import TuningRunManager
from opentuner.measurement.interface import DefaultMeasurementInterface as DMI
from opentuner.resultsdb.models import DesiredResult, Result
from opentuner.search.manipulator import (
ConfigurationManipulator,
EnumParameter,
FloatParameter,
IntegerParameter,
LogFloatParameter,
LogIntegerParameter,
ScaledNumericParameter,
)
from bayesmark.abstract_optimizer import AbstractOptimizer
from bayesmark.np_util import clip_chk
DEFAULT_TECHNIQUES = ("AUCBanditMetaTechniqueA",)
MEMORY_ONLY_DB = "sqlite://"
# Monkey patch here! Opentuner is messed up, TuningRunMain changes global log
# settings. We should file in issue report here and have them fix it.
opentuner.tuningrunmain.init_logging = lambda: None
def ClippedParam(cls, epsilon=1e-5):
"""Build wrapper class of opentuner parameter class that use clip check to
keep parameters in the allowed range despite numerical errors.
Class built on `ScaledNumericParameter` abstract class defined in:
`opentuner.search.manipulator.ScaledNumericParameter`.
Parameters
----------
cls : ScaledNumericParameter
Opentuner parameter class, such as `LogFloatParameter` or
`FloatParameter`, which transforms the domain of parameter.
Returns
-------
StableClass : ScaledNumericParameter
New class equivalent to original `cls` but it overwrites the orginal
`_unscale` method to enforce a clip check to keep the parameters within
their allowed range.
"""
assert issubclass(
cls, ScaledNumericParameter
), "this class cls should inherit from the ScaledNumericParameter class"
class StableClass(cls):
def _unscale(self, v):
unscaled_v = super(StableClass, self)._unscale(v)
unscaled_v = clip_chk(unscaled_v, self.min_value, self.max_value)
return unscaled_v
return StableClass
class OpentunerOptimizer(AbstractOptimizer):
primary_import = "opentuner"
def __init__(self, api_config, techniques=DEFAULT_TECHNIQUES, n_suggestions=1):
"""Build wrapper class to use opentuner optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
techniques : iterable of strings
A list or tuple of techniques to use in opentuner. If the list
has only one technique, then that technique will be used. If the
list has multiple techniques a bandit over those techniques
will be used.
n_suggestions : int
Default number of suggestions to be made in parallel.
"""
AbstractOptimizer.__init__(self, api_config)
# Opentuner requires DesiredResult to reference suggestion when making
# its observation. x_to_dr maps the dict suggestion to DesiredResult.
self.x_to_dr = {}
# Keep last suggested x and repeat it whenever opentuner gives up.
self.dummy_suggest = None
"""Setting up the arguments for opentuner. You can see all possible
arguments using:
```
>>> import opentuner
>>> opentuner.default_argparser().parse_args(['-h'])
```
We only change a few arguments (other arguments are set to defaults):
* database = MEMORY_ONLY_DB: to use an in-memory sqlite database
* parallelism = n_suggestions: num of suggestions to give in parallel
* technique = techniques: a list of techniques to be used by opentuner
* print_params = False: to avoid opentuner from exiting after printing
param spaces
"""
args = Namespace(
bail_threshold=500,
database=MEMORY_ONLY_DB,
display_frequency=10,
generate_bandit_technique=False,
label=None,
list_techniques=False,
machine_class=None,
no_dups=False,
parallel_compile=False,
parallelism=n_suggestions,
pipelining=0,
print_params=False,
print_search_space_size=False,
quiet=False,
results_log=None,
results_log_details=None,
seed_configuration=[],
stop_after=None,
technique=techniques,
test_limit=5000,
)
# Setup some dummy classes required by opentuner to actually run.
manipulator = OpentunerOptimizer.build_manipulator(api_config)
interface = DMI(args=args, manipulator=manipulator)
self.api = TuningRunManager(interface, args)
@staticmethod
def hashable_dict(d):
"""A custom function for hashing dictionaries.
Parameters
----------
d : dict or dict-like
The dictionary to be converted to immutable/hashable type.
Returns
-------
hashable_object : frozenset of tuple pairs
Bijective equivalent to dict that can be hashed.
"""
hashable_object = frozenset(d.items())
return hashable_object
@staticmethod
def build_manipulator(api_config):
"""Build a ConfigurationManipulator object to be used by opentuner.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
Returns
-------
manipulator : ConfigurationManipulator
Some over complexified class required by opentuner to run.
"""
manipulator = ConfigurationManipulator()
for pname in api_config:
ptype = api_config[pname]["type"]
pspace = api_config[pname].get("space", None)
pmin, pmax = api_config[pname].get("range", (None, None))
if ptype == "real":
if pspace in ("linear", "logit"):
ot_param = FloatParameter(pname, pmin, pmax)
elif pspace in ("log", "bilog"):
LogFloatParameter_ = ClippedParam(LogFloatParameter)
ot_param = LogFloatParameter_(pname, pmin, pmax)
else:
assert False, "unsupported param space = %s" % pspace
elif ptype == "int":
if pspace in ("linear", "logit"):
ot_param = IntegerParameter(pname, pmin, pmax)
elif pspace in ("log", "bilog"):
ot_param = LogIntegerParameter(pname, pmin, pmax)
else:
assert False, "unsupported param space = %s" % pspace
elif ptype == "bool":
# The actual bool parameter seems not to work in Py3 :(
ot_param = IntegerParameter(pname, 0, 1)
elif ptype in ("cat", "ordinal"):
# Treat ordinal and categorical variables the same for now.
assert "values" in api_config[pname]
pvalues = api_config[pname]["values"]
ot_param = EnumParameter(pname, pvalues)
else:
assert False, "type=%s/space=%s not handled in opentuner yet" % (ptype, pspace)
manipulator.add_parameter(ot_param)
return manipulator
def suggest(self, n_suggestions=1):
"""Make `n_suggestions` suggestions for what to evaluate next.
This requires the user observe all previous suggestions before calling
again.
Parameters
----------
n_suggestions : int
The number of suggestions to return.
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
assert n_suggestions >= 1, "invalid value for n_suggestions"
# Update the n_suggestions if it is different from the current setting.
if self.api.search_driver.args.parallelism != n_suggestions:
self.api.search_driver.args.parallelism = n_suggestions
warnings.warn("n_suggestions changed across suggest calls")
# Require the user to already observe all previous suggestions.
# Otherwise, opentuner will just recycle old suggestions.
assert len(self.x_to_dr) == 0, "all the previous suggestions should have been observed by now"
# The real meat of suggest from opentuner: Get next `n_suggestions`
# unique suggestions.
desired_results = [self.api.get_next_desired_result() for _ in range(n_suggestions)]
# Save DesiredResult object in dict since observe will need it.
X = []
using_dummy_suggest = False
for ii in range(n_suggestions):
# Opentuner can give up, but the API requires guessing forever.
if desired_results[ii] is None:
assert self.dummy_suggest is not None, "opentuner gave up on the first call!"
# Use the dummy suggestion in this case.
X.append(self.dummy_suggest)
using_dummy_suggest = True
continue
# Get the simple dict equivalent to suggestion.
x_guess = desired_results[ii].configuration.data
X.append(x_guess)
# Now save the desired result for future use in observe.
x_guess_ = OpentunerOptimizer.hashable_dict(x_guess)
assert x_guess_ not in self.x_to_dr, "the suggestions should not already be in the x_to_dr dict"
self.x_to_dr[x_guess_] = desired_results[ii]
# This will also catch None from opentuner.
assert isinstance(self.x_to_dr[x_guess_], DesiredResult)
assert len(X) == n_suggestions, "incorrect number of suggestions provided by opentuner"
# Log suggestion for repeating if opentuner gives up next time. We can
# only do this when it is not already being used since it we will be
# checking guesses against dummy_suggest in observe.
if not using_dummy_suggest:
self.dummy_suggest = X[-1]
return X
def observe(self, X, y):
"""Feed the observations back to opentuner.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated.
"""
assert len(X) == len(y)
for x_guess, y_ in zip(X, y):
x_guess_ = OpentunerOptimizer.hashable_dict(x_guess)
# If we can't find the dr object then it must be the dummy guess.
if x_guess_ not in self.x_to_dr:
assert x_guess == self.dummy_suggest, "Appears to be guess that did not originate from suggest"
continue
# Get the corresponding DesiredResult object.
dr = self.x_to_dr.pop(x_guess_, None)
# This will also catch None from opentuner.
assert isinstance(dr, DesiredResult), "DesiredResult object not available in x_to_dr"
# Opentuner's arg names assume we are minimizing execution time.
# So, if we want to minimize we have to pretend y is a 'time'.
result = Result(time=y_)
self.api.report_result(dr, result)
opt_wrapper = OpentunerOptimizer
================================================
FILE: bayesmark/builtin_opt/pysot_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import warnings
from copy import copy
import numpy as np
from poap.strategy import EvalRecord
from pySOT.experimental_design import SymmetricLatinHypercube
from pySOT.optimization_problems import OptimizationProblem
from pySOT.strategy import SRBFStrategy
from pySOT.surrogate import CubicKernel, LinearTail, RBFInterpolant
from bayesmark.abstract_optimizer import AbstractOptimizer
from bayesmark.space import JointSpace
class PySOTOptimizer(AbstractOptimizer):
primary_import = "pysot"
def __init__(self, api_config):
"""Build wrapper class to use an optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
"""
AbstractOptimizer.__init__(self, api_config)
self.space_x = JointSpace(api_config)
self.bounds = self.space_x.get_bounds()
self.create_opt_prob() # Sets up the optimization problem (needs self.bounds)
self.max_evals = np.iinfo(np.int32).max # NOTE: Largest possible int
self.batch_size = None
self.history = []
self.proposals = []
def create_opt_prob(self):
"""Create an optimization problem object."""
opt = OptimizationProblem()
opt.lb = self.bounds[:, 0] # In warped space
opt.ub = self.bounds[:, 1] # In warped space
opt.dim = len(self.bounds)
opt.cont_var = np.arange(len(self.bounds))
opt.int_var = []
assert len(opt.cont_var) + len(opt.int_var) == opt.dim
opt.objfun = None
self.opt = opt
def start(self, max_evals):
"""Starts a new pySOT run."""
self.history = []
self.proposals = []
# Symmetric Latin hypercube design
des_pts = max([self.batch_size, 2 * (self.opt.dim + 1)])
slhd = SymmetricLatinHypercube(dim=self.opt.dim, num_pts=des_pts)
# Warped RBF interpolant
rbf = RBFInterpolant(
dim=self.opt.dim,
lb=self.opt.lb,
ub=self.opt.ub,
kernel=CubicKernel(),
tail=LinearTail(self.opt.dim),
eta=1e-4,
)
# Optimization strategy
self.strategy = SRBFStrategy(
max_evals=self.max_evals,
opt_prob=self.opt,
exp_design=slhd,
surrogate=rbf,
asynchronous=True,
batch_size=1,
use_restarts=True,
)
def suggest(self, n_suggestions=1):
"""Get a suggestion from the optimizer.
Parameters
----------
n_suggestions : int
Desired number of parallel suggestions in the output
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
if self.batch_size is None: # First call to suggest
self.batch_size = n_suggestions
self.start(self.max_evals)
# Set the tolerances pretending like we are running batch
d, p = float(self.opt.dim), float(n_suggestions)
self.strategy.failtol = p * int(max(np.ceil(d / p), np.ceil(4 / p)))
# Now we can make suggestions
x_w = []
self.proposals = []
for _ in range(n_suggestions):
proposal = self.strategy.propose_action()
record = EvalRecord(proposal.args, status="pending")
proposal.record = record
proposal.accept() # This triggers all the callbacks
# It is possible that pySOT proposes a previously evaluated point
# when all variables are integers, so we just abort in this case
# since we have likely converged anyway. See PySOT issue #30.
x = list(proposal.record.params) # From tuple to list
x_unwarped, = self.space_x.unwarp(x)
if x_unwarped in self.history:
warnings.warn("pySOT proposed the same point twice")
self.start(self.max_evals)
return self.suggest(n_suggestions=n_suggestions)
# NOTE: Append unwarped to avoid rounding issues
self.history.append(copy(x_unwarped))
self.proposals.append(proposal)
x_w.append(copy(x_unwarped))
return x_w
def _observe(self, x, y):
# Find the matching proposal and execute its callbacks
idx = [x == xx for xx in self.history]
i = np.argwhere(idx)[0].item() # Pick the first index if there are ties
proposal = self.proposals[i]
proposal.record.complete(y)
self.proposals.pop(i)
self.history.pop(i)
def observe(self, X, y):
"""Send an observation of a suggestion back to the optimizer.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated
"""
assert len(X) == len(y)
for x_, y_ in zip(X, y):
# Just ignore, any inf observations we got, unclear if right thing
if np.isfinite(y_):
self._observe(x_, y_)
opt_wrapper = PySOTOptimizer
================================================
FILE: bayesmark/builtin_opt/random_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import bayesmark.random_search as rs
from bayesmark import np_util
from bayesmark.abstract_optimizer import AbstractOptimizer
class RandomOptimizer(AbstractOptimizer):
# Unclear what is best package to list for primary_import here.
primary_import = "bayesmark"
def __init__(self, api_config, random=np_util.random):
"""Build wrapper class to use random search function in benchmark.
Settings for `suggest_dict` can be passed using kwargs.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
"""
AbstractOptimizer.__init__(self, api_config)
self.random = random
def suggest(self, n_suggestions=1):
"""Get suggestion.
Parameters
----------
n_suggestions : int
Desired number of parallel suggestions in the output
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
x_guess = rs.suggest_dict([], [], self.api_config, n_suggestions=n_suggestions, random=self.random)
return x_guess
def observe(self, X, y):
"""Feed an observation back.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated
"""
# Random search so don't do anything
pass
# All optimizer wrappers need to assign their wrapper to the name opt_wrapper because experiment always tries to import
# opt_wrapper regardless of the optimizer it is importing.
opt_wrapper = RandomOptimizer
================================================
FILE: bayesmark/builtin_opt/scikit_optimizer.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import numpy as np
from scipy.interpolate import interp1d
from skopt import Optimizer as SkOpt
from skopt.space import Categorical, Integer, Real
from bayesmark.abstract_optimizer import AbstractOptimizer
class ScikitOptimizer(AbstractOptimizer):
primary_import = "scikit-optimize"
def __init__(self, api_config, base_estimator="GP", acq_func="gp_hedge", n_initial_points=5, **kwargs):
"""Build wrapper class to use an optimizer in benchmark.
Parameters
----------
api_config : dict-like of dict-like
Configuration of the optimization variables. See API description.
base_estimator : {'GP', 'RF', 'ET', 'GBRT'}
How to estimate the objective function.
acq_func : {'LCB', 'EI', 'PI', 'gp_hedge', 'EIps', 'PIps'}
Acquisition objective to decide next suggestion.
n_initial_points : int
Number of points to sample randomly before actual Bayes opt.
"""
AbstractOptimizer.__init__(self, api_config)
dimensions, self.round_to_values = ScikitOptimizer.get_sk_dimensions(api_config)
# Older versions of skopt don't copy over the dimensions names during
# normalization and hence the names are missing in
# self.skopt.space.dimensions. Therefore, we save our own copy of
# dimensions list to be safe. If we can commit to using the newer
# versions of skopt we can delete self.dimensions.
self.dimensions_list = tuple(dd.name for dd in dimensions)
# Undecided where we want to pass the kwargs, so for now just make sure
# they are blank
assert len(kwargs) == 0
self.skopt = SkOpt(
dimensions,
n_initial_points=n_initial_points,
base_estimator=base_estimator,
acq_func=acq_func,
acq_optimizer="auto",
acq_func_kwargs={},
acq_optimizer_kwargs={},
)
@staticmethod
def get_sk_dimensions(api_config, transform="normalize"):
"""Help routine to setup skopt search space in constructor.
Take api_config as argument so this can be static.
"""
# The ordering of iteration prob makes no difference, but just to be
# safe and consistnent with space.py, I will make sorted.
param_list = sorted(api_config.keys())
sk_dims = []
round_to_values = {}
for param_name in param_list:
param_config = api_config[param_name]
param_type = param_config["type"]
param_space = param_config.get("space", None)
param_range = param_config.get("range", None)
param_values = param_config.get("values", None)
# Some setup for case that whitelist of values is provided:
values_only_type = param_type in ("cat", "ordinal")
if (param_values is not None) and (not values_only_type):
assert param_range is None
param_values = np.unique(param_values)
param_range = (param_values[0], param_values[-1])
round_to_values[param_name] = interp1d(
param_values, param_values, kind="nearest", fill_value="extrapolate"
)
if param_type == "int":
# Integer space in sklearn does not support any warping => Need
# to leave the warping as linear in skopt.
sk_dims.append(Integer(param_range[0], param_range[-1], transform=transform, name=param_name))
elif param_type == "bool":
assert param_range is None
assert param_values is None
sk_dims.append(Integer(0, 1, transform=transform, name=param_name))
elif param_type in ("cat", "ordinal"):
assert param_range is None
# Leave x-form to one-hot as per skopt default
sk_dims.append(Categorical(param_values, name=param_name))
elif param_type == "real":
# Skopt doesn't support all our warpings, so need to pick
# closest substitute it does support.
prior = "log-uniform" if param_space in ("log", "logit") else "uniform"
sk_dims.append(Real(param_range[0], param_range[-1], prior=prior, transform=transform, name=param_name))
else:
assert False, "type %s not handled in API" % param_type
return sk_dims, round_to_values
def suggest(self, n_suggestions=1):
"""Get a suggestion from the optimizer.
Parameters
----------
n_suggestions : int
Desired number of parallel suggestions in the output
Returns
-------
next_guess : list of dict
List of `n_suggestions` suggestions to evaluate the objective
function. Each suggestion is a dictionary where each key
corresponds to a parameter being optimized.
"""
# First get list of lists from skopt.ask()
next_guess = self.skopt.ask(n_points=n_suggestions)
# Then convert to list of dicts
next_guess = [dict(zip(self.dimensions_list, x)) for x in next_guess]
# Now do the rounding, custom rounding is not supported in skopt. Note
# that there is not nec a round function for each dimension here.
for param_name, round_f in self.round_to_values.items():
for xx in next_guess:
xx[param_name] = round_f(xx[param_name])
return next_guess
def observe(self, X, y):
"""Send an observation of a suggestion back to the optimizer.
Parameters
----------
X : list of dict-like
Places where the objective function has already been evaluated.
Each suggestion is a dictionary where each key corresponds to a
parameter being optimized.
y : array-like, shape (n,)
Corresponding values where objective has been evaluated
"""
# Supposedly skopt can handle blocks, but not sure about interface for
# that. Just do loop to be safe for now.
for xx, yy in zip(X, y):
# skopt needs lists instead of dicts
xx = [xx[dim_name] for dim_name in self.dimensions_list]
# Just ignore, any inf observations we got, unclear if right thing
if np.isfinite(yy):
self.skopt.tell(xx, yy)
opt_wrapper = ScikitOptimizer
================================================
FILE: bayesmark/cmd_parse.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Anything related to parsing command line arguments goes in here. There are some custom data structures to
represent all the options available in the experiments here.
Not currently any doc strings in this file because it may become obsolete with the use of fire package.
"""
import argparse
import json
import os.path
import sys
import uuid as pyuuid
from enum import IntEnum, auto
from pathlib import PosixPath
import git
from git.exc import InvalidGitRepositoryError
from pathvalidate.argparse import sanitize_filename, validate_filename, validate_filepath
from bayesmark.builtin_opt.config import CONFIG
from bayesmark.constants import ARG_DELIM, DATA_LOADER_NAMES, METRICS, MODEL_NAMES, OPTIMIZERS_FILE, PY_INTERPRETER
from bayesmark.path_util import absopen, abspath
from bayesmark.util import shell_join
assert not any(ARG_DELIM in opt for opt in MODEL_NAMES)
assert not any(ARG_DELIM in opt for opt in DATA_LOADER_NAMES)
class CmdArgs(IntEnum):
uuid = auto()
db_root = auto()
optimizer_root = auto()
data_root = auto()
db = auto()
optimizer = auto()
data = auto()
classifier = auto()
metric = auto()
n_calls = auto()
n_suggest = auto()
n_repeat = auto()
n_jobs = auto()
jobs_file = auto()
ravel = auto()
verbose = auto()
dry_run = auto()
rev = auto()
opt_rev = auto()
timeout = auto()
CMD_STR = {
CmdArgs.uuid: ("-u", "--uuid"),
CmdArgs.db_root: ("-dir", "-db-root"),
CmdArgs.optimizer_root: ("-odir", "--opt-root"),
CmdArgs.data_root: ("-dr", "--data-root"),
CmdArgs.db: ("-b", "--db"),
CmdArgs.optimizer: ("-o", "--opt"),
CmdArgs.data: ("-d", "--data"),
CmdArgs.classifier: ("-c", "--classifier"),
CmdArgs.metric: ("-m", "--metric"),
CmdArgs.n_calls: ("-n", "--calls"),
CmdArgs.n_suggest: ("-p", "--suggestions"),
CmdArgs.n_repeat: ("-r", "--repeat"),
CmdArgs.n_jobs: ("-nj", "--num-jobs"),
CmdArgs.jobs_file: ("-ofile", "--jobs-file"),
CmdArgs.ravel: ("-rv", "--ravel"),
CmdArgs.verbose: ("-v", "--verbose"),
CmdArgs.timeout: ("-t", "--timeout"),
CmdArgs.dry_run: (None, "dry_run"), # Will not be specified from CLI
CmdArgs.rev: (None, "rev"), # Will not be specified from CLI
CmdArgs.opt_rev: (None, "opt_rev"), # Will not be specified from CLI. Which version of optimizer.
}
def arg_to_str(arg):
# We can change this so it is arg.value, or someway to be usable by field interface
_, dest = str(arg).split(".")
return dest
def namespace_to_dict(args_ns):
args = vars(args_ns)
args = {kk: args[arg_to_str(kk)] for kk in CMD_STR if (arg_to_str(kk) in args)}
return args
def serializable_dict(args):
args_str = {CMD_STR[kk][1]: args[kk] for kk in CMD_STR if (kk in args)}
assert len(args_str) == len(args)
return args_str
def unserializable_dict(args_str):
args = {kk: args_str[CMD_STR[kk][1]] for kk in CMD_STR if (CMD_STR[kk][1] in args_str)}
assert len(args_str) == len(args)
return args
def add_argument(parser, arg, **kwargs):
short_name, long_name = CMD_STR[arg]
dest = arg_to_str(arg)
parser.add_argument(short_name, long_name, dest=dest, **kwargs)
def filepath(value):
"""Work around for `pathvalidate` bug."""
if value == ".":
return value
validate_filepath(value, platform="auto")
return value
def filename(value):
validate_filename(value, platform="universal")
return value
def uuid(val_str):
val = str.lower(val_str)
uuid_ = pyuuid.UUID(hex=val)
assert val == uuid_.hex, "error in parsing uuid"
return val
def positive_int(val_str):
val = int(val_str)
if val <= 0:
msg = "expected positive, got %s" % val_str
raise argparse.ArgumentTypeError(msg)
return val
def joinable(val_str):
val = str(val_str) # just for good measure
validate_filename(val, platform="universal") # we choose to be at least as strict as filenames
if ARG_DELIM in val:
msg = "delimiter %s not allowed in choice %s" % (ARG_DELIM, val)
raise argparse.ArgumentTypeError(msg)
return val
def load_rev_number():
# This function uses a lot of language "power features" that could be considered bad form:
# 1) does a conditional import to get version
# 2) uses __file__ to try and extract and git repo version during execution
# We will let this fly anyway because:
# 1) The results of this are only used for logging anyway
# 2) This is a command parsing module of the code and inherently very non-pure and doing IO etc
# 3) Unclear if there is a cleaner way to do this
# Get rev from version file (if running inside the pip-installable wheel without the git repo)
try:
from bayesmark import version
rev_file = version.VERSION
except ImportError:
rev_file = None
else:
rev_file = rev_file.strip()
rev = rev_file
# Get rev from git API if inside git repo (and not built wheel from pip install ...)
wdir = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
try:
repo = git.Repo(path=wdir, search_parent_directories=False)
except InvalidGitRepositoryError:
rev_repo = None
else:
rev_repo = repo.head.commit.hexsha
rev_repo = rev_repo.strip()
rev = rev_repo
# Check coherence of what we found
if (rev_repo is None) and (rev_file is None):
raise RuntimeError("Must specify version.py if not inside a git repo.")
if (rev_repo is not None) and (rev_file is not None):
assert rev_repo == rev_file, "Rev file %s does not match rev git %s" % (rev_file, rev_repo)
assert rev == rev.strip()
# We could first enforce is_lower_hex if we want to enforce that
rev = rev[:7]
return rev
def base_parser():
parser = argparse.ArgumentParser(add_help=False)
add_argument(
parser, CmdArgs.db_root, default=".", type=filepath, help="root directory for all benchmark experiments output"
)
add_argument(
parser, CmdArgs.optimizer_root, default=".", type=filepath, help="Directory with optimization wrappers"
)
# Always a verbose flag option
add_argument(parser, CmdArgs.verbose, action="store_true", help="print the study logs to console")
return parser
def launcher_parser(description):
parser = argparse.ArgumentParser(description=description, parents=[base_parser()])
add_argument(parser, CmdArgs.uuid, type=uuid, help="length 32 hex UUID for this experiment")
add_argument(parser, CmdArgs.data_root, type=filepath, help="root directory for all custom csv files")
add_argument(parser, CmdArgs.db, type=filename, help="database ID of this benchmark experiment")
add_argument(parser, CmdArgs.optimizer, type=joinable, nargs="+", help="optimizers to use")
add_argument(parser, CmdArgs.data, type=joinable, nargs="+", help="data sets to use")
add_argument(parser, CmdArgs.classifier, type=joinable, nargs="+", help="classifiers to use")
add_argument(parser, CmdArgs.metric, type=str, choices=METRICS, nargs="+", help="scoring metric to use")
# Iterations counts used in experiments
add_argument(parser, CmdArgs.n_calls, default=100, type=positive_int, help="number of function evaluations")
add_argument(
parser, CmdArgs.n_suggest, default=1, type=positive_int, help="number of suggestions to provide in parallel"
)
add_argument(parser, CmdArgs.n_repeat, default=20, type=positive_int, help="number of repetitions of each study")
add_argument(parser, CmdArgs.timeout, default=0, type=int, help="Timeout per experiment (0 = no timeout)")
# Arguments for creating dry run jobs file
add_argument(
parser,
CmdArgs.n_jobs,
type=int,
default=0,
help="number of jobs to put in the dry run file, the default 0 value disables dry run (real run)",
)
# Using default of current dir for jobs file output since that is generally the default for everything
add_argument(
parser, CmdArgs.jobs_file, type=filepath, default="./jobs.txt", help="a jobs file with all commands to be run"
)
return parser
def experiment_parser(description):
parser = argparse.ArgumentParser(description=description, parents=[base_parser()])
add_argument(parser, CmdArgs.uuid, type=uuid, required=True, help="length 32 hex UUID for this experiment")
# This could be made simpler and use '.' default for dataroot, even if no custom data used.
add_argument(parser, CmdArgs.data_root, type=filepath, help="root directory for all custom csv files")
add_argument(parser, CmdArgs.db, type=filename, required=True, help="database ID of this benchmark experiment")
add_argument(parser, CmdArgs.optimizer, required=True, type=joinable, help="optimizer to use")
add_argument(parser, CmdArgs.data, required=True, type=joinable, help="data set to use")
add_argument(parser, CmdArgs.classifier, required=True, type=joinable, help="classifier to use")
add_argument(parser, CmdArgs.metric, required=True, type=str, choices=METRICS, help="scoring metric to use")
add_argument(parser, CmdArgs.n_calls, default=100, type=positive_int, help="number of function evaluations")
add_argument(
parser, CmdArgs.n_suggest, default=1, type=positive_int, help="number of suggestions to provide in parallel"
)
return parser
def agg_parser(description):
parser = argparse.ArgumentParser(description=description, parents=[base_parser()])
add_argument(parser, CmdArgs.db, type=filename, required=True, help="database ID of this benchmark experiment")
add_argument(
parser,
CmdArgs.ravel,
action="store_true",
help="ravel all studies to store batch suggestions as if they were serial (deprecated)",
)
return parser
def general_parser(description):
parser = argparse.ArgumentParser(description=description, parents=[base_parser()])
add_argument(parser, CmdArgs.db, type=filename, required=True, help="database ID of this benchmark experiment")
return parser
def parse_args(parser, argv=None):
"""Note that this argument parser does not check compatibility between clf/reg metric and data set.
"""
args = parser.parse_args(argv)
args = namespace_to_dict(args)
args[CmdArgs.dry_run] = (CmdArgs.n_jobs in args) and (args[CmdArgs.n_jobs] > 0)
# Does not check dir actually exists here, but whatever
args[CmdArgs.jobs_file] = abspath(args[CmdArgs.jobs_file], verify=False) if args[CmdArgs.dry_run] else None
# Then make sure all path vars are abspath:
# Dry run might be executing on diff system => cannot verify yet
args[CmdArgs.db_root] = abspath(args[CmdArgs.db_root], verify=not args[CmdArgs.dry_run])
args[CmdArgs.optimizer_root] = abspath(args[CmdArgs.optimizer_root], verify=True)
if (CmdArgs.data_root in args) and (args[CmdArgs.data_root] is not None):
args[CmdArgs.data_root] = abspath(args[CmdArgs.data_root], verify=not args[CmdArgs.dry_run])
# Get git version of the benchmark itself for meta-data, just in case we need it.
args[CmdArgs.rev] = load_rev_number()
# We may support ability to specify version at args in the future, from now it is implied
args[CmdArgs.opt_rev] = None
return args
def _cleanup(filename_str):
filename_str = sanitize_filename(filename_str, replacement_text="-", platform="universal")
filename_str = filename_str.replace(ARG_DELIM, "-")
return filename_str
def infer_settings(opt_root, opt_pattern="**/optimizer.py"):
opt_root = PosixPath(opt_root)
assert opt_root.is_dir(), "Opt root directory doesn't exist: %s" % opt_root
assert opt_root.is_absolute(), "Only absolute path should have even gotten this far."
# Always sort for reproducibility
source_files = sorted(opt_root.glob(opt_pattern))
source_files = [ss.relative_to(opt_root) for ss in source_files]
settings = {_cleanup(str(ss.parent)): [str(ss), {}] for ss in source_files}
assert all(joinable(kk) for kk in settings), "Something went wrong in name sanitization."
assert len(settings) == len(source_files), "Name collision after sanitization of %s" % repr(source_files)
assert len(set(CONFIG.keys()) & set(settings.keys())) == 0, "Name collision with builtin optimizers."
return settings
def load_optimizer_settings(opt_root):
try:
with absopen(os.path.join(opt_root, OPTIMIZERS_FILE), "r") as f:
settings = json.load(f)
except FileNotFoundError:
# Search for optimizers instead
settings = infer_settings(opt_root)
assert isinstance(settings, dict)
assert not any((ARG_DELIM in opt) for opt in settings), "optimizer names violates name convention"
return settings
def cmd_str():
cmd = "%s %s" % (PY_INTERPRETER, shell_join(sys.argv))
return cmd
================================================
FILE: bayesmark/constants.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""General constants that are used in multiple files in the code base.
"""
# Special constant for random search since it gets used as our reference point in the baselines
RANDOM_SEARCH = "RandomSearch"
OPTIMIZERS_FILE = "config.json"
ARG_DELIM = "_" # Delimeter used when concat cmd argument for any reason
PY_INTERPRETER = "python" # What command to call for sub process, we could specify version number here also.
# Variables to save in SAL
EVAL = "eval"
TIME = "time"
SUGGEST_LOG = "suggest_log"
EXP_VARS = (EVAL, TIME, SUGGEST_LOG)
# Derived variables to save in SAL
TIME_RESULTS = "time"
EVAL_RESULTS = "eval"
BASELINE = "baseline"
PERF_RESULTS = "perf"
MEAN_SCORE = "summary"
# Coordinate dim names needed in saved xr Datasets
ITER = "iter"
TEST_CASE = "function"
METHOD = "optimizer"
TRIAL = "study_id"
SUGGEST = "suggestion"
OBJECTIVE = "objective"
# Dataset variables for eval results
VISIBLE_TO_OPT = "_visible_to_opt"
# Dataset variables for time results
SUGGEST_PHASE = "suggest"
OBS_PHASE = "observe"
EVAL_PHASE = "eval"
EVAL_PHASE_SUM = "eval_sum"
EVAL_PHASE_MAX = "eval_max"
# Dataset variables for aggregate results
PERF_MED = "median"
LB_MED = "median LB"
UB_MED = "median UB"
NORMED_MED = "median normed"
PERF_MEAN = "mean"
LB_MEAN = "mean LB"
UB_MEAN = "mean UB"
NORMED_MEAN = "mean normed"
LB_NORMED_MEAN = "mean normed LB"
UB_NORMED_MEAN = "mean normed UB"
PERF_BEST = "best"
PERF_CLIP = "clip"
# Choices used for test problems, there is some redundant specification with sklearn funcs file here
MODEL_NAMES = ("DT", "MLP-adam", "MLP-sgd", "RF", "SVM", "ada", "kNN", "lasso", "linear")
DATA_LOADER_NAMES = ("breast", "digits", "iris", "wine", "boston", "diabetes")
SCORERS_CLF = ("nll", "acc")
SCORERS_REG = ("mae", "mse")
METRICS = tuple(sorted(SCORERS_CLF + SCORERS_REG))
================================================
FILE: bayesmark/data.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Module to deal with all matters relating to loading example data sets, which we tune ML models to.
"""
from enum import IntEnum, auto
import numpy as np
import pandas as pd # only needed for csv reader, maybe try something else
from sklearn import datasets
from bayesmark.constants import DATA_LOADER_NAMES, SCORERS_CLF, SCORERS_REG
from bayesmark.path_util import join_safe_r
from bayesmark.stats import robust_standardize
class ProblemType(IntEnum):
"""The different problem types we consider. Currently, just regression (`reg`) and classification (`clf`).
"""
clf = auto()
reg = auto()
DATA_LOADERS = {
"digits": (datasets.load_digits, ProblemType.clf),
"iris": (datasets.load_iris, ProblemType.clf),
"wine": (datasets.load_wine, ProblemType.clf),
"breast": (datasets.load_breast_cancer, ProblemType.clf),
"boston": (datasets.load_boston, ProblemType.reg),
"diabetes": (datasets.load_diabetes, ProblemType.reg),
}
assert sorted(DATA_LOADERS.keys()) == sorted(DATA_LOADER_NAMES)
# Arguably, this could go in constants, but doesn't cause extra imports being here.
METRICS_LOOKUP = {ProblemType.clf: SCORERS_CLF, ProblemType.reg: SCORERS_REG}
def get_problem_type(dataset_name):
"""Determine if this dataset is a regression of classification problem.
Parameters
----------
dataset : str
Which data set to use, must be key in `DATA_LOADERS` dict, or name of custom csv file.
Returns
-------
problem_type : ProblemType
`Enum` to indicate if regression of classification data set.
"""
if dataset_name in DATA_LOADERS:
_, problem_type = DATA_LOADERS[dataset_name]
return problem_type
# Maybe we can come up with a better system, but for now let's use a convention based on the naming of the csv file.
if dataset_name.startswith("reg-"):
return ProblemType.reg
if dataset_name.startswith("clf-"):
return ProblemType.clf
assert False, "Can't determine problem type from dataset name."
def _csv_loader(dataset_name, return_X_y, data_root, clip_x=100): # pragma: io
"""Load custom csv files for use in the benchmark.
This function assumes ``dataset_name + ".csv"`` is a csv file found in the `data_root` path. It also assumes the
last column of the csv file is the target and the other columns are features.
The target column should be `int` for classification and `float` for regression. Column names ending in ``"_cat"``
are assumed to be categorical and will be one-hot encoded.
The features (and target for regression) are robust standardized. The features are also clipped to be in
``[-clip_x, clip_x]`` *after* standardization.
"""
assert return_X_y, "Only returning (X,y) tuple supported right now."
assert clip_x >= 0
# Quantile range for robust standardization. The 86% range is the most efficient for Gaussians. See:
# https://github.com/scikit-learn/scikit-learn/issues/10139#issuecomment-344705040
q_level = 0.86
path = join_safe_r(data_root, dataset_name + ".csv")
# For now, use convention that can get problem type based on data set name
problem_type = get_problem_type(dataset_name)
# Assuming no missing data in source csv files at the moment, these will
# result in error.
df = pd.read_csv(
path, header=0, index_col=False, engine="c", na_filter=False, true_values=["true"], false_values=["false"]
)
label = df.columns[-1] # Assume last col is target
target = df.pop(label).values
if problem_type == ProblemType.clf:
assert target.dtype in (np.bool_, np.int_)
target = target.astype(np.int_) # convert to int for skl
if problem_type == ProblemType.reg:
assert target.dtype == np.float_
# 86% range is the most efficient (at least for Gaussians)
target = robust_standardize(target, q_level=q_level)
# Fill in an categorical variables (object dtype of cols names ..._cat)
cat_cols = sorted(cc for cc in df.columns if cc.endswith("_cat") or df[cc].dtype.kind == "O")
df = pd.get_dummies(df, columns=cat_cols, drop_first=True, dtype=np.float_)
# Could also sort all columns to be sure it will be reprod
# Everything should now be in float
assert (df.dtypes == np.float_).all()
data = df.values
data = robust_standardize(data, q_level=q_level)
# Debatable if we should include this, but there are a lot of outliers
data = np.clip(data, -clip_x, clip_x)
# We should probably do some logging or something to wrap up
return data, target, problem_type
def load_data(dataset_name, data_root=None): # pragma: io
"""Load a data set and return it in, pre-processed into numpy arrays.
Parameters
----------
dataset : str
Which data set to use, must be key in `DATA_LOADERS` dict, or name of custom csv file.
data_root : str
Root directory to look for all custom csv files. May be ``None`` for sklearn data sets.
Returns
-------
data : :class:`numpy:numpy.ndarray` of shape (n, d)
The feature matrix of the data set. It will be `float` array.
target : :class:`numpy:numpy.ndarray` of shape (n,)
The target vector for the problem, which is `int` for classification and `float` for regression.
problem_type : :class:`bayesmark.data.ProblemType`
`Enum` to indicate if regression of classification data set.
"""
if dataset_name in DATA_LOADERS:
loader_f, problem_type = DATA_LOADERS[dataset_name]
data, target = loader_f(return_X_y=True)
else: # try to load as custom csv
assert data_root is not None, "data root cannot be None when custom csv requested."
data, target, problem_type = _csv_loader(dataset_name, return_X_y=True, data_root=data_root)
return data, target, problem_type
================================================
FILE: bayesmark/expected_max.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Compute expected maximum or minimum from iid samples.
"""
import numpy as np
from scipy.special import gammaln, logsumexp
def get_expected_max_weights(n, m):
"""Get the L-estimator weights for computing unbiased estimator of expected ``max(x[1:m])`` on a data set.
Parameters
----------
n : int
Number of data points in data set ``len(x)``. Must be ``>= 1``.
m : `int` or :class:`numpy:numpy.ndarray` with dtype `int`
This function is for estimating the expected maximum over `m` iid draws. Require ``m >= 1``. This can be
broadcasted. If ``m > n``, the weights will be nan, because there is no way to get unbiased estimate in that
case.
Returns
-------
pdf : :class:`numpy:numpy.ndarray`, shape (n,)
The weights for L-estimator. Will be positive and sum to one.
"""
assert np.ndim(n) == 0
assert n >= 1 # otherwise makes no sense
m = np.asarray(m) # Must be np type for broadcasting
# We could also check dtype is int, but not bothering here
assert np.all(m >= 1) # otherwise makes no sense
m = m[..., None]
kk = 1 + np.arange(n)
lpdf = gammaln(kk) - gammaln(kk - (m - 1))
pdf = np.exp(lpdf - logsumexp(lpdf, axis=-1, keepdims=True))
# expect nan for m > n
assert np.all((m > n) | np.isclose(np.sum(pdf, axis=-1, keepdims=True), 1.0))
return pdf
def expected_max(x, m):
"""Compute unbiased estimator of expected ``max(x[1:m])`` on a data set.
Parameters
----------
x : :class:`numpy:numpy.ndarray` of shape (n,)
Data set we would like expected ``max(x[1:m])`` on.
m : `int` or :class:`numpy:numpy.ndarray` with dtype `int`
This function is for estimating the expected maximum over `m` iid draws. Require ``m >= 1``. This can be
broadcasted. If ``m > n``, the weights will be nan, because there is no way to get unbiased estimate in that
case.
Returns
-------
E_max_x : float
Unbiased estimate of mean max of `m` draws from distribution on `x`.
"""
assert np.ndim(x) == 1
# m is validated by get_expected_max_weights
# Get order stats for L-estimator
x = np.array(x, copy=True) # we will modify in place
x.sort() # in place!!
# Now get estimator weights
n, = x.shape
if n == 0:
return np.full(np.shape(m), np.nan)
pdf = get_expected_max_weights(n, m)
# Compute L-estimator
E_max_x = np.sum(x * pdf, axis=-1)
return E_max_x
def expected_min(x, m):
"""Compute unbiased estimator of expected ``min(x[1:m])`` on a data set.
Parameters
----------
x : :class:`numpy:numpy.ndarray` of shape (n,)
Data set we would like expected ``min(x[1:m])`` on. Require ``len(x) >= 1``.
m : `int` or :class:`numpy:numpy.ndarray` with dtype `int`
This function is for estimating the expected minimum over `m` iid draws. Require ``m >= 1``. This can be
broadcasted. If ``m > n``, the weights will be nan, because there is no way to get unbiased estimate in that
case.
Returns
-------
E_min_x : float
Unbiased estimate of mean min of `m` draws from distribution on `x`.
"""
x = np.asarray(x)
E_min_x = -expected_max(-x, m)
return E_min_x
================================================
FILE: bayesmark/experiment.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Perform a study.
"""
import json
import logging
import random as pyrandom
import uuid
import warnings
from collections import OrderedDict
from time import sleep, time
import numpy as np
import xarray as xr
import bayesmark.cmd_parse as cmd
import bayesmark.constants as cc
import bayesmark.random_search as rs
from bayesmark.builtin_opt.config import CONFIG
from bayesmark.cmd_parse import CmdArgs
from bayesmark.constants import ARG_DELIM, ITER, OBJECTIVE, SUGGEST
from bayesmark.data import METRICS_LOOKUP, get_problem_type
from bayesmark.np_util import argmin_2d, linear_rescale, random_seed
from bayesmark.serialize import XRSerializer
from bayesmark.signatures import analyze_signature_pair, get_func_signature
from bayesmark.sklearn_funcs import SklearnModel, SklearnSurrogate
from bayesmark.space import JointSpace
from bayesmark.util import chomp, str_join_safe
logger = logging.getLogger(__name__)
# For now treat the objective names as global const. However, in the future these could vary by type of problem.
OBJECTIVE_NAMES = SklearnModel.objective_names
def _build_test_problem(model_name, dataset, scorer, path):
"""Build the class with the class to use an objective. Sort of a factory.
Parameters
----------
model_name : str
Which sklearn model we are attempting to tune, must be an element of `constants.MODEL_NAMES`.
dataset : str
Which data set the model is being tuned to, which must be either a) an element of
`constants.DATA_LOADER_NAMES`, or b) the name of a csv file in the `data_root` folder for a custom data set.
scorer : str
Which metric to use when evaluating the model. This must be an element of `sklearn_funcs.SCORERS_CLF` for
classification models, or `sklearn_funcs.SCORERS_REG` for regression models.
path : str or None
Absolute path to folder containing custom data sets/pickle files with surrogate model.
Returns
-------
prob : :class:`.sklearn_funcs.TestFunction`
The test function to evaluate in experiments.
"""
if model_name.endswith("-surr"):
# Requires IO to test these, so will add the pargma here. Maybe that points towards a possible design change.
model_name = chomp(model_name, "-surr") # pragma: io
prob = SklearnSurrogate(model_name, dataset, scorer, path=path) # pragma: io
else:
prob = SklearnModel(model_name, dataset, scorer, data_root=path)
return prob
def run_study(optimizer, test_problem, n_calls, n_suggestions, n_obj=1, callback=None):
"""Run a study for a single optimizer on a single test problem.
This function can be used for benchmarking on general stateless objectives (not just `sklearn`).
Parameters
----------
optimizer : :class:`.abstract_optimizer.AbstractOptimizer`
Instance of one of the wrapper optimizers.
test_problem : :class:`.sklearn_funcs.TestFunction`
Instance of test function to attempt to minimize.
n_calls : int
How many iterations of minimization to run.
n_suggestions : int
How many parallel evaluation we run each iteration. Must be ``>= 1``.
n_obj : int
Number of different objectives measured, only objective 0 is seen by optimizer. Must be ``>= 1``.
callback : callable
Optional callback taking the current best function evaluation, and the number of iterations finished. Takes
array of shape `(n_obj,)`.
Returns
-------
function_evals : :class:`numpy:numpy.ndarray` of shape (n_calls, n_suggestions, n_obj)
Value of objective for each evaluation.
timing_evals : (:class:`numpy:numpy.ndarray`, :class:`numpy:numpy.ndarray`, :class:`numpy:numpy.ndarray`)
Tuple of 3 timing results: ``(suggest_time, eval_time, observe_time)`` with shapes ``(n_calls,)``,
``(n_calls, n_suggestions)``, and ``(n_calls,)``. These are the time to make each suggestion, the time for each
evaluation of the objective function, and the time to make an observe call.
suggest_log : list(list(dict(str, object)))
Log of the suggestions corresponding to the `function_evals`.
"""
assert n_suggestions >= 1, "batch size must be at least 1"
assert n_obj >= 1, "Must be at least one objective"
space_for_validate = JointSpace(test_problem.get_api_config())
if callback is not None:
# First do initial log at inf score, in case we don't even get to first eval before crash/job timeout
callback(np.full((n_obj,), np.inf, dtype=float), 0)
suggest_time = np.zeros(n_calls)
observe_time = np.zeros(n_calls)
eval_time = np.zeros((n_calls, n_suggestions))
function_evals = np.zeros((n_calls, n_suggestions, n_obj))
suggest_log = [None] * n_calls
for ii in range(n_calls):
tt = time()
try:
next_points = optimizer.suggest(n_suggestions)
except Exception as e:
logger.warning("Failure in optimizer suggest. Falling back to random search.")
logger.exception(e, exc_info=True)
print(json.dumps({"optimizer_suggest_exception": {ITER: ii}}))
api_config = test_problem.get_api_config()
next_points = rs.suggest_dict([], [], api_config, n_suggestions=n_suggestions)
suggest_time[ii] = time() - tt
logger.info("suggestion time taken %f iter %d next_points %s" % (suggest_time[ii], ii, str(next_points)))
assert len(next_points) == n_suggestions, "invalid number of suggestions provided by the optimizer"
# We could put this inside the TestProblem class, but ok here for now.
try:
space_for_validate.validate(next_points) # Fails if suggestions outside allowed range
except Exception:
raise ValueError("Optimizer suggestion is out of range.")
for jj, next_point in enumerate(next_points):
tt = time()
try:
f_current_eval = test_problem.evaluate(next_point)
except Exception as e:
logger.warning("Failure in function eval. Setting to inf.")
logger.exception(e, exc_info=True)
f_current_eval = np.full((n_obj,), np.inf, dtype=float)
eval_time[ii, jj] = time() - tt
assert np.shape(f_current_eval) == (n_obj,)
suggest_log[ii] = next_points
function_evals[ii, jj, :] = f_current_eval
logger.info(
"function_evaluation time %f value %f suggestion %s"
% (eval_time[ii, jj], f_current_eval[0], str(next_point))
)
# Note: this could be inf in the event of a crash in f evaluation, the optimizer must be able to handle that.
# Only objective 0 is seen by optimizer.
eval_list = function_evals[ii, :, 0].tolist()
if callback is not None:
idx_ii, idx_jj = argmin_2d(function_evals[: ii + 1, :, 0])
callback(function_evals[idx_ii, idx_jj, :], ii + 1)
tt = time()
try:
optimizer.observe(next_points, eval_list)
except Exception as e:
logger.warning("Failure in optimizer observe. Ignoring these observations.")
logger.exception(e, exc_info=True)
print(json.dumps({"optimizer_observe_exception": {ITER: ii}}))
observe_time[ii] = time() - tt
logger.info(
"observation time %f, current best %f at iter %d"
% (observe_time[ii], np.min(function_evals[: ii + 1, :, 0]), ii)
)
return function_evals, (suggest_time, eval_time, observe_time), suggest_log
def run_sklearn_study(
opt_class, opt_kwargs, model_name, dataset, scorer, n_calls, n_suggestions, data_root=None, callback=None
):
"""Run a study for a single optimizer on a single `sklearn` model/data set combination.
This routine is meant for benchmarking when tuning `sklearn` models, as opposed to the more general
:func:`.run_study`.
Parameters
----------
opt_class : :class:`.abstract_optimizer.AbstractOptimizer`
Type of wrapper optimizer must be subclass of :class:`.abstract_optimizer.AbstractOptimizer`.
opt_kwargs : kwargs
`kwargs` to use when instantiating the wrapper class.
model_name : str
Which sklearn model we are attempting to tune, must be an element of `constants.MODEL_NAMES`.
dataset : str
Which data set the model is being tuned to, which must be either a) an element of
`constants.DATA_LOADER_NAMES`, or b) the name of a csv file in the `data_root` folder for a custom data set.
scorer : str
Which metric to use when evaluating the model. This must be an element of `sklearn_funcs.SCORERS_CLF` for
classification models, or `sklearn_funcs.SCORERS_REG` for regression models.
n_calls : int
How many iterations of minimization to run.
n_suggestions : int
How many parallel evaluation we run each iteration. Must be ``>= 1``.
data_root : str
Absolute path to folder containing custom data sets. This may be ``None`` if no custom data sets are used.``
callback : callable
Optional callback taking the current best function evaluation, and the number of iterations finished. Takes
array of shape `(n_obj,)`.
Returns
-------
function_evals : :class:`numpy:numpy.ndarray` of shape (n_calls, n_suggestions, n_obj)
Value of objective for each evaluation.
timing_evals : (:class:`numpy:numpy.ndarray`, :class:`numpy:numpy.ndarray`, :class:`numpy:numpy.ndarray`)
Tuple of 3 timing results: ``(suggest_time, eval_time, observe_time)`` with shapes ``(n_calls,)``,
``(n_calls, n_suggestions)``, and ``(n_calls,)``. These are the time to make each suggestion, the time for each
evaluation of the objective function, and the time to make an observe call.
suggest_log : list(list(dict(str, object)))
Log of the suggestions corresponding to the `function_evals`.
"""
# Setup test function
function_instance = _build_test_problem(model_name, dataset, scorer, data_root)
# Setup optimizer
api_config = function_instance.get_api_config()
optimizer_instance = opt_class(api_config, **opt_kwargs)
assert function_instance.objective_names == OBJECTIVE_NAMES
assert OBJECTIVE_NAMES[0] == cc.VISIBLE_TO_OPT
n_obj = len(OBJECTIVE_NAMES)
# Now actually do the experiment
function_evals, timing, suggest_log = run_study(
optimizer_instance, function_instance, n_calls, n_suggestions, n_obj=n_obj, callback=callback
)
return function_evals, timing, suggest_log
def get_objective_signature(model_name, dataset, scorer, data_root=None):
"""Get signature of an objective function specified by an sklearn model and dataset.
This routine specializes :func:`.signatures.get_func_signature` for the `sklearn` study case.
Parameters
----------
model_name : str
Which sklearn model we are attempting to tune, must be an element of `constants.MODEL_NAMES`.
dataset : str
Which data set the model is being tuned to, which must be either a) an element of
`constants.DATA_LOADER_NAMES`, or b) the name of a csv file in the `data_root` folder for a custom data set.
scorer : str
Which metric to use when evaluating the model. This must be an element of `sklearn_funcs.SCORERS_CLF` for
classification models, or `sklearn_funcs.SCORERS_REG` for regression models.
data_root : str
Absolute path to folder containing custom data sets. This may be ``None`` if no custom data sets are used.``
Returns
-------
signature : list(str)
The signature of this test function.
"""
function_instance = _build_test_problem(model_name, dataset, scorer, data_root)
api_config = function_instance.get_api_config()
signature = get_func_signature(function_instance.evaluate, api_config)
return signature
def build_eval_ds(function_evals, objective_names):
"""Convert :class:`numpy:numpy.ndarray` with function evaluations to :class:`xarray:xarray.Dataset`.
This function is a data cleanup routine after running an experiment, before serializing the data to end the study.
Parameters
----------
function_evals : :class:`numpy:numpy.ndarray` of shape (n_calls, n_suggestions, n_obj)
Value of objective for each evaluation.
objective_names : list(str) of shape (n_obj,)
The names of each objective.
Returns
-------
eval_ds : :class:`xarray:xarray.Dataset`
:class:`xarray:xarray.Dataset` containing one variable for each objective with the objective function
evaluations. It has dimensions ``(ITER, SUGGEST)``.
"""
n_call, n_suggest, n_obj = np.shape(function_evals)
assert len(objective_names) == n_obj
assert len(set(objective_names)) == n_obj, "Objective names must be unique"
coords = {ITER: range(n_call), SUGGEST: range(n_suggest), OBJECTIVE: list(objective_names)}
dims = (ITER, SUGGEST, OBJECTIVE)
da = xr.DataArray(data=function_evals, coords=coords, dims=dims)
eval_ds = da.to_dataset(dim=OBJECTIVE)
return eval_ds
def build_timing_ds(suggest_time, eval_time, observe_time):
"""Convert :class:`numpy:numpy.ndarray` with timing evaluations to :class:`xarray:xarray.Dataset`.
This function is a data cleanup routine after running an experiment, before serializing the data to end the study.
Parameters
----------
suggest_time : :class:`numpy:numpy.ndarray` of shape (n_calls,)
The time to make each (batch) suggestion.
eval_time : :class:`numpy:numpy.ndarray` of shape (n_calls, n_suggestions)
The time for each evaluation of the objective function.
observe_time : :class:`numpy:numpy.ndarray` of shape (n_calls,)
The time for each (batch) evaluation of the objective function, and the time to make an observe call.
Returns
-------
time_ds : :class:`xarray:xarray.Dataset`
Dataset with variables ``(SUGGEST_PHASE, EVAL_PHASE, OBS_PHASE)`` which have dimensions ``(ITER,)``,
``(ITER, SUGGEST)``, and ``(ITER,)``, respectively. The variable `EVAL_PHASE` has the function evaluation time
for each parallel suggestion.
"""
n_call, n_suggest = np.shape(eval_time)
assert np.shape(suggest_time) == (n_call,)
assert np.shape(observe_time) == (n_call,)
coords = OrderedDict([(ITER, range(n_call)), (SUGGEST, range(n_suggest))])
data = OrderedDict()
data[cc.SUGGEST_PHASE] = ((ITER,), suggest_time)
data[cc.EVAL_PHASE] = ((ITER, SUGGEST), eval_time)
data[cc.OBS_PHASE] = ((ITER,), observe_time)
time_ds = xr.Dataset(data, coords=coords)
return time_ds
def build_suggest_ds(suggest_log):
"""Convert :class:`numpy:numpy.ndarray` with function evaluation inputs to :class:`xarray:xarray.Dataset`.
This function is a data cleanup routine after running an experiment, before serializing the data to end the study.
Parameters
----------
suggest_log : list(list(dict(str, object)))
Log of the suggestions. It has shape `(n_call, n_suggest)`.
Returns
-------
suggest_ds : :class:`xarray:xarray.Dataset`
:class:`xarray:xarray.Dataset` containing one variable for each input with the objective function evaluations.
It has dimensions ``(ITER, SUGGEST)``.
"""
n_call, n_suggest = np.shape(suggest_log)
assert n_call * n_suggest > 0
# Setup the dims
ds_vars = sorted(suggest_log[0][0].keys())
coords = OrderedDict([(ITER, range(n_call)), (SUGGEST, range(n_suggest))])
# There is prob a way to vectorize this more but good enough for now. Using np.full to infer dtype from 1st element
data = OrderedDict([(kk, ((ITER, SUGGEST), np.full((n_call, n_suggest), suggest_log[0][0][kk]))) for kk in ds_vars])
for ii in range(n_call):
for jj in range(n_suggest):
for kk in ds_vars:
data[kk][1][ii, jj] = suggest_log[ii][jj][kk]
suggest_ds = xr.Dataset(data, coords=coords)
return suggest_ds
def load_optimizer_kwargs(optimizer_name, opt_root): # pragma: io
"""Load the kwarg options for this optimizer being tested.
This is part of the general experiment setup before a study.
Parameters
----------
optimizer_name : str
Name of the optimizer being tested. This optimizer name must be present in optimizer config file.
opt_root : str
Absolute path to folder containing the config file.
Returns
-------
kwargs : dict(str, object)
The kwargs setting to pass into the optimizer wrapper constructor.
"""
if optimizer_name in CONFIG:
_, kwargs = CONFIG[optimizer_name]
else:
settings = cmd.load_optimizer_settings(opt_root)
assert optimizer_name in settings, "optimizer %s not found in settings file %s" % optimizer_name
_, kwargs = settings[optimizer_name]
return kwargs
def _setup_seeds(hex_str): # pragma: main
"""This function should only be called from main. Be careful with this function as it manipulates the global random
streams.
This is part of the general experiment setup before a study.
If torch becomes used in any of our optimizers then this will need to come back, could also do TF seed init.
```
torch.manual_seed(random_seed(master_stream))
if torch.cuda.is_available():
torch.cuda.manual_seed(random_seed(master_stream))
```
"""
# Set all random seeds: avoid correlated streams ==> must use diff seeds.
# Could use UUID class, but more direct to just convert the hex to py int.
# pyrandom is better for master because it is not limited to 32-bit seeds.
master_stream = pyrandom.Random(int(hex_str, 16))
pyrandom.seed(random_seed(master_stream))
np.random.seed(random_seed(master_stream))
def experiment_main(opt_class, args=None): # pragma: main
"""This is in effect the `main` routine for this experiment. However, it is called from the optimizer wrapper file
so the class can be passed in. The optimizers are assumed to be outside the package, so the optimizer class can't
be named from inside the main function without using hacky stuff like `eval`.
"""
if args is None:
description = "Run a study with one benchmark function and an optimizer"
args = cmd.parse_args(cmd.experiment_parser(description))
args[CmdArgs.opt_rev] = opt_class.get_version()
run_uuid = uuid.UUID(args[CmdArgs.uuid])
logging.captureWarnings(True)
# Setup logging to both a file and stdout (if verbose is set to True)
logger.setLevel(logging.INFO) # Note this is the module-wide logger
logfile = XRSerializer.logging_path(args[CmdArgs.db_root], args[CmdArgs.db], run_uuid)
logger_file_handler = logging.FileHandler(logfile, mode="w")
logger.addHandler(logger_file_handler)
if args[CmdArgs.verbose]:
logger.addHandler(logging.StreamHandler())
warnings_logger = logging.getLogger("py.warnings")
warnings_logger.addHandler(logger_file_handler)
if args[CmdArgs.verbose]:
warnings_logger.addHandler(logging.StreamHandler())
logger.info("running: %s" % str(cmd.serializable_dict(args)))
logger.info("cmd: %s" % cmd.cmd_str())
assert (
args[CmdArgs.metric] in METRICS_LOOKUP[get_problem_type(args[CmdArgs.data])]
), "reg/clf metrics can only be used on compatible dataset"
# Setup random streams for computing the signature, must use same seed
# across all runs to ensure signature is consistent. This seed is random:
_setup_seeds("7e9f2cabb0dd4f44bc10cf18e440b427") # pragma: allowlist secret
signature = get_objective_signature(
args[CmdArgs.classifier], args[CmdArgs.data], args[CmdArgs.metric], data_root=args[CmdArgs.data_root]
)
logger.info("computed signature: %s" % str(signature))
opt_kwargs = load_optimizer_kwargs(args[CmdArgs.optimizer], args[CmdArgs.optimizer_root])
# Setup the call back for intermediate logging
if cc.BASELINE not in XRSerializer.get_derived_keys(args[CmdArgs.db_root], db=args[CmdArgs.db]):
warnings.warn("Baselines not found. Will not log intermediate scores.")
callback = None
else:
test_case_str = SklearnModel.test_case_str(args[CmdArgs.classifier], args[CmdArgs.data], args[CmdArgs.metric])
optimizer_str = str_join_safe(ARG_DELIM, (args[CmdArgs.optimizer], args[CmdArgs.opt_rev], args[CmdArgs.rev]))
baseline_ds, baselines_meta = XRSerializer.load_derived(
args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.BASELINE
)
# Check the objective function signatures match in the baseline file
sig_errs, _ = analyze_signature_pair({test_case_str: signature[1]}, baselines_meta["signature"])
logger.info("Signature errors:\n%s" % sig_errs.to_string())
print(json.dumps({"exp sig errors": sig_errs.T.to_dict()}))
def log_mean_score_json(evals, iters):
assert evals.shape == (len(OBJECTIVE_NAMES),)
assert not np.any(np.isnan(evals))
log_msg = {
cc.TEST_CASE: test_case_str,
cc.METHOD: optimizer_str,
cc.TRIAL: args[CmdArgs.uuid],
cc.ITER: iters,
}
for idx, obj in enumerate(OBJECTIVE_NAMES):
assert OBJECTIVE_NAMES[idx] == obj
# Extract relevant rescaling info
slice_ = {cc.TEST_CASE: test_case_str, OBJECTIVE: obj}
best_opt = baseline_ds[cc.PERF_BEST].sel(slice_, drop=True).values.item()
base_clip_val = baseline_ds[cc.PERF_CLIP].sel(slice_, drop=True).values.item()
# Perform the same rescaling as found in experiment_analysis.compute_aggregates()
score = linear_rescale(evals[idx], best_opt, base_clip_val, 0.0, 1.0, enforce_bounds=False)
# Also, clip the score from below at -1 to limit max influence of single run on final average
score = np.clip(score, -1.0, 1.0)
score = score.item() # Make easiest for logging in JSON
assert isinstance(score, float)
# Note: This is not the raw score but the rescaled one!
log_msg[obj] = score
log_msg = json.dumps(log_msg)
print(log_msg, flush=True)
# One second safety delay to protect against subprocess stdout getting lost
sleep(1)
callback = log_mean_score_json
# Now set the seeds for the actual experiment
_setup_seeds(args[CmdArgs.uuid])
# Now do the experiment
logger.info(
"starting sklearn study %s %s %s %s %d %d"
% (
args[CmdArgs.optimizer],
args[CmdArgs.classifier],
args[CmdArgs.data],
args[CmdArgs.metric],
args[CmdArgs.n_calls],
args[CmdArgs.n_suggest],
)
)
logger.info("with data root: %s" % args[CmdArgs.data_root])
function_evals, timing, suggest_log = run_sklearn_study(
opt_class,
opt_kwargs,
args[CmdArgs.classifier],
args[CmdArgs.data],
args[CmdArgs.metric],
args[CmdArgs.n_calls],
args[CmdArgs.n_suggest],
data_root=args[CmdArgs.data_root],
callback=callback,
)
# Curate results into clean dataframes
eval_ds = build_eval_ds(function_evals, OBJECTIVE_NAMES)
time_ds = build_timing_ds(*timing)
suggest_ds = build_suggest_ds(suggest_log)
# setup meta:
meta = {"args": cmd.serializable_dict(args), "signature": signature}
logger.info("saving meta data: %s" % str(meta))
# Now the final IO to export the results
logger.info("saving results")
XRSerializer.save(eval_ds, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.EVAL, uuid_=run_uuid)
logger.info("saving timing")
XRSerializer.save(time_ds, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.TIME, uuid_=run_uuid)
logger.info("saving suggest log")
XRSerializer.save(suggest_ds, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.SUGGEST_LOG, uuid_=run_uuid)
logger.info("done")
def _get_opt_class(opt_name):
"""Load the relevant wrapper class based on this optimizer name.
There is inherently a bit ugly, but is only called at the main() level before the inner workings get going. There
are a few ways to do this with some pro and con:
1) The way done here: based on the filename, load that module via conditional imports and if-else. cons:
- uses conditional imports
- must manually repeat yourself in the if-else, but these are checked in unit testing
2) Import everything and then pick the right optimizer based on a dict of name_str -> class. cons:
- loads every dependency no matter which is used so could be slow
- also a stupid dependency might change global state in a way that corrupts experiments
3) Use the wrapper file as the entry point and add that to setup.py. cons:
- Will clutter the CLI namespace with one command for each wrapper
4) Use importlib to import the specified file. cons:
- Makes assumptions about relative path structure. For pip-installed packages, probably safer to let python
find the file via import.
This option (1) seems least objectionable. However, this function could easily be switched to use importlib without
any changes elsewhere.
"""
wrapper_file, _ = CONFIG[opt_name]
if wrapper_file == "hyperopt_optimizer.py":
import bayesmark.builtin_opt.hyperopt_optimizer as opt
elif wrapper_file == "nevergrad_optimizer.py":
import bayesmark.builtin_opt.nevergrad_optimizer as opt
elif wrapper_file == "opentuner_optimizer.py":
import bayesmark.builtin_opt.opentuner_optimizer as opt
elif wrapper_file == "pysot_optimizer.py":
import bayesmark.builtin_opt.pysot_optimizer as opt
elif wrapper_file == "random_optimizer.py":
import bayesmark.builtin_opt.random_optimizer as opt
elif wrapper_file == "scikit_optimizer.py":
import bayesmark.builtin_opt.scikit_optimizer as opt
else:
assert False, "CONFIG for built in optimizers has added a new optimizer, but not updated this function."
opt_class = opt.opt_wrapper
return opt_class
def main(): # pragma: main
"""This is where experiments happen. Usually called by the experiment launcher."""
description = "Run a study with one benchmark function and an optimizer"
args = cmd.parse_args(cmd.experiment_parser(description))
opt_class = _get_opt_class(args[CmdArgs.optimizer])
experiment_main(opt_class, args=args)
if __name__ == "__main__":
main() # pragma: main
================================================
FILE: bayesmark/experiment_aggregate.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Aggregate the results of many studies to prepare analysis.
"""
import json
import logging
from collections import Counter
import numpy as np
import xarray as xr
import bayesmark.constants as cc
import bayesmark.xr_util as xru
from bayesmark.cmd_parse import CmdArgs, agg_parser, parse_args, serializable_dict, unserializable_dict
from bayesmark.constants import ARG_DELIM, EVAL_RESULTS, ITER, METHOD, SUGGEST, TEST_CASE, TIME_RESULTS, TRIAL
from bayesmark.serialize import XRSerializer
from bayesmark.signatures import analyze_signatures
from bayesmark.sklearn_funcs import SklearnModel
from bayesmark.util import str_join_safe
logger = logging.getLogger(__name__)
def validate_time(all_time):
"""Validate the aggregated time data set."""
assert isinstance(all_time, xr.Dataset)
assert all_time[cc.SUGGEST_PHASE].dims == (ITER,)
assert all_time[cc.EVAL_PHASE].dims == (ITER, SUGGEST)
assert all_time[cc.OBS_PHASE].dims == (ITER,)
assert xru.is_simple_coords(all_time.coords, min_side=1)
def validate_perf(perf_da):
"""Validate the input eval data arrays."""
assert isinstance(perf_da, xr.Dataset)
assert perf_da.dims == (ITER, SUGGEST)
assert xru.is_simple_coords(perf_da.coords)
assert not np.any(np.isnan(perf_da.values))
def validate_agg_perf(perf_da, min_trial=1):
"""Validate the aggregated eval data set."""
assert isinstance(perf_da, xr.DataArray)
assert perf_da.dims == (ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)
assert xru.is_simple_coords(perf_da.coords, dims=(ITER, SUGGEST, TRIAL))
assert not np.any(np.isnan(perf_da.values))
assert perf_da.sizes[TRIAL] >= min_trial
def summarize_time(all_time):
"""Transform a single timing dataset from an experiment into a form better for aggregation.
Parameters
----------
all_time : :class:`xarray:xarray.Dataset`
Dataset with variables ``(SUGGEST_PHASE, EVAL_PHASE, OBS_PHASE)`` which have dimensions ``(ITER,)``,
``(ITER, SUGGEST)``, and ``(ITER,)``, respectively. The variable `EVAL_PHASE` has the function evaluation time
for each parallel suggestion.
Returns
-------
time_summary : :class:`xarray:xarray.Dataset`
Dataset with variables ``(SUGGEST_PHASE, OBS_PHASE, EVAL_PHASE_MAX, EVAL_PHASE_SUM)`` which all have dimensions
``(ITER,)``. The maximum `EVAL_PHASE_MAX` is relevant for wall clock time, while `EVAL_PHASE_SUM` is relevant
for CPU time.
"""
validate_time(all_time)
time_summary = xr.Dataset(coords=all_time.coords)
time_summary[cc.SUGGEST_PHASE] = all_time[cc.SUGGEST_PHASE]
time_summary[cc.OBS_PHASE] = all_time[cc.OBS_PHASE]
time_summary[cc.EVAL_PHASE_MAX] = all_time[cc.EVAL_PHASE].max(dim=SUGGEST)
time_summary[cc.EVAL_PHASE_SUM] = all_time[cc.EVAL_PHASE].sum(dim=SUGGEST)
return time_summary
def concat_experiments(all_experiments, ravel=False):
"""Aggregate the Datasets from a series of experiments into combined Dataset.
Parameters
----------
all_experiments : typing.Iterable
Iterable (possible from a generator) with the Datasets from each experiment. Each item in `all_experiments` is
a pair containing ``(meta_data, data)``. See `load_experiments` for details on these variables,
ravel : bool
If true, ravel all studies to store batch suggestions as if they were serial.
Returns
-------
all_perf : :class:`xarray:xarray.Dataset`
DataArray containing all of the `perf_da` from the experiments. The meta-data from the experiments are included
as extra dimensions. `all_perf` has dimensions ``(ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)``. To convert the
`uuid` to a trial, there must be an equal number of repetition in the experiments for each `TEST_CASE`,
`METHOD` combination. Likewise, all of the experiments need an equal number of `ITER` and `SUGGEST`. If `ravel`
is true, then the `SUGGEST` is singleton.
all_time : :class:`xarray:xarray.Dataset`
Dataset containing all of the `time_ds` from the experiments. The new dimensions are
``(ITER, TEST_CASE, METHOD, TRIAL)``. It has the same variables as `time_ds`.
all_suggest : :class:`xarray:xarray.Dataset`
DataArray containing all of the `suggest_ds` from the experiments. It has dimensions
``(ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)``.
all_sigs : dict(str, list(list(float)))
Aggregate of all experiment signatures.
"""
all_perf = {}
all_time = {}
all_suggest = {}
all_sigs = {}
trial_counter = Counter()
for (test_case, optimizer, uuid), (perf_ds, time_ds, suggest_ds, sig) in all_experiments:
if ravel:
raise NotImplementedError("ravel is deprecated. Just reshape in analysis steps instead.")
case_key = (test_case, optimizer, trial_counter[(test_case, optimizer)])
trial_counter[(test_case, optimizer)] += 1
# Process perf data
assert all(perf_ds[kk].dims == (ITER, SUGGEST) for kk in perf_ds)
all_perf[case_key] = perf_ds
# Process time data
all_time[case_key] = summarize_time(time_ds)
# Process suggestion data
all_suggest_curr = all_suggest.setdefault(test_case, {})
all_suggest_curr[case_key] = suggest_ds
# Handle the signatures
all_sigs.setdefault(test_case, []).append(sig)
assert min(trial_counter.values()) == max(trial_counter.values()), "Uneven number of trials per test case"
# Now need to concat dict of datasets into single dataset
all_perf = xru.ds_concat(all_perf, dims=(TEST_CASE, METHOD, TRIAL))
assert all(all_perf[kk].dims == (ITER, SUGGEST, TEST_CASE, METHOD, TRIAL) for kk in all_perf)
assert not any(
np.any(np.isnan(all_perf[kk].values)) for kk in all_perf
), "Missing combinations of method and test case"
all_time = xru.ds_concat(all_time, dims=(TEST_CASE, METHOD, TRIAL))
assert all(all_time[kk].dims == (ITER, TEST_CASE, METHOD, TRIAL) for kk in all_time)
assert not any(np.any(np.isnan(all_time[kk].values)) for kk in all_time)
assert xru.coord_compat((all_perf, all_time), (ITER, TEST_CASE, METHOD, TRIAL))
for test_case in all_suggest:
all_suggest[test_case] = xru.ds_concat(all_suggest[test_case], dims=(TEST_CASE, METHOD, TRIAL))
assert all(
all_suggest[test_case][kk].dims == (ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)
for kk in all_suggest[test_case]
)
assert not any(np.any(np.isnan(all_suggest[test_case][kk].values)) for kk in all_suggest[test_case])
assert xru.coord_compat((all_perf, all_suggest[test_case]), (ITER, METHOD, TRIAL))
assert all_suggest[test_case].coords[TEST_CASE].shape == (1,), "test case should be singleton"
return all_perf, all_time, all_suggest, all_sigs
def load_experiments(uuid_list, db_root, dbid): # pragma: io
"""Generator to load the results of the experiments.
Parameters
----------
uuid_list : list(uuid.UUID)
List of UUIDs corresponding to experiments to load.
db_root : str
Root location for data store as requested by the serializer used.
dbid : str
Name of the data store as requested by the serializer used.
Yields
------
meta_data : (str, str, str)
The `meta_data` contains a `tuple` of `str` with ``test_case, optimizer, uuid``.
data : (:class:`xarray:xarray.Dataset`, :class:`xarray:xarray.Dataset`, :class:`xarray:xarray.Dataset` list(float))
The `data` contains a tuple of ``(perf_ds, time_ds, suggest_ds, sig)``. The `perf_ds` is a
:class:`xarray:xarray.Dataset` containing the evaluation results with dimensions ``(ITER, SUGGEST)``, each
variable is an objective. The `time_ds` is an :class:`xarray:xarray.Dataset` containing the timing results of
the form accepted by `summarize_time`. The coordinates must be compatible with `perf_ds`. The suggest_ds is a
:class:`xarray:xarray.Dataset` containing the inputs to the function evaluations. Each variable is a function
input. Finally, `sig` contains the `test_case` signature and must be `list(float)`.
"""
uuids_seen = set()
for uuid_ in uuid_list:
logger.info(uuid_.hex)
# Load perf and timing data
perf_ds, meta = XRSerializer.load(db_root, db=dbid, key=cc.EVAL, uuid_=uuid_)
time_ds, meta_t = XRSerializer.load(db_root, db=dbid, key=cc.TIME, uuid_=uuid_)
assert meta == meta_t, "meta data should between time and eval files"
suggest_ds, meta_t = XRSerializer.load(db_root, db=dbid, key=cc.SUGGEST_LOG, uuid_=uuid_)
assert meta == meta_t, "meta data should between suggest and eval files"
# Get signature to pass out as well
_, sig = meta["signature"]
logger.info(meta)
logger.info(sig)
# Build the new indices for combined data, this could be put in function for easier testing
eval_args = unserializable_dict(meta["args"]) # Unpack meta-data
test_case = SklearnModel.test_case_str(
eval_args[CmdArgs.classifier], eval_args[CmdArgs.data], eval_args[CmdArgs.metric]
)
optimizer = str_join_safe(
ARG_DELIM, (eval_args[CmdArgs.optimizer], eval_args[CmdArgs.opt_rev], eval_args[CmdArgs.rev])
)
args_uuid = eval_args[CmdArgs.uuid]
# Check UUID sanity
assert isinstance(args_uuid, str)
assert args_uuid == uuid_.hex, "UUID meta-data does not match filename"
assert args_uuid not in uuids_seen, "uuids being reused between studies"
uuids_seen.add(args_uuid)
# Return key -> data so this generator can be iterated over in dict like manner
meta_data = (test_case, optimizer, args_uuid)
data = (perf_ds, time_ds, suggest_ds, sig)
yield meta_data, data
def main():
"""See README for instructions on calling aggregate.
"""
description = "Aggregate study results across functions and optimizers"
args = parse_args(agg_parser(description))
logger.setLevel(logging.INFO) # Note this is the module-wide logger
if args[CmdArgs.verbose]:
logger.addHandler(logging.StreamHandler())
# Get list of UUIDs
uuid_list = XRSerializer.get_uuids(args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.EVAL)
uuid_list_ = XRSerializer.get_uuids(args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.TIME)
assert uuid_list == uuid_list_, "UUID list does not match between time and eval results"
uuid_list_ = XRSerializer.get_uuids(args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.SUGGEST_LOG)
assert uuid_list == uuid_list_, "UUID list does not match between suggest log and eval results"
# Get iterator of all experiment data dumps, load in and process, and concat
data_G = load_experiments(uuid_list, args[CmdArgs.db_root], args[CmdArgs.db])
all_perf, all_time, all_suggest, all_sigs = concat_experiments(data_G, ravel=args[CmdArgs.ravel])
# Check the concat signatures make are coherent
sig_errs, signatures_median = analyze_signatures(all_sigs)
logger.info("Signature errors:\n%s" % sig_errs.to_string())
print(json.dumps({"exp-agg sig errors": sig_errs.T.to_dict()}))
# Dump and save it all out
logger.info("saving")
meta = {"args": serializable_dict(args), "signature": signatures_median}
XRSerializer.save_derived(all_perf, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=EVAL_RESULTS)
XRSerializer.save_derived(all_time, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=TIME_RESULTS)
for test_case, ds in all_suggest.items():
XRSerializer.save_derived(ds, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=test_case)
logger.info("done")
if __name__ == "__main__":
main() # pragma: main
================================================
FILE: bayesmark/experiment_analysis.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Perform analysis to compare different optimizers across problems.
"""
import json
import logging
import warnings
from collections import OrderedDict
import numpy as np
import xarray as xr
import bayesmark.constants as cc
import bayesmark.quantiles as qt
import bayesmark.xr_util as xru
from bayesmark.cmd_parse import CmdArgs, general_parser, parse_args, serializable_dict
from bayesmark.constants import (
ITER,
LB_MEAN,
LB_MED,
LB_NORMED_MEAN,
METHOD,
NORMED_MEAN,
NORMED_MED,
OBJECTIVE,
PERF_BEST,
PERF_CLIP,
PERF_MEAN,
PERF_MED,
SUGGEST,
TEST_CASE,
TRIAL,
UB_MEAN,
UB_MED,
UB_NORMED_MEAN,
)
from bayesmark.experiment_aggregate import validate_agg_perf
from bayesmark.experiment_baseline import do_baseline
from bayesmark.np_util import cummin, linear_rescale
from bayesmark.serialize import XRSerializer
from bayesmark.signatures import analyze_signature_pair
from bayesmark.stats import t_EB
# Mathematical settings
EVAL_Q = 0.5 # Evaluate based on median loss across n_trials
ALPHA = 0.05 # ==> 95% CIs
logger = logging.getLogger(__name__)
def get_perf_array(evals, evals_visible):
"""Get the actual (e.g., generalization loss) over iterations.
Parameters
----------
evals : :class:`numpy:numpy.ndarray` of shape (n_iter, n_batch, n_trials)
The actual loss (e.g., generalization) for a given experiment.
evals_visible : :class:`numpy:numpy.ndarray` of shape (n_iter, n_batch, n_trials)
The observable loss (e.g., validation) for a given experiment.
Returns
-------
perf_array : :class:`numpy:numpy.ndarray` of shape (n_iter, n_trials)
The best performance so far at iteration i from `evals`. Where the best has been selected according to
`evals_visible`.
"""
n_iter, _, n_trials = evals.shape
assert evals.size > 0, "perf array not supported for empty arrays"
assert evals_visible.shape == evals.shape
assert not np.any(np.isnan(evals))
assert not np.any(np.isnan(evals_visible))
idx = np.argmin(evals_visible, axis=1)
perf_array = np.take_along_axis(evals, idx[:, None, :], axis=1).squeeze(axis=1)
assert perf_array.shape == (n_iter, n_trials)
visible_perf_array = np.min(evals_visible, axis=1)
assert visible_perf_array.shape == (n_iter, n_trials)
# Get the minimum from the visible loss
perf_array = cummin(perf_array, visible_perf_array)
return perf_array
def compute_aggregates(perf_da, baseline_ds, visible_perf_da=None):
"""Aggregate function evaluations in the experiments to get performance summaries of each method.
Parameters
----------
perf_da : :class:`xarray:xarray.DataArray`
Aggregate experimental results with each function evaluation in the experiments according to true loss
(e.g., generalization). `perf_da` has dimensions ``(ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)`` as is assumed
to have no nan values.
baseline_ds : :class:`xarray:xarray.Dataset`
Dataset with baseline performance. It was variables ``(PERF_MED, PERF_MEAN, PERF_CLIP, PERF_BEST)`` with
dimensions ``(ITER, TEST_CASE)``, ``(ITER, TEST_CASE)``, ``(TEST_CASE,)``, and ``(TEST_CASE,)``, respectively.
`PERF_MED` is a baseline of performance based on random search when using medians to summarize performance.
Likewise, `PERF_MEAN` is for means. `PERF_CLIP` is an upperbound to clip poor performance when using the mean.
`PERF_BEST` is an estimate on the global minimum.
visible_perf_da : :class:`xarray:xarray.DataArray`
Aggregate experimental results with each function evaluation in the experiments according to visible loss
(e.g., validation). `visible_perf_da` has dimensions ``(ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)`` as is
assumed to have no nan values. If `None`, we set ``visible_perf_da = perf_da``.
Returns
-------
agg_result : :class:`xarray:xarray.Dataset`
Dataset with summary of performance for each method and test case combination. Contains variables:
``(PERF_MED, LB_MED, UB_MED, NORMED_MED, PERF_MEAN, LB_MEAN, UB_MEAN, NORMED_MEAN)``
each with dimensions ``(ITER, METHOD, TEST_CASE)``. `PERF_MED` is a median summary of performance with `LB_MED`
and `UB_MED` as error bars. `NORMED_MED` is a rescaled `PERF_MED` so we expect the optimal performance is 0,
and random search gives 1 at all `ITER`. Likewise, `PERF_MEAN`, `LB_MEAN`, `UB_MEAN`, `NORMED_MEAN` are for
mean performance.
summary : :class:`xarray:xarray.Dataset`
Dataset with overall summary of performance of each method. Contains variables
``(PERF_MED, LB_MED, UB_MED, PERF_MEAN, LB_MEAN, UB_MEAN)``
each with dimensions ``(ITER, METHOD)``.
"""
validate_agg_perf(perf_da, min_trial=1)
assert isinstance(baseline_ds, xr.Dataset)
assert tuple(baseline_ds[PERF_BEST].dims) == (TEST_CASE,)
assert tuple(baseline_ds[PERF_CLIP].dims) == (TEST_CASE,)
assert tuple(baseline_ds[PERF_MED].dims) == (ITER, TEST_CASE)
assert tuple(baseline_ds[PERF_MEAN].dims) == (ITER, TEST_CASE)
assert xru.coord_compat((perf_da, baseline_ds), (ITER, TEST_CASE))
assert not any(np.any(np.isnan(baseline_ds[kk].values)) for kk in baseline_ds)
# Now actually get the aggregate performance numbers per test case
agg_result = xru.ds_like(
perf_da,
(PERF_MED, LB_MED, UB_MED, NORMED_MED, PERF_MEAN, LB_MEAN, UB_MEAN, NORMED_MEAN),
(ITER, METHOD, TEST_CASE),
)
baseline_mean_da = xru.only_dataarray(xru.ds_like(perf_da, ["ref"], (ITER, TEST_CASE)))
# Using values here since just clearer to get raw items than xr object for func_name
for func_name in perf_da.coords[TEST_CASE].values:
rand_perf_med = baseline_ds[PERF_MED].sel({TEST_CASE: func_name}, drop=True).values
rand_perf_mean = baseline_ds[PERF_MEAN].sel({TEST_CASE: func_name}, drop=True).values
best_opt = baseline_ds[PERF_BEST].sel({TEST_CASE: func_name}, drop=True).values
base_clip_val = baseline_ds[PERF_CLIP].sel({TEST_CASE: func_name}, drop=True).values
assert np.all(np.diff(rand_perf_med) <= 0), "Baseline should be decreasing with iteration"
assert np.all(np.diff(rand_perf_mean) <= 0), "Baseline should be decreasing with iteration"
assert np.all(rand_perf_med > best_opt)
assert np.all(rand_perf_mean > best_opt)
assert np.all(rand_perf_mean <= base_clip_val)
baseline_mean_da.loc[{TEST_CASE: func_name}] = linear_rescale(
rand_perf_mean, best_opt, base_clip_val, 0.0, 1.0, enforce_bounds=False
)
for method_name in perf_da.coords[METHOD].values:
# Take the minimum over all suggestion at given iter + sanity check perf_da
curr_da = perf_da.sel({METHOD: method_name, TEST_CASE: func_name}, drop=True)
assert curr_da.dims == (ITER, SUGGEST, TRIAL)
if visible_perf_da is None:
perf_array = get_perf_array(curr_da.values, curr_da.values)
curr_da_ = perf_da.sel({METHOD: method_name, TEST_CASE: func_name}, drop=True).min(dim=SUGGEST)
assert curr_da_.dims == (ITER, TRIAL)
perf_array_ = np.minimum.accumulate(curr_da_.values, axis=0)
assert np.allclose(perf_array, perf_array_)
else:
curr_visible_da = visible_perf_da.sel({METHOD: method_name, TEST_CASE: func_name}, drop=True)
assert curr_visible_da.dims == (ITER, SUGGEST, TRIAL)
perf_array = get_perf_array(curr_da.values, curr_visible_da.values)
# Compute median perf and CI on it
med_perf, LB, UB = qt.quantile_and_CI(perf_array, EVAL_Q, alpha=ALPHA)
assert med_perf.shape == rand_perf_med.shape
agg_result[PERF_MED].loc[{TEST_CASE: func_name, METHOD: method_name}] = med_perf
agg_result[LB_MED].loc[{TEST_CASE: func_name, METHOD: method_name}] = LB
agg_result[UB_MED].loc[{TEST_CASE: func_name, METHOD: method_name}] = UB
# Now store normed version, which is better for aggregation
normed = linear_rescale(med_perf, best_opt, rand_perf_med, 0.0, 1.0, enforce_bounds=False)
agg_result[NORMED_MED].loc[{TEST_CASE: func_name, METHOD: method_name}] = normed
# Store normed mean version
normed = linear_rescale(perf_array, best_opt, base_clip_val, 0.0, 1.0, enforce_bounds=False)
# Also, clip the score from below at -1 to limit max influence of single run on final average
normed = np.clip(normed, -1.0, 1.0)
normed = np.mean(normed, axis=1)
agg_result[NORMED_MEAN].loc[{TEST_CASE: func_name, METHOD: method_name}] = normed
# Compute mean perf and CI on it
perf_array = np.minimum(base_clip_val, perf_array)
mean_perf = np.mean(perf_array, axis=1)
assert mean_perf.shape == rand_perf_mean.shape
EB = t_EB(perf_array, alpha=ALPHA, axis=1)
assert EB.shape == rand_perf_mean.shape
agg_result[PERF_MEAN].loc[{TEST_CASE: func_name, METHOD: method_name}] = mean_perf
agg_result[LB_MEAN].loc[{TEST_CASE: func_name, METHOD: method_name}] = mean_perf - EB
agg_result[UB_MEAN].loc[{TEST_CASE: func_name, METHOD: method_name}] = mean_perf + EB
assert not any(np.any(np.isnan(agg_result[kk].values)) for kk in agg_result)
# Compute summary score over all test cases, summarize performance of each method
summary = xru.ds_like(
perf_da,
(PERF_MED, LB_MED, UB_MED, PERF_MEAN, LB_MEAN, UB_MEAN, NORMED_MEAN, LB_NORMED_MEAN, UB_NORMED_MEAN),
(ITER, METHOD),
)
summary[PERF_MED], summary[LB_MED], summary[UB_MED] = xr.apply_ufunc(
qt.quantile_and_CI,
agg_result[NORMED_MED],
input_core_dims=[[TEST_CASE]],
kwargs={"q": EVAL_Q, "alpha": ALPHA},
output_core_dims=[[], [], []],
)
summary[PERF_MEAN] = agg_result[NORMED_MEAN].mean(dim=TEST_CASE)
EB = xr.apply_ufunc(t_EB, agg_result[NORMED_MEAN], input_core_dims=[[TEST_CASE]])
summary[LB_MEAN] = summary[PERF_MEAN] - EB
summary[UB_MEAN] = summary[PERF_MEAN] + EB
normalizer = baseline_mean_da.mean(dim=TEST_CASE)
summary[NORMED_MEAN] = summary[PERF_MEAN] / normalizer
summary[LB_NORMED_MEAN] = summary[LB_MEAN] / normalizer
summary[UB_NORMED_MEAN] = summary[UB_MEAN] / normalizer
assert all(tuple(summary[kk].dims) == (ITER, METHOD) for kk in summary)
return agg_result, summary
def main():
"""See README for instructions on calling analysis.
"""
description = "Analyze results from aggregated studies"
args = parse_args(general_parser(description))
# Metric used on leaderboard
leaderboard_metric = cc.VISIBLE_TO_OPT
logger.setLevel(logging.INFO) # Note this is the module-wide logger
if args[CmdArgs.verbose]:
logger.addHandler(logging.StreamHandler())
# Load in the eval data and sanity check
perf_ds, meta = XRSerializer.load_derived(args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.EVAL_RESULTS)
logger.info("Meta data from source file: %s" % str(meta["args"]))
# Check if there is baselines file, other make one
if cc.BASELINE not in XRSerializer.get_derived_keys(args[CmdArgs.db_root], db=args[CmdArgs.db]):
warnings.warn("Baselines not found. Need to construct baseline.")
do_baseline(args)
# Load in baseline scores data and sanity check (including compatibility with eval data)
baseline_ds, meta_ref = XRSerializer.load_derived(args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.BASELINE)
logger.info("baseline data from source ref file: %s" % str(meta_ref["args"]))
# Check test case signatures match between eval data and baseline data
sig_errs, signatures = analyze_signature_pair(meta["signature"], meta_ref["signature"])
logger.info("Signature errors:\n%s" % sig_errs.to_string())
print(json.dumps({"exp-anal sig errors": sig_errs.T.to_dict()}))
# Subset baseline to only the test cases run in the experiments
test_cases_run = perf_ds.coords[TEST_CASE].values.tolist()
assert set(test_cases_run) <= set(
baseline_ds.coords[TEST_CASE].values.tolist()
), "Data set contains test cases not found in baseline."
baseline_ds = baseline_ds.sel({TEST_CASE: test_cases_run})
# Also subset to allow shorter runs
iters_run = perf_ds.coords[ITER].values.tolist()
assert set(iters_run) <= set(
baseline_ds.coords[ITER].values.tolist()
), "Data set not same batch size or too many iters compared to baseline."
baseline_ds = baseline_ds.sel({ITER: iters_run})
# Do the actual computation
perf_visible = perf_ds[cc.VISIBLE_TO_OPT]
agg_result = OrderedDict()
summary = OrderedDict()
for metric_for_scoring in sorted(perf_ds):
perf_da = perf_ds[metric_for_scoring]
baseline_ds_ = baseline_ds.sel({OBJECTIVE: metric_for_scoring}, drop=True)
agg_result[(metric_for_scoring,)], summary[(metric_for_scoring,)] = compute_aggregates(
perf_da, baseline_ds_, perf_visible
)
agg_result = xru.ds_concat(agg_result, dims=(cc.OBJECTIVE,))
summary = xru.ds_concat(summary, dims=(cc.OBJECTIVE,))
for metric_for_scoring in sorted(perf_ds):
# Print summary by problem
# Recall that:
# ... summary[PERF_MEAN] = agg_result[NORMED_MEAN].mean(dim=TEST_CASE)
# ... summary[NORMED_MEAN] = summary[PERF_MEAN] / normalizer
# Where normalizer is constant across all problems, optimizers
print("Scores by problem (JSON):\n")
agg_df = agg_result[NORMED_MEAN].sel({cc.OBJECTIVE: metric_for_scoring}, drop=True)[{ITER: -1}].to_pandas().T
print(json.dumps({metric_for_scoring: agg_df.to_dict()}))
print("\n")
final_score = summary[PERF_MED].sel({cc.OBJECTIVE: metric_for_scoring}, drop=True)[{ITER: -1}]
logger.info("median score @ %d:\n%s" % (summary.sizes[ITER], xru.da_to_string(final_score)))
final_score = summary[PERF_MEAN].sel({cc.OBJECTIVE: metric_for_scoring}, drop=True)[{ITER: -1}]
logger.info("mean score @ %d:\n%s" % (summary.sizes[ITER], xru.da_to_string(final_score)))
print("Final scores (JSON):\n")
print(json.dumps({metric_for_scoring: final_score.to_series().to_dict()}))
print("\n")
final_score = summary[NORMED_MEAN].sel({cc.OBJECTIVE: metric_for_scoring}, drop=True)[{ITER: -1}]
logger.info("normed mean score @ %d:\n%s" % (summary.sizes[ITER], xru.da_to_string(final_score)))
# Now saving results
meta = {"args": serializable_dict(args), "signature": signatures}
XRSerializer.save_derived(agg_result, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.PERF_RESULTS)
XRSerializer.save_derived(summary, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.MEAN_SCORE)
final_msg = xru.da_to_string(
100 * (1.0 - summary[PERF_MEAN].sel({cc.OBJECTIVE: leaderboard_metric}, drop=True)[{ITER: -1}])
)
logger.info("-" * 20)
logger.info("Final score `100 x (1-loss)` for leaderboard:\n%s" % final_msg)
if __name__ == "__main__":
main() # pragma: main
================================================
FILE: bayesmark/experiment_baseline.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Build performance baselines from aggregate results to prepare analysis.
"""
import logging
import warnings
from collections import OrderedDict
import numpy as np
import bayesmark.constants as cc
import bayesmark.expected_max as em
import bayesmark.quantiles as qt
from bayesmark.cmd_parse import CmdArgs, general_parser, parse_args
from bayesmark.constants import ARG_DELIM, ITER, METHOD, PERF_BEST, PERF_CLIP, PERF_MEAN, PERF_MED, SUGGEST, TEST_CASE
from bayesmark.experiment_aggregate import validate_agg_perf
from bayesmark.serialize import XRSerializer
from bayesmark.util import str_join_safe
from bayesmark.xr_util import ds_concat, ds_like_mixed
# Mathematical settings
# We could move these to constants to eliminate repetition but we will probably phase out anyway
EVAL_Q = 0.5 # Evaluate based on median loss across n_trials
ALPHA = 0.05 # ==> 95% CIs
MIN_POS = np.nextafter(0, 1)
PAD_FACTOR = 10000
logger = logging.getLogger(__name__)
def validate(baseline_ds):
"""Perform same tracks as will happen in analysis."""
for func_name in baseline_ds.coords[TEST_CASE].values:
rand_perf_med = baseline_ds[PERF_MED].sel({TEST_CASE: func_name}, drop=True).values
rand_perf_mean = baseline_ds[PERF_MEAN].sel({TEST_CASE: func_name}, drop=True).values
best_opt = baseline_ds[PERF_BEST].sel({TEST_CASE: func_name}, drop=True).values
base_clip_val = baseline_ds[PERF_CLIP].sel({TEST_CASE: func_name}, drop=True).values
assert np.all(np.diff(rand_perf_med) <= 0), "Baseline should be decreasing with iteration"
assert np.all(np.diff(rand_perf_mean) <= 0), "Baseline should be decreasing with iteration"
assert np.all(rand_perf_med > best_opt)
assert np.all(rand_perf_mean > best_opt)
assert np.all(rand_perf_mean <= base_clip_val)
def compute_baseline(perf_da):
"""Compute a performance baseline of base and best performance from the aggregate experimental results.
Parameters
----------
perf_da : :class:`xarray:xarray.DataArray`
Aggregate experimental results with each function evaluation in the experiments. `all_perf` has dimensions
``(ITER, SUGGEST, TEST_CASE, METHOD, TRIAL)`` as is assumed to have no nan values.
Returns
-------
baseline_ds : :class:`xarray:xarray.Dataset`
Dataset with baseline performance. It was variables ``(PERF_MED, PERF_MEAN, PERF_CLIP, PERF_BEST)`` with
dimensions ``(ITER, TEST_CASE)``, ``(ITER, TEST_CASE)``, ``(TEST_CASE,)``, and ``(TEST_CASE,)``, respectively.
`PERF_MED` is a baseline of performance based on random search when using medians to summarize performance.
Likewise, `PERF_MEAN` is for means. `PERF_CLIP` is an upperbound to clip poor performance when using the mean.
`PERF_BEST` is an estimate on the global minimum.
"""
validate_agg_perf(perf_da)
ref_prefix = str_join_safe(ARG_DELIM, (cc.RANDOM_SEARCH, ""))
ref_random = [kk for kk in perf_da.coords[METHOD].values if kk.startswith(ref_prefix)]
assert len(ref_random) > 0, "Did not find any random search in methods."
# Now many points we will have after each batch
trials_grid = perf_da.sizes[SUGGEST] * (1 + np.arange(perf_da.sizes[ITER]))
# Now iterate over problems and get baseline performance
baseline_ds = ds_like_mixed(
perf_da,
[
(PERF_MED, [ITER, TEST_CASE]),
(PERF_MEAN, [ITER, TEST_CASE]),
(PERF_CLIP, [TEST_CASE]),
(PERF_BEST, [TEST_CASE]),
],
(ITER, TEST_CASE),
)
for func_name in perf_da.coords[TEST_CASE].values:
random_evals = np.ravel(perf_da.sel({METHOD: ref_random, TEST_CASE: func_name}, drop=True).values)
assert random_evals.size > 0
# We will likely change this to a min mean (instead of median) using a different util in near future:
assert np.all(trials_grid == perf_da.sizes[SUGGEST] * (1 + baseline_ds.coords[ITER].values))
rand_perf, _, _ = qt.min_quantile_CI(random_evals, EVAL_Q, trials_grid, alpha=ALPHA)
baseline_ds[PERF_MED].loc[{TEST_CASE: func_name}] = rand_perf
# Decide on a level to clip when computing the mean
base_clip_val = qt.quantile(random_evals, EVAL_Q)
assert np.isfinite(base_clip_val), "Median random search performance is not even finite."
assert (perf_da.sizes[SUGGEST] > 1) or np.isclose(base_clip_val, rand_perf[0])
baseline_ds[PERF_CLIP].loc[{TEST_CASE: func_name}] = base_clip_val
# Estimate the global min via best of any method
best_opt = np.min(perf_da.sel({TEST_CASE: func_name}, drop=True).values)
if np.any(rand_perf <= best_opt):
warnings.warn(
"Random search is also the best search on %s, the normalized score may be meaningless." % func_name,
RuntimeWarning,
)
assert np.isfinite(best_opt), "Best performance found is not even finite."
logger.info("best %s %f" % (func_name, best_opt))
# Now make sure strictly less than to avoid assert error in linear_rescale. This will likely give normalized
# scores of +inf or -inf, but with median summary that is ok. When everything goes to mean, we will need to
# change this:
pad = PAD_FACTOR * np.spacing(-np.maximum(MIN_POS, np.abs(best_opt)))
assert pad < 0
best_opt = best_opt + pad
assert np.isfinite(best_opt), "Best performance too close to limit of float range."
assert np.all(rand_perf > best_opt)
baseline_ds[PERF_BEST].loc[{TEST_CASE: func_name}] = best_opt
random_evals = np.minimum(base_clip_val, random_evals)
assert np.all(np.isfinite(random_evals))
assert np.all(best_opt <= random_evals)
rand_perf = em.expected_min(random_evals, trials_grid)
rand_perf_fixed = np.minimum(base_clip_val, rand_perf)
assert np.allclose(rand_perf, rand_perf_fixed)
rand_perf_fixed = np.minimum.accumulate(rand_perf_fixed)
assert np.allclose(rand_perf, rand_perf_fixed)
baseline_ds[PERF_MEAN].loc[{TEST_CASE: func_name}] = rand_perf_fixed
assert not any(np.any(np.isnan(baseline_ds[kk].values)) for kk in baseline_ds)
validate(baseline_ds)
return baseline_ds
def do_baseline(args): # pragma: io
"""Alternate entry into the program without calling the actual main.
"""
# Load in the eval data and sanity check
perf_ds, meta = XRSerializer.load_derived(args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.EVAL_RESULTS)
logger.info("Meta data from source file: %s" % str(meta["args"]))
D = OrderedDict()
for kk in perf_ds:
perf_da = perf_ds[kk]
D[(kk,)] = compute_baseline(perf_da)
baseline_ds = ds_concat(D, dims=(cc.OBJECTIVE,))
# Keep in same order for cleanliness
baseline_ds = baseline_ds.sel({cc.OBJECTIVE: list(perf_ds)})
assert list(perf_ds) == baseline_ds.coords[cc.OBJECTIVE].values.tolist()
# Could optionally remove this once we think things have enough tests
for kk in D:
assert baseline_ds.sel({cc.OBJECTIVE: kk[0]}, drop=True).identical(D[kk])
# Now dump the results
XRSerializer.save_derived(baseline_ds, meta, args[CmdArgs.db_root], db=args[CmdArgs.db], key=cc.BASELINE)
def main():
"""See README for instructions on calling baseline.
"""
description = "Aggregate the baselines for later analysis in benchmark"
args = parse_args(general_parser(description))
logger.setLevel(logging.INFO) # Note this is the module-wide logger
if args[CmdArgs.verbose]:
logger.addHandler(logging.StreamHandler())
do_baseline(args)
logger.info("done")
if __name__ == "__main__":
main() # pragma: main
================================================
FILE: bayesmark/experiment_db_init.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Tool to create new datebase for results. This is just wrapper on serializer init call.
"""
import logging
import bayesmark.cmd_parse as cmd
from bayesmark.cmd_parse import CmdArgs
from bayesmark.constants import EXP_VARS
from bayesmark.serialize import XRSerializer
EXIST_OK = True
logger = logging.getLogger(__name__)
def main():
"""See README for instructions on calling db_init.
"""
description = "Initialize the directories for running the experiments"
args = cmd.parse_args(cmd.general_parser(description))
assert not args[CmdArgs.dry_run], "Dry run doesn't make any sense when building dirs"
logger.setLevel(logging.INFO) # Note this is the module-wide logger
if args[CmdArgs.verbose]:
logger.addHandler(logging.StreamHandler())
XRSerializer.init_db(args[CmdArgs.db_root], db=args[CmdArgs.db], keys=EXP_VARS, exist_ok=EXIST_OK)
logger.info("done")
if __name__ == "__main__":
main() # pragma: main
================================================
FILE: bayesmark/experiment_launcher.py
================================================
# Copyright (c) 2019 Uber Technologies, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Launch studies in separate studies or do dry run to build jobs file with lists of commands to run.
"""
import json
import logging
import random as pyrandom
import uuid as pyuuid
import warnings
from itertools import product
from subprocess import TimeoutExpired, call
import numpy as np
import bayesmark.cmd_parse as cmd
from bayesmark.builtin_opt.config import CONFIG
from bayesmark.cmd_parse import CMD_STR, CmdArgs, serializable_dict
from bayesmark.constants import ARG_DELIM, DATA_LOADER_NAMES, EXP_VARS, METRICS, MODEL_NAMES, PY_INTERPRETER
from bayesmark.data import METRICS_LOOKUP, get_problem_type
from bayesmark.np_util import random as np_random
from bayesmark.np_util import random_seed, strat_split
from bayesmark.path_util import absopen
from bayesmark.serialize import XRSerializer
from bayesmark.util import range_str, shell_join, str_join_safe, strict_sorted
# How much of uuid to put in job name to avoid name clashes
UUID_JOB_CHARS = 7
# Warning: this name is also specified in setup.py, and violates the DRY principle. So if it gets changed in setup.py,
# it must also be changed here!
EXPERIMENT_ENTRY = "bayesmark-exp"
logger = logging.getLogger(__name__)
def _is_arg_safe(ss):
"""Check if `str` is safe as argument to `argparse`."""
if len(ss) == 0:
return False
safe = ss[0] != "-"
return safe
def arg_safe_str(val):
"""Cast value as `str`, raise error if not safe as argument to `argparse`."""
ss = str(val)
if not _is_arg_safe(ss):
raise ValueError("%s is not safe for argparse" % ss)
return ss
def gen_commands(args, opt_file_lookup, run_uuid):
"""Generator providing commands to launch processes for experiments.
Parameters
----------
args : dict(CmdArgs, [int, str])
Arguments of options to pass to the experiments being launched. The keys corresponds to the same arguments
passed to this program.
opt_file_lookup : dict(str, str)
Mapping from method name to filename containing wrapper class for the method.
run_uuid : uuid.UUID
UUID for this launcher run. Needed to generate different experiments UUIDs on each call. This function is
deterministic provided the same `run_uuid`.
Yields
------
iteration_key : (str, str, str, str)
Tuple containing ``(trial, classifier, data, optimizer)`` to index the experiment.
full_cmd : tuple(str)
Strings containing command and arguments to run a process with experiment. Join with whitespace or use
:func:`.util.shell_join` to get string with executable command. The command omits ``--opt-root`` which means it
will default to ``.`` if the command is executed. As such, the command assumes it is executed with
``--opt-root`` as the working directory.
"""
args_to_pass_thru = [CmdArgs.n_calls, CmdArgs.n_suggest, CmdArgs.db_root, CmdArgs.db]
# This could be made simpler and avoid if statement if we just always pass dataroot, even if no custom data used.
if args[CmdArgs.data_root] is not None:
args_to_pass_thru.append(CmdArgs.data_root)
# Possibilities to iterate over. Put them in sorted order just for good measure.
c_list = strict_sorted(MODEL_NAMES if args[CmdArgs.classifier] is None else args[CmdArgs.classifier])
d_list = strict_sorted(DATA_LOADER_NAMES if args[CmdArgs.data] is None else args[CmdArgs.data])
o_list = strict_sorted(
list(opt_file_lookup.keys()) + list(CONFIG.keys())
if args[CmdArgs.optimizer] is None
else args[CmdArgs.optimizer]
)
assert all(
((optimizer in opt_file_lookup) or (optimizer in CONFIG)) for optimizer in o_list
), "unknown optimizer in optimizer list"
m_set = set(METRICS if args[CmdArgs.metric] is None else args[CmdArgs.metric])
m_lookup = {problem_type: sorted(m_set.intersection(mm)) for problem_type, mm in METRICS_LOOKUP.items()}
assert all(
(len(m_lookup[get_problem_type(data)]) > 0) for data in d_list
), "At one metric needed for each problem type of data sets"
G = product(range_str(args[CmdArgs.n_repeat]), c_list, d_list, o_list) # iterate all combos
for rep, classifier, data, optimizer in G:
_, rep_str = rep
problem_type = get_problem_type(data)
for metric in m_lookup[problem_type]:
# Get a reproducible string based (conditioned on having same (run uuid), but should also never give
# a duplicate (unless we force the same run uuid twice).
iteration_key = (rep_str, classifier, data, optimizer, metric)
iteration_id = str_join_safe(ARG_DELIM, iteration_key)
sub_uuid = pyuuid.uuid5(run_uuid, iteration_id).hex
# Build the argument list for subproc, passing some args thru
cmd_args_pass_thru = [[CMD_STR[vv][0], arg_safe_str(args[vv])] for vv in args_to_pass_thru]
# Technically, the optimizer is is not actually needed here for non-built in optimizers because it already
# specified via the entry point: optimizer_wrapper_file
cmd_args = [
[CMD_STR[CmdArgs.classifier][0], arg_safe_str(classifier)],
[CMD_STR[CmdArgs.data][0], arg_safe_str(data)],
[CMD_STR[CmdArgs.optimizer][0], arg_safe_str(optimizer)],
[CMD_STR[CmdArgs.uuid][0], arg_safe_str(sub_uuid)],
[CMD_STR[CmdArgs.metric][0], arg_safe_str(metric)],
]
cmd_args = tuple(sum(cmd_args + cmd_args_pass_thru, []))
logger.info(" ".join(cmd_args))
# The experiment command without the arguments
if optimizer in CONFIG: # => built in optimizer wrapper
experiment_cmd = (EXPERIMENT_ENTRY,)
else:
optimizer_wrapper_file = opt_file_lookup[optimizer]
assert optimizer_wrapper_file.endswith(".py"), "optimizer wrapper should a be .py file"
experiment_cmd = (PY_INTERPRETER, optimizer_wrapper_file)
# Check arg safe again, off elements in list need to be argsafe
assert all((_is_arg_safe(ss) == (ii % 2 == 1)) for ii, ss in enumerate(cmd_args))
full_cmd = experiment_cmd + cmd_args
yield iteration_key, full_cmd
def dry_run(args, opt_file_lookup, run_uuid, fp, random=np_random):
"""Write to buffer description of commands for running all experiments.
This function is almost pure by writing to a buffer, but it could be switched to a generator.
Parameters
----------
args : dict(CmdArgs, [int, str])
Arguments of options to pass to the experiments being launched. The keys corresponds to the same arguments
passed to this program.
opt_file_lookup : dict(str, str)
Mapping from method name to filename containing wrapper class for the method.
run_uuid : uuid.UUID
UUID for this launcher run. Needed to generate different experiments UUIDs on each call. This function is
deterministic provided the same `run_uuid`.
fp : writable buffer
File handle to write out sequence of commands to execute (broken into jobs on each line) to execute all the
experiments (possibly each job in parallel).
random : RandomState
Random stream to use for reproducibility.
"""
assert args[CmdArgs.n_jobs] > 0, "Must have non-zero jobs for dry run"
# Taking in file pointer since then we can test without actual file. Could also build generator that returns lines
# to write.
manual_setup_info = XRSerializer.init_db_manual(args[CmdArgs.db_root], db=args[CmdArgs.db], keys=EXP_VARS)
warnings.warn(manual_setup_info, UserWarning)
# Get the commands
dry_run_commands = {}
G = gen_commands(args, opt_file_lookup, run_uuid)
for (_, _, _, optimizer, _), full_cmd in G:
cmd_str = shell_join(full_cmd)
dry_run_commands.setdefault(optimizer, []).append(cmd_str)
# Make sure we never have any empty jobs, which is a waste
n_commands = sum(len(v) for v in dry_run_commands.values())
n_jobs = min(args[CmdArgs.n_jobs], n_commands)
# Would prob also work with pyrandom, but only tested np random so far
subcommands = strat_split(list(dry_run_commands.values()), n_jobs, random=random)
# Make sure have same commands overall, delete once we trust strat_split
assert sorted(np.concatenate(subcommands)) == sorted(sum(list(dry_run_commands.values()), []))
job_suffix = run_uuid.hex[:UUID_JOB_CHARS]
# Include comments as reproducibility lines
args_str = serializable_dict(args)
fp.write("# running: %s\n" % str(args_str))
fp.write("# cmd: %s\n" % cmd.cmd_str())
for ii, ii_str in range_str(n_jobs):
assert len(subcommands[ii]) > 0
fp.write("job_%s_%s %s\n" % (job_suffix, ii_str, " && ".join(subcommands[ii])))
def real_run(args, opt_file_lookup, run_uuid, timeout=None): # pragma: io
"""Run sequence of independent experiments to fully run the benchmark.
This uses `subprocess` to launch a separate process (in serial) for each experiment.
Parameters
----------
args : dict(CmdArgs, [int, str])
Arguments of options to pass to the experiments being launched. The keys corresponds to the same arguments
passed to this program.
opt_file_lookup : dict(str, str)
Mapping from method name to filename containing wrapper class for the method.
run_uuid : uuid.UUID
UUID for this launcher run. Needed to generate different experiments UUIDs on each call. This function is
deterministic provided the same `run_uuid`.
timeout : int
Max seconds per experiment
"""
args[CmdArgs.db] = XRSerializer.init_db(args[CmdArgs.db_root], db=args[CmdArgs.db], keys=EXP_VARS, exist_ok=True)
logger.info("Supply --db %s to append to this experiment or reproduce jobs file." % args[CmdArgs.db])
# Get and run the commands in a sub-process
counter = 0
G = gen_commands(args, opt_file_lookup, run_uuid)
for _, full_cmd in G:
try:
status = call(full_cmd, shell=False, cwd=args[CmdArgs.optimizer_root], timeout=timeout)
if status != 0:
gitextract_vn3wrx6j/
├── .coveragerc
├── .gitignore
├── .pre-commit-config.yaml
├── .readthedocs.yml
├── .secrets.baseline
├── .travis.yml
├── LICENSE
├── MANIFEST.in
├── README.rst
├── bayesmark/
│ ├── __init__.py
│ ├── abstract_optimizer.py
│ ├── builtin_opt/
│ │ ├── __init__.py
│ │ ├── config.py
│ │ ├── hyperopt_optimizer.py
│ │ ├── nevergrad_optimizer.py
│ │ ├── opentuner_optimizer.py
│ │ ├── pysot_optimizer.py
│ │ ├── random_optimizer.py
│ │ └── scikit_optimizer.py
│ ├── cmd_parse.py
│ ├── constants.py
│ ├── data.py
│ ├── expected_max.py
│ ├── experiment.py
│ ├── experiment_aggregate.py
│ ├── experiment_analysis.py
│ ├── experiment_baseline.py
│ ├── experiment_db_init.py
│ ├── experiment_launcher.py
│ ├── np_util.py
│ ├── path_util.py
│ ├── quantiles.py
│ ├── random_search.py
│ ├── serialize.py
│ ├── signatures.py
│ ├── sklearn_funcs.py
│ ├── space.py
│ ├── stats.py
│ ├── util.py
│ └── xr_util.py
├── build_wheel.sh
├── docs/
│ ├── .gitignore
│ ├── Makefile
│ ├── authors.rst
│ ├── code.rst
│ ├── conf.py
│ ├── dummy.py
│ ├── index.rst
│ ├── readme.rst
│ └── scoring.rst
├── example_opt_root/
│ ├── config.json
│ ├── flaky_optimizer.py
│ ├── hyperopt_optimizer.py
│ ├── nevergrad_optimizer.py
│ ├── opentuner_optimizer.py
│ ├── pysot_optimizer.py
│ ├── random_optimizer.py
│ └── scikit_optimizer.py
├── integration_test.sh
├── integration_test_with_setup.sh
├── notebooks/
│ ├── dummy.py
│ ├── plot_mean_score.ipynb
│ └── plot_test_case.ipynb
├── requirements/
│ ├── base.in
│ ├── base.txt
│ ├── docs.in
│ ├── docs.txt
│ ├── ipynb.in
│ ├── ipynb.txt
│ ├── optimizers.in
│ ├── optimizers.txt
│ ├── pipreqs_edits.sed
│ ├── self.txt
│ ├── test.in
│ ├── test.txt
│ ├── tools.in
│ └── tools.txt
├── setup.py
├── test/
│ ├── data_test.py
│ ├── dummy.py
│ ├── expected_max_test.py
│ ├── experiment_aggregate_test.py
│ ├── experiment_analysis_test.py
│ ├── experiment_baseline_test.py
│ ├── experiment_db_init_test.py
│ ├── experiment_launcher_test.py
│ ├── experiment_test.py
│ ├── hypothesis_util.py
│ ├── np_util_test.py
│ ├── quantiles_test.py
│ ├── random_search_test.py
│ ├── serialize_test.py
│ ├── signatures_test.py
│ ├── sklearn_funcs_test.py
│ ├── space_test.py
│ ├── stats_test.py
│ ├── util.py
│ ├── util_test.py
│ └── xr_util_test.py
├── test.sh
└── tools/
├── archive_branch.sh
└── deploy.sh
SYMBOL INDEX (465 symbols across 55 files)
FILE: bayesmark/abstract_optimizer.py
class AbstractOptimizer (line 21) | class AbstractOptimizer(ABC):
method __init__ (line 28) | def __init__(self, api_config, **kwargs):
method get_version (line 39) | def get_version(cls):
method suggest (line 53) | def suggest(self, n_suggestions):
method observe (line 71) | def observe(self, X, y):
FILE: bayesmark/builtin_opt/hyperopt_optimizer.py
function dummy_f (line 27) | def dummy_f(x):
function only (line 31) | def only(x):
class HyperoptOptimizer (line 36) | class HyperoptOptimizer(AbstractOptimizer):
method __init__ (line 39) | def __init__(self, api_config, random=np_random):
method hashable_dict (line 61) | def hashable_dict(d):
method get_hyperopt_dimensions (line 78) | def get_hyperopt_dimensions(api_config):
method get_trial (line 132) | def get_trial(self, trial_id):
method cleanup_guess (line 142) | def cleanup_guess(self, x_guess):
method _suggest (line 157) | def _suggest(self):
method suggest (line 175) | def suggest(self, n_suggestions=1):
method observe (line 212) | def observe(self, X, y):
FILE: bayesmark/builtin_opt/nevergrad_optimizer.py
class NevergradOptimizer (line 24) | class NevergradOptimizer(AbstractOptimizer):
method __init__ (line 27) | def __init__(self, api_config, tool, budget):
method get_nvg_dimensions (line 46) | def get_nvg_dimensions(api_config):
method prewarp (line 102) | def prewarp(self, xx):
method postwarp (line 125) | def postwarp(self, xxw):
method suggest (line 146) | def suggest(self, n_suggestions=1):
method observe (line 171) | def observe(self, X, y):
FILE: bayesmark/builtin_opt/opentuner_optimizer.py
function ClippedParam (line 73) | def ClippedParam(cls, epsilon=1e-5):
class OpentunerOptimizer (line 106) | class OpentunerOptimizer(AbstractOptimizer):
method __init__ (line 109) | def __init__(self, api_config, techniques=DEFAULT_TECHNIQUES, n_sugges...
method hashable_dict (line 176) | def hashable_dict(d):
method build_manipulator (line 193) | def build_manipulator(api_config):
method suggest (line 241) | def suggest(self, n_suggestions=1):
method observe (line 305) | def observe(self, X, y):
FILE: bayesmark/builtin_opt/pysot_optimizer.py
class PySOTOptimizer (line 28) | class PySOTOptimizer(AbstractOptimizer):
method __init__ (line 31) | def __init__(self, api_config):
method create_opt_prob (line 49) | def create_opt_prob(self):
method start (line 61) | def start(self, max_evals):
method suggest (line 91) | def suggest(self, n_suggestions=1):
method _observe (line 141) | def _observe(self, x, y):
method observe (line 150) | def observe(self, X, y):
FILE: bayesmark/builtin_opt/random_optimizer.py
class RandomOptimizer (line 19) | class RandomOptimizer(AbstractOptimizer):
method __init__ (line 23) | def __init__(self, api_config, random=np_util.random):
method suggest (line 36) | def suggest(self, n_suggestions=1):
method observe (line 54) | def observe(self, X, y):
FILE: bayesmark/builtin_opt/scikit_optimizer.py
class ScikitOptimizer (line 22) | class ScikitOptimizer(AbstractOptimizer):
method __init__ (line 25) | def __init__(self, api_config, base_estimator="GP", acq_func="gp_hedge...
method get_sk_dimensions (line 65) | def get_sk_dimensions(api_config, transform="normalize"):
method suggest (line 116) | def suggest(self, n_suggestions=1):
method observe (line 143) | def observe(self, X, y):
FILE: bayesmark/cmd_parse.py
class CmdArgs (line 40) | class CmdArgs(IntEnum):
function arg_to_str (line 87) | def arg_to_str(arg):
function namespace_to_dict (line 93) | def namespace_to_dict(args_ns):
function serializable_dict (line 99) | def serializable_dict(args):
function unserializable_dict (line 105) | def unserializable_dict(args_str):
function add_argument (line 111) | def add_argument(parser, arg, **kwargs):
function filepath (line 117) | def filepath(value):
function filename (line 125) | def filename(value):
function uuid (line 130) | def uuid(val_str):
function positive_int (line 137) | def positive_int(val_str):
function joinable (line 145) | def joinable(val_str):
function load_rev_number (line 154) | def load_rev_number():
function base_parser (line 197) | def base_parser():
function launcher_parser (line 211) | def launcher_parser(description):
function experiment_parser (line 246) | def experiment_parser(description):
function agg_parser (line 266) | def agg_parser(description):
function general_parser (line 278) | def general_parser(description):
function parse_args (line 284) | def parse_args(parser, argv=None):
function _cleanup (line 309) | def _cleanup(filename_str):
function infer_settings (line 315) | def infer_settings(opt_root, opt_pattern="**/optimizer.py"):
function load_optimizer_settings (line 333) | def load_optimizer_settings(opt_root):
function cmd_str (line 346) | def cmd_str():
FILE: bayesmark/data.py
class ProblemType (line 27) | class ProblemType(IntEnum):
function get_problem_type (line 50) | def get_problem_type(dataset_name):
function _csv_loader (line 75) | def _csv_loader(dataset_name, return_X_y, data_root, clip_x=100): # pra...
function load_data (line 133) | def load_data(dataset_name, data_root=None): # pragma: io
FILE: bayesmark/expected_max.py
function get_expected_max_weights (line 20) | def get_expected_max_weights(n, m):
function expected_max (line 53) | def expected_max(x, m):
function expected_min (line 88) | def expected_min(x, m):
FILE: bayesmark/experiment.py
function _build_test_problem (line 47) | def _build_test_problem(model_name, dataset, scorer, path):
function run_study (line 77) | def run_study(optimizer, test_problem, n_calls, n_suggestions, n_obj=1, ...
function run_sklearn_study (line 187) | def run_sklearn_study(
function get_objective_signature (line 248) | def get_objective_signature(model_name, dataset, scorer, data_root=None):
function build_eval_ds (line 277) | def build_eval_ds(function_evals, objective_names):
function build_timing_ds (line 306) | def build_timing_ds(suggest_time, eval_time, observe_time):
function build_suggest_ds (line 342) | def build_suggest_ds(suggest_log):
function load_optimizer_kwargs (line 376) | def load_optimizer_kwargs(optimizer_name, opt_root): # pragma: io
function _setup_seeds (line 402) | def _setup_seeds(hex_str): # pragma: main
function experiment_main (line 423) | def experiment_main(opt_class, args=None): # pragma: main
function _get_opt_class (line 569) | def _get_opt_class(opt_name):
function main (line 609) | def main(): # pragma: main
FILE: bayesmark/experiment_aggregate.py
function validate_time (line 35) | def validate_time(all_time):
function validate_perf (line 44) | def validate_perf(perf_da):
function validate_agg_perf (line 52) | def validate_agg_perf(perf_da, min_trial=1):
function summarize_time (line 61) | def summarize_time(all_time):
function concat_experiments (line 89) | def concat_experiments(all_experiments, ravel=False):
function load_experiments (line 169) | def load_experiments(uuid_list, db_root, dbid): # pragma: io
function main (line 231) | def main():
FILE: bayesmark/experiment_analysis.py
function get_perf_array (line 62) | def get_perf_array(evals, evals_visible):
function compute_aggregates (line 96) | def compute_aggregates(perf_da, baseline_ds, visible_perf_da=None):
function main (line 237) | def main():
FILE: bayesmark/experiment_baseline.py
function validate (line 42) | def validate(baseline_ds):
function compute_baseline (line 57) | def compute_baseline(perf_da):
function do_baseline (line 145) | def do_baseline(args): # pragma: io
function main (line 170) | def main():
FILE: bayesmark/experiment_db_init.py
function main (line 28) | def main():
FILE: bayesmark/experiment_launcher.py
function _is_arg_safe (line 46) | def _is_arg_safe(ss):
function arg_safe_str (line 54) | def arg_safe_str(val):
function gen_commands (line 62) | def gen_commands(args, opt_file_lookup, run_uuid):
function dry_run (line 149) | def dry_run(args, opt_file_lookup, run_uuid, fp, random=np_random):
function real_run (line 204) | def real_run(args, opt_file_lookup, run_uuid, timeout=None): # pragma: io
function main (line 241) | def main():
FILE: bayesmark/np_util.py
function random_seed (line 25) | def random_seed(random=random):
function shuffle_2d (line 45) | def shuffle_2d(X, random=random):
function strat_split (line 62) | def strat_split(X, n_splits, inplace=False, random=random):
function isclose_lte (line 99) | def isclose_lte(x, y):
function clip_chk (line 122) | def clip_chk(x, lb, ub, allow_nan=False):
function snap_to (line 159) | def snap_to(x, fixed_val=None):
function linear_rescale (line 186) | def linear_rescale(X, lb0, ub0, lb1, ub1, enforce_bounds=True):
function argmin_2d (line 230) | def argmin_2d(X):
function cummin (line 238) | def cummin(x_val, x_key):
FILE: bayesmark/path_util.py
function abspath (line 20) | def abspath(path, verify=True): # pragma: io
function absopen (line 41) | def absopen(path, mode): # pragma: io
function _join_safe (line 61) | def _join_safe(*args): # pragma: io
function join_safe_r (line 76) | def join_safe_r(*args): # pragma: io
function join_safe_w (line 94) | def join_safe_w(*args): # pragma: io
FILE: bayesmark/quantiles.py
function ensure_shape (line 22) | def ensure_shape(x, y):
function order_stats (line 33) | def order_stats(X):
function _quantile (line 60) | def _quantile(n, q):
function quantile (line 65) | def quantile(X, q):
function _quantile_CI (line 97) | def _quantile_CI(n, q, alpha):
function quantile_CI (line 117) | def quantile_CI(X, q, alpha=0.05):
function max_quantile_CI (line 153) | def max_quantile_CI(X, q, m, alpha=0.05):
function min_quantile_CI (line 200) | def min_quantile_CI(X, q, m, alpha=0.05):
function quantile_and_CI (line 248) | def quantile_and_CI(X, q, alpha=0.05):
FILE: bayesmark/random_search.py
function suggest_dict (line 22) | def suggest_dict(X, y, meta, n_suggestions=1, random=np_util.random):
function _check_x_y (line 61) | def _check_x_y(X, y, allow_impute=False): # pragma: validator
function _check_bounds (line 87) | def _check_bounds(bounds, n_params): # pragma: validator
FILE: bayesmark/serialize.py
class Serializer (line 45) | class Serializer(ABC):
method init_db (line 51) | def init_db(db_root, keys, db=None, exist_ok=True):
method get_keys (line 74) | def get_keys(db_root, db):
method get_derived_keys (line 93) | def get_derived_keys(db_root, db):
method get_uuids (line 112) | def get_uuids(db_root, db, key):
method save (line 133) | def save(data, meta, db_root, db, key, uuid_):
method load (line 140) | def load(db_root, db, key, uuid_):
method save_derived (line 147) | def save_derived(data, meta, db_root, db, key):
method load_derived (line 154) | def load_derived(db_root, db, key):
class XRSerializer (line 160) | class XRSerializer(Serializer):
method init_db (line 164) | def init_db(db_root, keys, db=None, exist_ok=True): # pragma: io
method init_db_manual (line 183) | def init_db_manual(db_root, keys, db):
method get_keys (line 208) | def get_keys(db_root, db): # pragma: io
method get_derived_keys (line 216) | def get_derived_keys(db_root, db): # pragma: io
method get_uuids (line 223) | def get_uuids(db_root, db, key): # pragma: io
method save (line 230) | def save(data, meta, db_root, db, key, uuid_): # pragma: io
method load (line 255) | def load(db_root, db, key, uuid_): # pragma: io
method save_derived (line 284) | def save_derived(data, meta, db_root, db, key): # pragma: io
method load_derived (line 307) | def load_derived(db_root, db, key): # pragma: io
method logging_path (line 334) | def logging_path(db_root, db, uuid_): # pragma: io
method _fname_to_uuid (line 358) | def _fname_to_uuid(fname):
method _uuid_to_fname (line 362) | def _uuid_to_fname(uuid_):
method _key_to_fname (line 368) | def _key_to_fname(key):
method _fname_to_key (line 372) | def _fname_to_key(fname):
method _validate (line 376) | def _validate(db_root, keys=(), db=None):
function _dump_xr (line 387) | def _dump_xr(f, ds, meta): # pragma: io
function _load_xr (line 406) | def _load_xr(f): # pragma: io
FILE: bayesmark/signatures.py
function get_func_signature (line 28) | def get_func_signature(f, api_config):
function analyze_signatures (line 57) | def analyze_signatures(signatures):
function analyze_signature_pair (line 101) | def analyze_signature_pair(signatures, signatures_ref):
FILE: bayesmark/sklearn_funcs.py
class TestFunction (line 209) | class TestFunction(ABC):
method __init__ (line 213) | def __init__(self):
method evaluate (line 222) | def evaluate(self, params):
method get_api_config (line 226) | def get_api_config(self):
class SklearnModel (line 238) | class SklearnModel(TestFunction):
method __init__ (line 253) | def __init__(self, model, dataset, metric, shuffle_seed=0, data_root=N...
method evaluate (line 309) | def evaluate(self, params):
method test_case_str (line 359) | def test_case_str(model, dataset, scorer):
method inverse_test_case_str (line 365) | def inverse_test_case_str(test_case):
class SklearnSurrogate (line 372) | class SklearnSurrogate(TestFunction):
method __init__ (line 379) | def __init__(self, model, dataset, scorer, path):
method evaluate (line 417) | def evaluate(self, params):
FILE: bayesmark/space.py
function unravel_index (line 42) | def unravel_index(dims):
function encode (line 74) | def encode(X, labels, assume_sorted=False, dtype=bool, assume_valid=False):
function decode (line 113) | def decode(Y, labels, assume_sorted=False):
function _error (line 142) | def _error(msg, pre=False): # pragma: validator
function check_array (line 153) | def check_array(
function identity (line 247) | def identity(x):
function bilog (line 264) | def bilog(x):
function biexp (line 284) | def biexp(x):
class Space (line 309) | class Space(object):
method __init__ (line 313) | def __init__(self, dtype, default_round, warp="linear", values=None, r...
method validate (line 386) | def validate(self, X, pre=False):
method validate_warped (line 400) | def validate_warped(self, X, pre=False):
method warp (line 409) | def warp(self, X):
method unwarp (line 433) | def unwarp(self, X_w):
method get_bounds (line 457) | def get_bounds(self):
method grid (line 471) | def grid(self, max_interp=N_GRID_DEFAULT):
class Real (line 500) | class Real(Space):
method __init__ (line 504) | def __init__(self, warp="linear", values=None, range_=None):
class Integer (line 523) | class Integer(Space):
method __init__ (line 527) | def __init__(self, warp="linear", values=None, range_=None):
class Boolean (line 546) | class Boolean(Space):
method __init__ (line 550) | def __init__(self, warp=None, values=None, range_=None):
class Categorical (line 576) | class Categorical(Space):
method __init__ (line 580) | def __init__(self, warp=None, values=None, range_=None):
method _encode (line 612) | def _encode(self, x):
method _decode (line 615) | def _decode(self, x):
method warp (line 618) | def warp(self, X):
method unwarp (line 643) | def unwarp(self, X_w):
class JointSpace (line 676) | class JointSpace(object):
method __init__ (line 680) | def __init__(self, meta):
method validate (line 707) | def validate(self, X):
method warp (line 718) | def warp(self, X):
method unwarp (line 753) | def unwarp(self, X_w, fixed_vals={}):
method get_bounds (line 787) | def get_bounds(self):
method grid (line 801) | def grid(self, max_interp=N_GRID_DEFAULT):
FILE: bayesmark/stats.py
function robust_standardize (line 20) | def robust_standardize(X, q_level=0.5):
function t_EB (line 60) | def t_EB(x, alpha=0.05, axis=-1):
FILE: bayesmark/util.py
function in_or_none (line 19) | def in_or_none(x, L):
function all_unique (line 24) | def all_unique(L):
function strict_sorted (line 41) | def strict_sorted(L):
function range_str (line 59) | def range_str(stop):
function str_join_safe (line 83) | def str_join_safe(delim, str_vec, append=False):
function shell_join (line 129) | def shell_join(argv, delim=" "):
function chomp (line 152) | def chomp(str_val, ext="\n"):
function preimage_func (line 175) | def preimage_func(f, x):
FILE: bayesmark/xr_util.py
function is_simple_coords (line 25) | def is_simple_coords(coords, min_side=0, dims=None):
function ds_like (line 58) | def ds_like(ref, vars_, dims, fill=np.nan):
function ds_like_mixed (line 88) | def ds_like_mixed(ref, vars_, dims, fill=np.nan):
function only_dataarray (line 120) | def only_dataarray(ds):
function coord_compat (line 140) | def coord_compat(da_seq, dims):
function da_to_string (line 169) | def da_to_string(da):
function da_concat (line 187) | def da_concat(da_dict, dims):
function ds_concat (line 230) | def ds_concat(ds_dict, dims):
FILE: example_opt_root/flaky_optimizer.py
class FlakyOptimizer (line 9) | class FlakyOptimizer(AbstractOptimizer):
method __init__ (line 10) | def __init__(self, api_config, random=np_util.random):
method suggest (line 24) | def suggest(self, n_suggestions=1):
method observe (line 51) | def observe(self, X, y):
FILE: example_opt_root/hyperopt_optimizer.py
function dummy_f (line 15) | def dummy_f(x):
function only (line 19) | def only(x):
class HyperoptOptimizer (line 24) | class HyperoptOptimizer(AbstractOptimizer):
method __init__ (line 27) | def __init__(self, api_config, random=np_random):
method hashable_dict (line 49) | def hashable_dict(d):
method get_hyperopt_dimensions (line 66) | def get_hyperopt_dimensions(api_config):
method get_trial (line 120) | def get_trial(self, trial_id):
method cleanup_guess (line 130) | def cleanup_guess(self, x_guess):
method _suggest (line 145) | def _suggest(self):
method suggest (line 163) | def suggest(self, n_suggestions=1):
method observe (line 200) | def observe(self, X, y):
FILE: example_opt_root/nevergrad_optimizer.py
class NevergradOptimizer (line 12) | class NevergradOptimizer(AbstractOptimizer):
method __init__ (line 15) | def __init__(self, api_config, tool="OnePlusOne", budget=300):
method get_nvg_dimensions (line 34) | def get_nvg_dimensions(api_config):
method prewarp (line 90) | def prewarp(self, xx):
method postwarp (line 113) | def postwarp(self, xxw):
method suggest (line 134) | def suggest(self, n_suggestions=1):
method observe (line 159) | def observe(self, X, y):
FILE: example_opt_root/opentuner_optimizer.py
function ClippedParam (line 61) | def ClippedParam(cls, epsilon=1e-5):
class OpentunerOptimizer (line 94) | class OpentunerOptimizer(AbstractOptimizer):
method __init__ (line 97) | def __init__(self, api_config, techniques=DEFAULT_TECHNIQUES, n_sugges...
method hashable_dict (line 164) | def hashable_dict(d):
method build_manipulator (line 181) | def build_manipulator(api_config):
method suggest (line 229) | def suggest(self, n_suggestions=1):
method observe (line 293) | def observe(self, X, y):
FILE: example_opt_root/pysot_optimizer.py
class PySOTOptimizer (line 16) | class PySOTOptimizer(AbstractOptimizer):
method __init__ (line 19) | def __init__(self, api_config):
method create_opt_prob (line 37) | def create_opt_prob(self):
method start (line 49) | def start(self, max_evals):
method suggest (line 79) | def suggest(self, n_suggestions=1):
method _observe (line 129) | def _observe(self, x, y):
method observe (line 138) | def observe(self, X, y):
FILE: example_opt_root/random_optimizer.py
class RandomOptimizer (line 7) | class RandomOptimizer(AbstractOptimizer):
method __init__ (line 11) | def __init__(self, api_config, random=np_util.random):
method suggest (line 24) | def suggest(self, n_suggestions=1):
method observe (line 42) | def observe(self, X, y):
FILE: example_opt_root/scikit_optimizer.py
class ScikitOptimizer (line 10) | class ScikitOptimizer(AbstractOptimizer):
method __init__ (line 13) | def __init__(self, api_config, base_estimator="GP", acq_func="gp_hedge...
method get_sk_dimensions (line 49) | def get_sk_dimensions(api_config, transform="normalize"):
method suggest (line 100) | def suggest(self, n_suggestions=1):
method observe (line 127) | def observe(self, X, y):
FILE: setup.py
function read_requirements (line 23) | def read_requirements(name):
FILE: test/data_test.py
function test_get_problem_type (line 23) | def test_get_problem_type(dataset_name):
FILE: test/expected_max_test.py
function test_get_expected_max_weights (line 21) | def test_get_expected_max_weights(n, m):
function test_expected_max (line 27) | def test_expected_max(x, m):
function test_expected_min (line 33) | def test_expected_min(x, m):
FILE: test/experiment_aggregate_test.py
function data_to_concat (line 37) | def data_to_concat():
function time_datasets (line 94) | def time_datasets():
function perf_dataarrays (line 102) | def perf_dataarrays():
function test_summarize_time (line 110) | def test_summarize_time(all_time):
function test_concat_experiments (line 117) | def test_concat_experiments(all_experiments):
FILE: test/experiment_analysis_test.py
function test_get_perf_array (line 25) | def test_get_perf_array(args):
function test_compute_aggregates (line 42) | def test_compute_aggregates(perf_da):
function test_compute_aggregates_with_aux (line 61) | def test_compute_aggregates_with_aux(perf_da):
FILE: test/experiment_baseline_test.py
function test_compute_baseline (line 24) | def test_compute_baseline(perf_da):
FILE: test/experiment_db_init_test.py
function test_main (line 17) | def test_main():
FILE: test/experiment_launcher_test.py
function filepaths (line 44) | def filepaths():
function filenames (line 57) | def filenames(suffix=""):
function joinables (line 70) | def joinables():
function datasets (line 75) | def datasets():
function launcher_args (line 79) | def launcher_args(opts, min_jobs=0):
function launcher_args_and_config (line 101) | def launcher_args_and_config(min_jobs=0):
function test_is_arg_safe_empty (line 114) | def test_is_arg_safe_empty():
function test_gen_commands (line 122) | def test_gen_commands(args, run_uuid):
function test_dry_run (line 144) | def test_dry_run(args, run_uuid, seed):
FILE: test/experiment_test.py
class RandomOptimizer (line 34) | class RandomOptimizer(AbstractOptimizer):
method __init__ (line 38) | def __init__(self, api_config, random=np_util.random, flaky=False):
method suggest (line 43) | def suggest(self, n_suggestions=1):
method observe (line 49) | def observe(self, X, y):
class OutOfBoundsOptimizer (line 55) | class OutOfBoundsOptimizer(AbstractOptimizer):
method __init__ (line 56) | def __init__(self, api_config, random=np_util.random):
method suggest (line 61) | def suggest(self, n_suggestions=1):
method observe (line 78) | def observe(self, X, y):
class FlakyProblem (line 82) | class FlakyProblem(TestFunction):
method __init__ (line 83) | def __init__(self, api_config, random):
method evaluate (line 88) | def evaluate(self, params):
function test_run_study (line 102) | def test_run_study(model_name, dataset, scorer, n_calls, n_suggestions, ...
function test_run_study_bounds_fail (line 120) | def test_run_study_bounds_fail(model_name, dataset, scorer, n_calls, n_s...
function test_run_study_callback (line 148) | def test_run_study_callback(model_name, dataset, scorer, n_calls, n_sugg...
function test_run_study_flaky (line 184) | def test_run_study_flaky(api_config, n_calls, n_suggestions, seed1, seed2):
function test_run_sklearn_study (line 203) | def test_run_sklearn_study(api_config, model_name, dataset, scorer, n_ca...
function test_run_sklearn_study_real (line 220) | def test_run_sklearn_study_real(api_config, model_name, dataset, scorer,...
function test_get_objective_signature (line 235) | def test_get_objective_signature(model_name, dataset, scorer):
function test_build_eval_ds (line 243) | def test_build_eval_ds(args):
function test_build_timing_ds (line 249) | def test_build_timing_ds(args):
function test_build_suggest_ds (line 261) | def test_build_suggest_ds(suggest_ds):
function test_get_opt_class_module (line 278) | def test_get_opt_class_module():
FILE: test/hypothesis_util.py
function identity (line 20) | def identity(x):
function seeds (line 25) | def seeds():
function probs (line 29) | def probs():
function mfloats (line 33) | def mfloats():
function gufunc_floats (line 37) | def gufunc_floats(signature, min_side=0, max_side=5, unique=False, **kwa...
function close_enough (line 43) | def close_enough(x, y, equal_nan=False, rtol=1e-5, atol=1e-8):
function broadcasted (line 61) | def broadcasted(
function broadcast_tester (line 179) | def broadcast_tester(
function multi_broadcast_tester (line 230) | def multi_broadcast_tester(
FILE: test/np_util_test.py
function test_random_seed (line 23) | def test_random_seed(seed):
function test_shuffle_2d (line 29) | def test_shuffle_2d(X, seed):
function test_strat_split (line 35) | def test_strat_split(X, n_splits, seed):
function test_isclose_lte_pass (line 46) | def test_isclose_lte_pass(args):
function test_isclose_lte_fail (line 53) | def test_isclose_lte_fail(args):
function test_isclose_broadcast (line 65) | def test_isclose_broadcast():
function test_clip_chk_pass (line 70) | def test_clip_chk_pass(args):
function test_clip_chk_pass_nan (line 81) | def test_clip_chk_pass_nan(args):
function test_snap_to_pass (line 92) | def test_snap_to_pass(args):
function test_linear_rescale_bounds (line 101) | def test_linear_rescale_bounds(args):
function test_linear_rescale_inner (line 119) | def test_linear_rescale_inner(args):
function test_linear_rescale_inverse (line 138) | def test_linear_rescale_inverse(args):
function test_linear_rescale_bound_modes (line 161) | def test_linear_rescale_bound_modes(args):
function pair_sort (line 179) | def pair_sort(X, Y):
function test_linear_rescale_broadcast (line 187) | def test_linear_rescale_broadcast():
function test_isclose (line 217) | def test_isclose(x, y):
function test_isclose_2 (line 251) | def test_isclose_2():
function test_argmin_2d_no_nan (line 261) | def test_argmin_2d_no_nan(args):
function test_argmin_2d_nan (line 269) | def test_argmin_2d_nan(args):
function test_cummin (line 277) | def test_cummin(args):
function test_cummin_nan (line 294) | def test_cummin_nan(args):
FILE: test/quantiles_test.py
function order_stats_trim (line 35) | def order_stats_trim(X):
function test_order_stats (line 50) | def test_order_stats(args):
function test_quantile (line 69) | def test_quantile(args):
function test_quantile_to_np (line 84) | def test_quantile_to_np(args):
function test_quantile_CI (line 98) | def test_quantile_CI(args):
function test_quantile_CI_monotone_x (line 116) | def test_quantile_CI_monotone_x(args):
function test_quantile_CI_monotone_q (line 137) | def test_quantile_CI_monotone_q(args):
function test_quantile_CI_monotone_alpha (line 150) | def test_quantile_CI_monotone_alpha(args):
function test_max_quantile_CI (line 167) | def test_max_quantile_CI(args):
function test_min_quantile_CI (line 189) | def test_min_quantile_CI(args):
function test_min_quantile_CI_to_max (line 211) | def test_min_quantile_CI_to_max(args):
function test_quantile_and_CI (line 234) | def test_quantile_and_CI(args):
function test_order_stats_broadcast (line 250) | def test_order_stats_broadcast():
function test_quantile_broadcast_0 (line 254) | def test_quantile_broadcast_0():
function test_quantile_broadcast_1 (line 260) | def test_quantile_broadcast_1():
function test_quantile_CI_broadcast_0 (line 266) | def test_quantile_CI_broadcast_0():
function test_quantile_CI_broadcast_1 (line 277) | def test_quantile_CI_broadcast_1():
function test_max_quantile_CI_broadcast_0 (line 288) | def test_max_quantile_CI_broadcast_0():
function test_max_quantile_CI_broadcast_1 (line 299) | def test_max_quantile_CI_broadcast_1():
function test_min_quantile_CI_broadcast_0 (line 310) | def test_min_quantile_CI_broadcast_0():
function test_min_quantile_CI_broadcast_1 (line 321) | def test_min_quantile_CI_broadcast_1():
function test_quantile_and_CI_broadcast_0 (line 332) | def test_quantile_and_CI_broadcast_0():
function test_quantile_and_CI_broadcast_1 (line 343) | def test_quantile_and_CI_broadcast_1():
function mc_test_quantile_CI (line 354) | def mc_test_quantile_CI(mc_runs=1000, n=2000, q=0.5, alpha=0.05, random=...
function mc_test_max_quantile_CI (line 367) | def mc_test_max_quantile_CI(mc_runs=1000, n=2000, q=0.5, m=100, alpha=0....
function mc_test_min_quantile_CI (line 381) | def mc_test_min_quantile_CI(mc_runs=1000, n=2000, q=0.5, m=100, alpha=0....
function test_all_mc (line 395) | def test_all_mc():
FILE: test/random_search_test.py
function test_random_search_suggest_sanity (line 27) | def test_random_search_suggest_sanity(api_args, n_suggest, seed):
function test_random_search_suggest_diff (line 68) | def test_random_search_suggest_diff(api_args, n_suggest, seed):
FILE: test/serialize_test.py
function filepaths (line 24) | def filepaths():
function filenames (line 37) | def filenames(suffix=""):
function test_init_db_manual (line 51) | def test_init_db_manual(db_root, keys, db):
function test_uuid_to_fname (line 56) | def test_uuid_to_fname(uu):
function test_key_to_fname (line 66) | def test_key_to_fname(key):
function test_validate (line 73) | def test_validate(db_root, keys, db):
FILE: test/signatures_test.py
function bsigs (line 29) | def bsigs():
function sigs (line 34) | def sigs():
function sig_pair (line 39) | def sig_pair():
function some_mock_f (line 59) | def some_mock_f(x):
function test_get_func_signature (line 68) | def test_get_func_signature(api_config):
function test_analyze_signatures (line 75) | def test_analyze_signatures(signatures):
function test_analyze_signature_pair (line 82) | def test_analyze_signature_pair(args):
FILE: test/sklearn_funcs_test.py
function test_sklearn_model (line 31) | def test_sklearn_model(model, dataset, metric, shuffle_seed, rs_seed):
function test_inverse_test_case_str (line 48) | def test_inverse_test_case_str(model, dataset, scorer):
function test_sklearn_model_surr (line 59) | def test_sklearn_model_surr(model, dataset, metric, model_seed, rs_seed):
FILE: test/space_test.py
function encoder_gen (line 35) | def encoder_gen(args):
function decoder_gen (line 45) | def decoder_gen(args):
function decoder_gen_broadcast (line 54) | def decoder_gen_broadcast(args):
function test_encode_decode (line 77) | def test_encode_decode(args):
function test_encoder_to_sklearn (line 106) | def test_encoder_to_sklearn(args):
function test_decode_encode (line 128) | def test_decode_encode(args):
function test_decode_to_sklearn (line 151) | def test_decode_to_sklearn(args):
function test_encode_broadcast_bool (line 166) | def test_encode_broadcast_bool():
function test_encode_broadcast_int (line 180) | def test_encode_broadcast_int():
function test_encode_broadcast_float (line 194) | def test_encode_broadcast_float():
function test_decode_broadcast_bool (line 208) | def test_decode_broadcast_bool():
function test_decode_broadcast_int (line 222) | def test_decode_broadcast_int():
function test_decode_broadcast_float (line 236) | def test_decode_broadcast_float():
function test_bilog_props (line 251) | def test_bilog_props(args):
function test_bilog_monotonic (line 262) | def test_bilog_monotonic(args):
function test_bilog_biexp (line 271) | def test_bilog_biexp(args):
function test_bilog_broadcast (line 277) | def test_bilog_broadcast():
function test_biexp_broadcast (line 281) | def test_biexp_broadcast():
function test_real_values_warp_unwarp (line 286) | def test_real_values_warp_unwarp(warp, args):
function test_real_range_warp_unwarp (line 325) | def test_real_range_warp_unwarp(warp, args):
function test_real_range_unwarp_warp (line 363) | def test_real_range_unwarp_warp(warp, args):
function test_int_values_warp_unwarp (line 404) | def test_int_values_warp_unwarp(warp, args):
function test_log_int_values_warp_unwarp (line 435) | def test_log_int_values_warp_unwarp(args):
function test_int_range_warp_unwarp (line 468) | def test_int_range_warp_unwarp(warp, args):
function test_bool_warp_unwarp (line 511) | def test_bool_warp_unwarp(args):
function test_cat_warp_unwarp (line 544) | def test_cat_warp_unwarp(args):
function test_joint_space_unwarp_warp (line 574) | def test_joint_space_unwarp_warp(args):
function test_joint_space_warp_missing (line 595) | def test_joint_space_warp_missing(args):
function test_joint_space_warp_fixed_vars (line 623) | def test_joint_space_warp_fixed_vars(args):
function test_joint_grid (line 651) | def test_joint_grid(args, max_interp):
function test_unravel_index_empty (line 694) | def test_unravel_index_empty():
function test_unravel_index_empty_2 (line 699) | def test_unravel_index_empty_2(dims):
FILE: test/stats_test.py
function t_test_ (line 25) | def t_test_(x):
function test_robust_standardize_to_sklearn (line 56) | def test_robust_standardize_to_sklearn(args):
function test_robust_standardize_broadcast (line 71) | def test_robust_standardize_broadcast():
function test_t_EB_zero_var (line 93) | def test_t_EB_zero_var(N, val, alpha):
function test_t_EB_inf (line 103) | def test_t_EB_inf(N, val, alpha):
function test_t_EB_coverage (line 115) | def test_t_EB_coverage(seed, alpha, N):
function test_t_test_to_EB (line 135) | def test_t_test_to_EB(x):
FILE: test/util.py
function _easy_text (line 39) | def _easy_text():
function _hashable (line 45) | def _hashable():
function space_vars (line 60) | def space_vars(draw, max_values=5):
function space_configs (line 105) | def space_configs(draw, max_vars=5, max_len=5, allow_missing=False, uniq...
function perf_dataarrays (line 138) | def perf_dataarrays(min_trial=1):
FILE: test/util_test.py
function some_mock_f (line 24) | def some_mock_f(x):
function test_in_or_none (line 33) | def test_in_or_none(x, L):
function test_in_or_none_on_none (line 41) | def test_in_or_none_on_none(x):
function test_in_or_none_self (line 48) | def test_in_or_none_self(L):
function test_all_unique (line 56) | def test_all_unique(L):
function test_strict_sorted (line 61) | def test_strict_sorted(L):
function test_range_str (line 66) | def test_range_str(stop):
function test_str_join_safe (line 71) | def test_str_join_safe(delim, str_vec):
function test_str_join_safe_append (line 77) | def test_str_join_safe_append(delim, str_vec0, str_vec):
function test_shell_join (line 86) | def test_shell_join(argv):
function test_chomp (line 93) | def test_chomp(str_val, ext):
function test_preimage_func (line 98) | def test_preimage_func(x):
FILE: test/xr_util_test.py
function intersect_seq (line 36) | def intersect_seq(L):
function ds_vars_dims (line 46) | def ds_vars_dims():
function ds_vars_dims_mixed (line 61) | def ds_vars_dims_mixed():
function test_is_simple_coords (line 82) | def test_is_simple_coords(da, min_side):
function test_is_simple_coords_pass (line 87) | def test_is_simple_coords_pass(da):
function test_ds_like (line 93) | def test_ds_like(args, fill):
function test_ds_like_mixed (line 100) | def test_ds_like_mixed(args, fill):
function test_only_dataarray (line 107) | def test_only_dataarray(var_, da):
function test_coord_compat (line 116) | def test_coord_compat(ds):
function test_coord_compat_false (line 126) | def test_coord_compat_false(ds):
function test_da_to_string (line 142) | def test_da_to_string(da):
function test_da_concat (line 148) | def test_da_concat(da, n):
function da_split (line 158) | def da_split(da, n):
function test_ds_concat (line 173) | def test_ds_concat(ds, n):
Condensed preview — 102 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (565K chars).
[
{
"path": ".coveragerc",
"chars": 190,
"preview": "[report]\nexclude_lines =\n pragma: no cover\n @abstract\n ValueError\n NotImplementedError\n assert\n _error"
},
{
"path": ".gitignore",
"chars": 507,
"preview": ".*\n!.gitignore\n!.gitmodules\n!.flake8\n!.coveragerc\n!.pre-commit-config.yaml\n!.secrets.baseline\n!.travis.yml\n!.readthedocs"
},
{
"path": ".pre-commit-config.yaml",
"chars": 2146,
"preview": "- repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v1.2.3\n hooks:\n - id: flake8\n exclude: "
},
{
"path": ".readthedocs.yml",
"chars": 569,
"preview": "# .readthedocs.yml\n# Read the Docs configuration file\n# See https://docs.readthedocs.io/en/stable/config-file/v2.html fo"
},
{
"path": ".secrets.baseline",
"chars": 638,
"preview": "{\n \"exclude\": {\n \"files\": null,\n \"lines\": null\n },\n \"generated_at\": \"2019-09-18T01:04:54Z\",\n \"plugins_used\": ["
},
{
"path": ".travis.yml",
"chars": 360,
"preview": "language: python\npython:\n - \"3.6\"\n\nbefore_script:\n - \"curl -H 'Cache-Control: no-cache' https://raw.githubuserconten"
},
{
"path": "LICENSE",
"chars": 11358,
"preview": "\n Apache License\n Version 2.0, January 2004\n "
},
{
"path": "MANIFEST.in",
"chars": 129,
"preview": "include requirements/base.in\ninclude requirements/optimizers.in\ninclude requirements/ipynb.in\ninclude LICENSE\ninclude RE"
},
{
"path": "README.rst",
"chars": 26815,
"preview": "Installation\n============\n\nThis project provides a benchmark framework to easily compare Bayesian optimization methods o"
},
{
"path": "bayesmark/__init__.py",
"chars": 75,
"preview": "__version__ = \"0.0.8\"\n__author__ = \"Ryan Turner\"\n__license__ = \"Apache v2\"\n"
},
{
"path": "bayesmark/abstract_optimizer.py",
"chars": 2975,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "bayesmark/builtin_opt/config.py",
"chars": 1610,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/hyperopt_optimizer.py",
"chars": 9347,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/nevergrad_optimizer.py",
"chars": 7074,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/opentuner_optimizer.py",
"chars": 13464,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/pysot_optimizer.py",
"chars": 6119,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/random_optimizer.py",
"chars": 2637,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/builtin_opt/scikit_optimizer.py",
"chars": 7087,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/cmd_parse.py",
"chars": 13533,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/constants.py",
"chars": 2397,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/data.py",
"chars": 6468,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/expected_max.py",
"chars": 3874,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/experiment.py",
"chars": 27515,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/experiment_aggregate.py",
"chars": 12481,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/experiment_analysis.py",
"chars": 15994,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/experiment_baseline.py",
"chars": 8386,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/experiment_db_init.py",
"chars": 1558,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/experiment_launcher.py",
"chars": 13083,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/np_util.py",
"chars": 8714,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/path_util.py",
"chars": 3586,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/quantiles.py",
"chars": 10010,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/random_search.py",
"chars": 3734,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/serialize.py",
"chars": 13860,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/signatures.py",
"chars": 4987,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/sklearn_funcs.py",
"chars": 18214,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/space.py",
"chars": 32699,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/stats.py",
"chars": 3325,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/util.py",
"chars": 5297,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "bayesmark/xr_util.py",
"chars": 9311,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "build_wheel.sh",
"chars": 452,
"preview": "#!/bin/bash\n\nset -ex\nset -o pipefail\n\n# Display what version is being used for logging\npython --version\n\n# Fail if untra"
},
{
"path": "docs/.gitignore",
"chars": 7,
"preview": "_build\n"
},
{
"path": "docs/Makefile",
"chars": 6161,
"preview": "# Makefile for Sphinx documentation\n#\n\n# You can set these variables from the command line.\nSPHINXOPTS =\nSPHINXBUILD "
},
{
"path": "docs/authors.rst",
"chars": 169,
"preview": "-------\nCredits\n-------\n\n~~~~~~~~~~~~~~~~\nDevelopment lead\n~~~~~~~~~~~~~~~~\n\nRyan Turner (rdturnermtl)\n\n~~~~~~~~~~~~\nCon"
},
{
"path": "docs/code.rst",
"chars": 2284,
"preview": "-------------\nCode Overview\n-------------\n\n.. _bayesmark:\n\n~~~~\nData\n~~~~\n\n.. automodule:: bayesmark.data\n :members:\n "
},
{
"path": "docs/conf.py",
"chars": 8895,
"preview": "#!/usr/bin/env python\n# -*- coding: utf-8 -*-\n#\n# bayesmark documentation build configuration file.\n#\n# This file is exe"
},
{
"path": "docs/dummy.py",
"chars": 745,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "docs/index.rst",
"chars": 400,
"preview": ".. bayesmark documentation master file, created by\n sphinx-quickstart on Tue Jul 9 22:26:36 2013.\n You can adapt th"
},
{
"path": "docs/readme.rst",
"chars": 27,
"preview": ".. include:: ../README.rst\n"
},
{
"path": "docs/scoring.rst",
"chars": 7579,
"preview": ".. _how-scoring-works:\n\nHow scoring works\n=================\n\nThe scoring system is about aggregating the function evalua"
},
{
"path": "example_opt_root/config.json",
"chars": 1509,
"preview": "{\n \"Flaky\": [\n \"flaky_optimizer.py\",\n {}\n ],\n \"HyperOpt-New\": [\n \"hyperopt_optimizer.py\",\n"
},
{
"path": "example_opt_root/flaky_optimizer.py",
"chars": 2365,
"preview": "from time import sleep\n\nimport bayesmark.random_search as rs\nfrom bayesmark import np_util\nfrom bayesmark.abstract_optim"
},
{
"path": "example_opt_root/hyperopt_optimizer.py",
"chars": 8839,
"preview": "import numpy as np\nfrom hyperopt import hp, tpe\nfrom hyperopt.base import JOB_STATE_DONE, JOB_STATE_NEW, STATUS_OK, Doma"
},
{
"path": "example_opt_root/nevergrad_optimizer.py",
"chars": 6583,
"preview": "import nevergrad.optimization as optimization\nimport numpy as np\nfrom nevergrad import instrumentation as inst\nfrom scip"
},
{
"path": "example_opt_root/opentuner_optimizer.py",
"chars": 12956,
"preview": "\"\"\"\nIn opentuner, many search techniques are already available. All the names of\nthe techniques can be found as follows:"
},
{
"path": "example_opt_root/pysot_optimizer.py",
"chars": 5611,
"preview": "import warnings\nfrom copy import copy\n\nimport numpy as np\nfrom poap.strategy import EvalRecord\nfrom pySOT.experimental_d"
},
{
"path": "example_opt_root/random_optimizer.py",
"chars": 1950,
"preview": "import bayesmark.random_search as rs\nfrom bayesmark import np_util\nfrom bayesmark.abstract_optimizer import AbstractOpti"
},
{
"path": "example_opt_root/scikit_optimizer.py",
"chars": 6431,
"preview": "import numpy as np\nfrom scipy.interpolate import interp1d\nfrom skopt import Optimizer as SkOpt\nfrom skopt.space import C"
},
{
"path": "integration_test.sh",
"chars": 1525,
"preview": "#!/bin/bash\n\nset -ex\nset -o pipefail\n\n# Be able to check if using version out of tar ball\nwhich bayesmark-launch\nwhich b"
},
{
"path": "integration_test_with_setup.sh",
"chars": 968,
"preview": "#!/bin/bash\n\nset -ex\nset -o pipefail\n\nexport PIP_REQUIRE_VIRTUALENV=false\n\n# Handy to know what we are working with\ngit "
},
{
"path": "notebooks/dummy.py",
"chars": 836,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "notebooks/plot_mean_score.ipynb",
"chars": 9681,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": "
},
{
"path": "notebooks/plot_test_case.ipynb",
"chars": 4963,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {},\n \"outputs\": [],\n \"source\": "
},
{
"path": "requirements/base.in",
"chars": 142,
"preview": "scipy==1.2.0\npandas==0.24.0\npathvalidate==0.29.0\nnumpy==1.16.1\nGitPython==2.1.11\nimportlib-metadata==0.18\nscikit-learn=="
},
{
"path": "requirements/base.txt",
"chars": 1001,
"preview": "# SHA1:7ebe4df9e60f001b676e74ae561d5dc3202c3dd0\n#\n# This file is autogenerated by pip-compile-multi\n# To update, run:\n#\n"
},
{
"path": "requirements/docs.in",
"chars": 25,
"preview": "-r base.in\nSphinx==2.1.2\n"
},
{
"path": "requirements/docs.txt",
"chars": 1209,
"preview": "# SHA1:cde26afc07f6c9c1c6cb169e125fc5142a0c59ae\n#\n# This file is autogenerated by pip-compile-multi\n# To update, run:\n#\n"
},
{
"path": "requirements/ipynb.in",
"chars": 112,
"preview": "-r base.in\nipykernel==5.1.1\nnbconvert==5.6.0\njupyter==1.0.0\njupyter-core==4.6.0\nmatplotlib==3.1.1\nnumpy==1.16.1\n"
},
{
"path": "requirements/ipynb.txt",
"chars": 2735,
"preview": "# SHA1:6c16d140e48d7e7fa0e157c053953db7d76f0caf\n#\n# This file is autogenerated by pip-compile-multi\n# To update, run:\n#\n"
},
{
"path": "requirements/optimizers.in",
"chars": 137,
"preview": "-r base.in\nopentuner==0.8.2\nnumpy==1.16.1\nscipy==1.2.0\nnevergrad==0.1.4\nhyperopt==0.1.1\nPOAP==0.1.26\nscikit-optimize==0."
},
{
"path": "requirements/optimizers.txt",
"chars": 2272,
"preview": "# SHA1:08174a35f9973427450f549131b4438e2f116a88\n#\n# This file is autogenerated by pip-compile-multi\n# To update, run:\n#\n"
},
{
"path": "requirements/pipreqs_edits.sed",
"chars": 47,
"preview": "/argparse/d\n/appnope/d\n/certifi/d\n/bayesmark/d\n"
},
{
"path": "requirements/self.txt",
"chars": 17,
"preview": "bayesmark==0.0.8\n"
},
{
"path": "requirements/test.in",
"chars": 191,
"preview": "-r base.in\n-r optimizers.in\nhypothesis==4.32.3\nhypothesis-gufunc==0.0.5rc2\nnumpy==1.16.1\npathvalidate==0.29.0\nscipy==1.2"
},
{
"path": "requirements/test.txt",
"chars": 456,
"preview": "# SHA1:0dd8b5c26e6671e320706ddd399f6f62e19f3189\n#\n# This file is autogenerated by pip-compile-multi\n# To update, run:\n#\n"
},
{
"path": "requirements/tools.in",
"chars": 130,
"preview": "detect-secrets==0.12.5\nipykernel==5.1.1\nnbconvert==5.6.0\npip-compile-multi==1.4.0\npipreqs==0.4.9\npre-commit==1.15.2\npyte"
},
{
"path": "requirements/tools.txt",
"chars": 3488,
"preview": "# SHA1:08f4ed4790290aab315dd20169793be4f0a974af\n#\n# This file is autogenerated by pip-compile-multi\n# To update, run:\n#\n"
},
{
"path": "setup.py",
"chars": 2559,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/data_test.py",
"chars": 977,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/dummy.py",
"chars": 774,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/expected_max_test.py",
"chars": 1148,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/experiment_aggregate_test.py",
"chars": 4661,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/experiment_analysis_test.py",
"chars": 3057,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/experiment_baseline_test.py",
"chars": 1006,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/experiment_db_init_test.py",
"chars": 745,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/experiment_launcher_test.py",
"chars": 5582,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/experiment_test.py",
"chars": 10215,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/hypothesis_util.py",
"chars": 9499,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/np_util_test.py",
"chars": 8789,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/quantiles_test.py",
"chars": 11880,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/random_search_test.py",
"chars": 4085,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/serialize_test.py",
"chars": 2122,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/signatures_test.py",
"chars": 2650,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/sklearn_funcs_test.py",
"chars": 3371,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/space_test.py",
"chars": 19819,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/stats_test.py",
"chars": 4367,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/util.py",
"chars": 5834,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/util_test.py",
"chars": 2702,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test/xr_util_test.py",
"chars": 4978,
"preview": "# Copyright (c) 2019 Uber Technologies, Inc.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you m"
},
{
"path": "test.sh",
"chars": 6831,
"preview": "#!/bin/bash\n\nset -ex\nset -o pipefail\n\n# Set conda paths\nexport CONDA_PATH=./tmp/conda\nexport CONDA_ENVS=env\n\n# Sometime "
},
{
"path": "tools/archive_branch.sh",
"chars": 612,
"preview": "#!/bin/bash\n\nset -ex\nset -o pipefail\n\nDATE=$(date +\"%Y%m%d\")\nTAGNAME=archive/$DATE-$1\n\n# Fail if untracked files\ntest -z"
},
{
"path": "tools/deploy.sh",
"chars": 4963,
"preview": "#!/bin/bash\n#\n# Note that\n# UUID=$(uuidgen)\n# works on Mac OS by default, but requires installation on linux.\n\nset -ex\ns"
}
]
About this extraction
This page contains the full source code of the uber/bayesmark GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 102 files (525.6 KB), approximately 138.2k tokens, and a symbol index with 465 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.