Showing preview only (1,068K chars total). Download the full file or copy to clipboard to get everything.
Repository: imbue-ai/carbs
Branch: main
Commit: 8f7fee08658a
Files: 15
Total size: 1.0 MB
Directory structure:
gitextract_wm5wtx1e/
├── .gitignore
├── LICENSE.txt
├── README.md
├── carbs/
│ ├── __init__.py
│ ├── carbs.py
│ ├── model.py
│ ├── serialization.py
│ ├── test_carbs.py
│ └── utils.py
├── excluded.txt
├── notebooks/
│ ├── analyze_carbs.sync.ipynb
│ ├── carbs_demo.ipynb
│ └── carbs_simple_2d.sync.ipynb
├── pyproject.toml
└── setup.py
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
.python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
# poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
.pdm.toml
.pdm-python
.pdm-build/
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
================================================
FILE: LICENSE.txt
================================================
MIT License
Copyright (c) 2024 Imbue
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
# Cost Aware pareto-Region Bayesian Search
CARBS is a hyperparameter optimizer that can optimize both regular hyperparameters (like learning rate) and cost-related hyperparameters (like the number of epochs over data). It is a local search algorithm, so it benefits significantly from a good starting point. It searches around the pareto frontier of cost and performance, making it effective in finding compute efficient solutions to problems. See more in [our paper](https://arxiv.org/abs/2306.08055) or the [related blog post](https://imbue.com/research/carbs/). We have [used CARBS extensively](https://imbue.com/research/70b-carbs/) in training and scaling up large language models.
## Installing
CARBS depends primarily on pytorch and pyro for the Gaussian Process model. To get started, clone this directory and run,
```bash
pip install -e /path/to/carbs
```
## Using CARBS
The primary CARBS interface is through `suggest` (which will return a new point to test) and `observe` (to report the result).
Here is the core part of calling CARBS, (for the full example see `notebooks/carbs_demo.ipynb`):
```python
param_spaces = [
Param(name="learning_rate", space=LogSpace(scale=0.5), search_center=1e-4),
Param(name="momentum", space=LogitSpace(), search_center=0.9),
Param(name="epochs", space=LogSpace(is_integer=True, min=2, max=512), search_center=10),
]
carbs_params = CARBSParams(
better_direction_sign=-1,
is_wandb_logging_enabled=False,
resample_frequency=0,
)
carbs = CARBS(carbs_params, param_spaces)
for i in range(10):
suggestion = carbs.suggest().suggestion
observed_value = run_test_fn(suggestion)
obs_out = carbs.observe(ObservationInParam(input=suggestion, output=observed_value, cost=suggestion["epochs"]))
```
By default, suggestions will be remembered and included in the GP model using Thompson sampling, to avoid suggesting the same point repeatedly if experiments are being done in parallel. Use `suggest(is_suggestion_remembered=False)` to disable this behavior.
### Configuration
Options for CARBS are described on the `CARBSParams` class in `carbs/utils.py`.
On the configuration class `CARBSParams`, be sure to set:
* `better_direction_sign` to 1 to find a maximum or -1 to find a minimum.
* `wandb_params` there to configure wandb logging.
Optionally also set:
* `max_suggestion_cost` (defaults to None) which is a soft restriction on the maximum cost of a suggestion made by the algorithm. This uses the GP model of the cost, so it may not be completely accurate early on in an experiment.
* `num_random_samples` (defaults to 4) will be the number of observations seen before CARBS starts make its own suggestions
* `is_saved_on_every_observation` (defaults to True) will pickle and save the whole class to the wandb run on each observation
* `resample_frequency` (defaults to 5) will be the frequency at which CARBS resamples points on the pareto front (starting from the lowest cost point). Set to 0 to disable resampling
* `min_pareto_cost_fraction` (defaults to 0.2) has the effect of bucketing together the lowest cost observations -- in the default case 20% -- As these observations are typically much noisier and less interesting than the high cost observations. Set to 0.0 to disable.
* `initial_search_radius` (defaults to 0.3) will change the scale over all search variables. We've found 0.3 to be fairly good across different types of problems.
### Search space
CARBS only supports continuous and integer search spaces. The spaces do not need to have bounds, but `min` and `max` values may be specified. The three main types are:
* `LinearSpace`: Be sure to set a `scale` parameter to describe a relevant scale length for the problem. If you are using an integer space and the default radius of 0.3, you will need to choose a scale >3 to ensure that neighboring integers can be reached.
* `LogSpace`: Good for cost or scale related variables, as well as other typically log distributed continuous variables. If using an integer space, you will need to set a minimum value.
* `LogitSpace`: This will only output values between 0 to 1, so cannot be used with integer values.
## Concepts
Here are some concepts to be familiar with when using CARBS:
### Cost
We usually use number of seconds of runtime as cost.
It is recommended to start out the search in a low cost region, so the algorithm can get many iterations in quickly. If increasing the cost will increase the performance (as it usually does), CARBS will explore the higher cost area later.
The `max_suggestion_cost` argument to `CARBSParams` is roughly used to cap the cost of suggestions. CARBS will not make any suggestions that it thinks will cost more than `max_suggestion_cost`. Because its cost model is not completely accurate, some suggestions will take longer than this time. They will not be truncated at the `max_suggestion_cost` amount of runtime.
### Success / Failure
CARBS keeps a separate model for whether a run will succeed or fail. Usually, we report a success if we are able to measure the target metric during eval at the end of training. A run should be reported as a failure if the hyperparameters suggested by CARBS caused the failure, for example a batch size that is too large that caused an OOM failure. If a failure occurs that is not related to the hyperparameters, it is better to forget the suggestion or retry it. Report a failure by making an `ObservationInParam` with `is_failure=True`
### Basic / Param Space
We map parameter spaces into a more natural search space internally to CARBS for modeling purposes. We call the raw parameter space, used for input and output, **Parameter Space**. We map that to a **Basic Space** using the parameter type, so a `LogSpace` will be transformed by the `log`/`exp` functions. We also use the `scale` factor in this transformation.
### Integer spaces and rounding
Log and Linear spaces can take the flag `is_integer=True` and a `rounding_factor` to round to a nearest value (eg, to round to the nearest multiple of 8). One potential gotcha here is that if the `search_radius` (which defaults to 0.3) does not reach the next integer value, CARBS will not be able to vary this parameter. Adding a `scale` factor here that is at least `1/search_radius` is necessary for `LinearSpace` to work properly. `LogSpace` is a little more complicated, but if search space starts too small (<4) or at a small multiple of the `rounding_factor`, the same issues can occur and a higher `scale` may be required.
### Observations, Suggestions, and Candidates
* **Observations** are the result of a full experiment.
* **Suggestions** are points that have an outstanding request to get results from a full experiment, but which have not yet been observed.
* **Candidates** are points that are under consideration to become suggestions.
### Surrogate model fitting
The `SurrogateModel` builds a surrogate model for the function you are testing. It in turn has four fit functions, for different inputs:
* **fit_observations**: Used to fit `success_observations` to produce the initial models of the target function's outputs and costs.
* **fit_suggestions**: Modifies the model of the target function output to include predictions for outstanding suggestions, using either Thompson sampling or a kriging believer.
* **fit_failures**: Pass both `success_observations` and `failure_observations` to create a model of the failure probability
* **fit_pareto_set**: Pass observations in the pareto set, to create a model of the pareto output value versus cost.
================================================
FILE: carbs/__init__.py
================================================
from carbs.carbs import CARBS
from carbs.utils import *
================================================
FILE: carbs/carbs.py
================================================
# %%
import base64
import io
import math
import os
import random
import threading
import traceback
import uuid
from collections import OrderedDict
from pathlib import Path
from typing import Any
from typing import Dict
from typing import List
from typing import Optional
from typing import Sequence
from typing import Tuple
from typing import Union
from typing import cast
import numpy as np
import torch
import wandb
from attr import evolve
from loguru import logger
from torch import Tensor
from torch.distributions import Categorical
from torch.distributions import Distribution
from torch.distributions import Normal
from wandb.sdk.lib import RunDisabled
from wandb.sdk.wandb_run import Run
from carbs.model import SurrogateModel
from carbs.utils import CARBSParams, load_latest_checkpoint_from_wandb_run
from carbs.utils import CARBS_CHECKPOINT_PREFIX
from carbs.utils import CARBS_CHECKPOINT_SUFFIX
from carbs.utils import ObservationGroup
from carbs.utils import ObservationInBasic
from carbs.utils import ObservationInParam
from carbs.utils import ObserveOutput
from carbs.utils import Param
from carbs.utils import ParamDictType
from carbs.utils import RealNumberSpace
from carbs.utils import SUGGESTION_ID_DICT_KEY
from carbs.utils import SuggestOutput
from carbs.utils import SuggestionInBasic
from carbs.utils import SurrogateModelParams
from carbs.utils import add_dict_key_prefix
from carbs.utils import aggregate_logical_and_across_dim
from carbs.utils import assert_empty
from carbs.utils import expected_improvement
from carbs.utils import get_pareto_curve_plot
from carbs.utils import get_pareto_groups
from carbs.utils import get_pareto_groups_conservative
from carbs.utils import group_observations
from carbs.utils import observation_group_cost
from carbs.utils import observation_group_output
from carbs.utils import ordered_dict_index
from carbs.utils import pareto_area_from_groups
class CARBS:
"""
C.A.R.B.S = Cost Aware (pareto-) Regional Bayesian Search
Definitions
Target function: the real function we're trying to find optimal hyperparameters for
Surrogate (fitness): the estimated output of the target function from the Gaussian process model
Param space: original space of all the hyperparameters before transforms to the given target function
Basic space: intermediate space that contains the actual input space of our target function
e.g. Learning rate in param space is any real positive number and in basic space we reduce the search range
to a log space `lr = LogSpace(max=1)`
"""
def __init__(self, config: CARBSParams, params: List[Param]) -> None:
logger.info(f"Running CARBS with config {config}")
self.config = config
self.params = params
experiment_name = os.environ.get(
"EXPERIMENT_NAME", config.wandb_params.run_name
)
self.experiment_name = (
experiment_name if experiment_name is not None else "carbs_experiment"
)
self.param_space_by_name = {param.name: param.space for param in params}
self._real_number_space_by_name = OrderedDict(
(k, dim)
for k, dim in self.param_space_by_name.items()
if isinstance(dim, RealNumberSpace)
)
assert len(self._real_number_space_by_name) == len(
self.param_space_by_name
), "Real numbers only supported now"
self.real_dim = len(self._real_number_space_by_name)
self.search_center_in_basic = torch.zeros(
(self.real_dim,)
) # Immediately overwritten in set_search_center
self.search_radius_in_basic = torch.tensor(
float(self.config.initial_search_radius)
)
self.set_search_center({param.name: param.search_center for param in params})
self.success_observations: List[ObservationInBasic] = []
self.failure_observations: List[ObservationInBasic] = []
self.min_bounds_in_basic = torch.tensor(
[
dim.basic_from_param(dim.min_bound)
for dim in self._real_number_space_by_name.values()
]
)
self.max_bounds_in_basic = torch.tensor(
[
dim.basic_from_param(dim.max_bound)
for dim in self._real_number_space_by_name.values()
]
)
self._suggest_or_observe_lock = threading.Lock()
self.outstanding_suggestions: Dict[str, SuggestionInBasic] = (
{}
) # Keys are UUIDs
self._set_seed(self.config.seed)
# Only used so far to keep track of how many resamples we have suggested
self.resample_count: int = 0
num_dims_with_bounds = sum(
(not math.isinf(dim.min_bound)) or (not math.isinf(dim.max_bound))
for dim in self._real_number_space_by_name.values()
)
if num_dims_with_bounds > 10:
logger.info(
f"Many dimensions with bounds ({num_dims_with_bounds}), sampling may be slow"
)
self.overgenerate_candidate_factor = 2 ** (num_dims_with_bounds // 2)
self.wandb_run: Union[Run, RunDisabled, None] = None
self._init_wandb()
self.device = "cpu"
if torch.cuda.is_available():
self.device = f"cuda:{torch.cuda.current_device()}"
def set_search_center(self, input_in_param: ParamDictType) -> None:
self.search_center_in_basic = self._param_space_real_to_basic_space_real(
input_in_param
)
def suggest(
self, suggestion_id: Optional[str] = None, is_suggestion_remembered: bool = True
) -> SuggestOutput:
"""
Return a new suggestion
Keyword arguments:
suggestion_id: Optional[str] -- will be used to keep track of the suggestion
is_suggestion_remembered: bool -- whether to store the suggestion, which be fit with a surrogate value until it is observed
Returns:
SuggestOutput, which contains:
suggestion: ParamDictType -- Dictionary of parameters suggested
log: Dict[str, Any] -- Additional details about the suggestion, such as predicted cost and output
"""
with self._suggest_or_observe_lock:
if self._is_random_sampling():
return self._get_random_suggestion(
suggestion_id, is_suggestion_remembered
)
suggestion_in_basic: Optional[SuggestionInBasic]
if (
self.config.resample_frequency > 0
and len(self.success_observations)
> (self.resample_count + 1) * self.config.resample_frequency
):
suggestion_in_basic = self._get_resample_suggestion()
suggestion_in_param = self._basic_space_to_param_space(
suggestion_in_basic.real_number_input
)
if is_suggestion_remembered:
self._remember_suggestion(
suggestion_in_param,
suggestion_in_basic,
suggestion_id=suggestion_id,
)
return SuggestOutput(suggestion=suggestion_in_param)
try:
suggestion_in_basic = self._generate_candidate()
except Exception as e:
logger.warning(
f"Got error generating candidate {e}: {traceback.format_exc()}"
)
suggestion_in_basic = None
if suggestion_in_basic is None:
logger.warning("No candidates found, choosing at random")
return self._get_random_suggestion(
suggestion_id, is_suggestion_remembered
)
suggestion_in_param = self._basic_space_to_param_space(
suggestion_in_basic.real_number_input
)
if is_suggestion_remembered:
self._remember_suggestion(
suggestion_in_param,
suggestion_in_basic,
suggestion_id=suggestion_id,
)
log_dict = self._get_suggestion_log(
suggestion_in_param, suggestion_in_basic
)
return SuggestOutput(suggestion=suggestion_in_param, log=log_dict)
def observe(self, new_observation_in_param: ObservationInParam) -> ObserveOutput:
"""
Observe a new data point
Argument:
new_observation_in_param: ObservationInParam(
input: ParamDictType -- Dictionary mapping search vars to the values used for input to the function
output: float -- Target function output
cost: float = 1.0 -- Usually the time in seconds that the target function took to run
is_failure: bool = False -- Should be True if training did not complete properly (eg OOM error)
)
"""
with self._suggest_or_observe_lock:
self.forget_suggestion(new_observation_in_param.input)
self._add_observation(new_observation_in_param)
logs = self._get_observation_log(new_observation_in_param)
if self.config.is_saved_on_every_observation:
self._autosave()
return ObserveOutput(logs=logs)
def forget_suggestion(self, suggestion_to_forget: ParamDictType) -> None:
"""
Removes suggestion from outstanding_suggestions
"""
if SUGGESTION_ID_DICT_KEY in suggestion_to_forget:
suggestion_index = suggestion_to_forget[SUGGESTION_ID_DICT_KEY]
assert isinstance(suggestion_index, str)
if suggestion_index in self.outstanding_suggestions:
del self.outstanding_suggestions[suggestion_index]
else:
logger.info(f"Got unrecognized suggestion uuid `{suggestion_index}`")
else:
pass # It's fine to forget a suggestion without a UUID, but it doesn't do anything, we weren't remembering it anyway.
def initialize_from_observations(
self, observations: List[ObservationInParam]
) -> ObserveOutput:
logs: Dict[str, Any] = {}
for observation in observations:
self._add_observation(observation)
# logs.update(self._get_observation_log(observation))
if len(observations) > 0:
best_obs = max(
observations, key=lambda x: x.output * self.config.better_direction_sign
)
self.set_search_center(best_obs.input)
else:
logger.warning("No way to set the search center!")
return ObserveOutput(logs=logs)
def __getstate__(self) -> Dict[str, object]:
state = self.__dict__.copy()
state["wandb_run"] = None
state.pop("_suggest_or_observe_lock")
return state
def __setstate__(self, state: Dict[str, object]) -> None:
self.__dict__.update(state)
self._suggest_or_observe_lock = threading.Lock()
def _set_seed(self, seed: int) -> None:
torch.manual_seed(seed)
np.random.seed(seed)
random.seed(seed)
def _get_mask_for_invalid_points_in_basic(self, input_in_basic: Tensor) -> Tensor:
# mypy understand these types but pycharm does not :'(
is_above_min_bounds = aggregate_logical_and_across_dim(
input_in_basic > self.min_bounds_in_basic.unsqueeze(0)
)
is_below_max_bounds = aggregate_logical_and_across_dim(
input_in_basic < self.max_bounds_in_basic.unsqueeze(0)
)
mask = torch.logical_and(is_above_min_bounds, is_below_max_bounds)
return mask
def _round_integer_values_in_basic(self, input_in_basic: Tensor) -> Tensor:
for idx, space in enumerate(self._real_number_space_by_name.values()):
input_in_basic[..., idx] = space.round_tensor_in_basic(
input_in_basic[..., idx]
)
return input_in_basic
def _param_space_real_to_basic_space_real(
self, input_in_param: ParamDictType
) -> Tensor:
return torch.tensor(
[
dim.basic_from_param(input_in_param[k])
for k, dim in self._real_number_space_by_name.items()
]
)
def _param_space_obs_to_basic_space_obs(
self, observation_in_params: ObservationInParam
) -> ObservationInBasic:
input_in_param = observation_in_params.input
suggestion_id = input_in_param.pop(SUGGESTION_ID_DICT_KEY, None)
real_number_input = self._param_space_real_to_basic_space_real(input_in_param)
return ObservationInBasic(
real_number_input=real_number_input,
output=float(observation_in_params.output),
cost=float(observation_in_params.cost),
suggestion_id=str(suggestion_id) if suggestion_id else None,
)
def _basic_space_to_param_space(self, real_number_input: Tensor) -> ParamDictType:
return {
k: dim.param_from_basic(float(v))
for (k, dim), v in zip(
self._real_number_space_by_name.items(), real_number_input
)
}
def _basic_space_to_unrounded_param_space(
self, real_number_input: Tensor
) -> ParamDictType:
return {
k: dim.param_from_basic(float(v), is_rounded=False)
for (k, dim), v in zip(
self._real_number_space_by_name.items(), real_number_input
)
}
def _remember_suggestion(
self,
suggestion_in_param: ParamDictType,
suggestion_in_basic: SuggestionInBasic,
suggestion_id: Optional[str],
) -> ParamDictType:
if suggestion_id is None:
suggestion_id = str(uuid.uuid4())
suggestion_in_param[SUGGESTION_ID_DICT_KEY] = suggestion_id
self.outstanding_suggestions[suggestion_id] = suggestion_in_basic
return suggestion_in_param
def _add_observation(
self, observation_in_param: ObservationInParam
) -> Optional[ObservationInBasic]:
observation_in_basic = self._param_space_obs_to_basic_space_obs(
observation_in_param
)
if (
observation_in_param.is_failure
or not np.isfinite(observation_in_basic.output)
or np.isnan(observation_in_basic.output)
):
self.failure_observations.append(observation_in_basic)
return None
self.success_observations.append(observation_in_basic)
return observation_in_basic
@property
def _search_distribution_in_basic(self) -> Distribution:
return Normal(0, self.search_radius_in_basic)
def _sample_around_origins_in_basic(
self, num_samples: int, origins_in_basic: Tensor
) -> Tuple[Tensor, Tensor]:
assert (
len(origins_in_basic.shape) == 2
and origins_in_basic.shape[1] == self.real_dim
), f"Bad shape for origins_in_natural: {origins_in_basic.shape}"
origin_index = Categorical(
logits=torch.zeros((origins_in_basic.shape[0],))
).sample(torch.Size((num_samples,)))
origin_samples = origins_in_basic[origin_index]
real_samples: Tensor = (
origin_samples
+ self._search_distribution_in_basic.sample(
torch.Size((num_samples, self.real_dim))
)
)
probabilities = self._get_probability_in_search_space(
input_in_basic=real_samples, origins_in_basic=origins_in_basic
)
return real_samples, probabilities
def _get_probability_in_search_space(
self,
input_in_basic: Tensor,
origins_in_basic: Tensor,
) -> Tensor:
input_distance = torch.norm(
input_in_basic.unsqueeze(1) - origins_in_basic.unsqueeze(0), dim=-1
).to(self.device)
real_relative_probability_per_origin = torch.exp(
self._search_distribution_in_basic.log_prob(input_distance)
)
# we take the max probability origin
real_relative_probability: Tensor = real_relative_probability_per_origin.max(
dim=-1
).values
return real_relative_probability
def _is_random_sampling(self) -> bool:
# Random sampling as opposed to Bayesian optimization.
return len(self.success_observations) < self.config.num_random_samples
def sample_search_space(self, num_samples: int) -> List[SuggestionInBasic]:
# The length of the list returned will be less than or equal to num_samples. Filtering is done by _get_mask_for_invalid_points_in_basic, which checks min and max values.
if self._is_random_sampling():
# Main case
origins_in_basic = self.search_center_in_basic.unsqueeze(0)
else:
# Edge case that only occurs as a fallback when we can't generate a candidate
pareto_groups = self._get_pareto_groups(
is_conservative=self.config.is_pareto_group_selection_conservative
)
origins_in_basic = torch.stack(
[x[0].real_number_input for x in pareto_groups], dim=0
)
samples_in_basic, _ = self._sample_around_origins_in_basic(
num_samples * self.overgenerate_candidate_factor, origins_in_basic
)
valid_sample_mask = self._get_mask_for_invalid_points_in_basic(samples_in_basic)
samples_in_basic = samples_in_basic[valid_sample_mask][:num_samples]
if samples_in_basic.shape[0] < num_samples:
logger.warning(
f"Undergenerated valid samples, requested {num_samples} generated {samples_in_basic.shape[0]}"
)
self._crank_oversampling_up()
suggestions_in_basic = [
SuggestionInBasic(real_number_input=x) for x in samples_in_basic
]
return suggestions_in_basic
@torch.no_grad()
def _generate_candidate(self) -> Optional[SuggestionInBasic]:
surrogate_model = self.get_surrogate_model()
surrogate_model.fit_observations(self.success_observations)
surrogate_model.fit_suggestions(list(self.outstanding_suggestions.values()))
surrogate_model.fit_failures(
self.success_observations, self.failure_observations
)
pareto_groups = self._get_pareto_groups(
is_conservative=self.config.is_pareto_group_selection_conservative
)
all_pareto_observations = [obs for group in pareto_groups for obs in group]
surrogate_model.fit_pareto_set(all_pareto_observations)
num_samples_to_generate = (
self.config.num_candidates_for_suggestion_per_dim * self.real_dim
)
origins_in_basic = torch.stack(
[x[0].real_number_input for x in pareto_groups], dim=0
)
samples_in_basic, probabilities = self._sample_around_origins_in_basic(
num_samples_to_generate * self.overgenerate_candidate_factor,
origins_in_basic,
)
# do rounding first
samples_in_basic = self._round_integer_values_in_basic(samples_in_basic)
assert samples_in_basic.shape[0] > 0
# This is why we overgenerated
valid_sample_mask = self._get_mask_for_invalid_points_in_basic(samples_in_basic)
samples_in_basic = samples_in_basic[valid_sample_mask][:num_samples_to_generate]
probabilities = probabilities[valid_sample_mask][:num_samples_to_generate]
if samples_in_basic.shape[0] < num_samples_to_generate:
logger.info(
f"Undergenerated valid samples, requested {num_samples_to_generate} generated {samples_in_basic.shape[0]}"
)
self._crank_oversampling_up()
if samples_in_basic.shape[0] == 0:
return None
surrogate_model_outputs = surrogate_model.observe_surrogate(samples_in_basic)
assert surrogate_model_outputs.pareto_surrogate is not None
assert surrogate_model_outputs.pareto_estimate is not None
pareto_surrogate = surrogate_model_outputs.pareto_surrogate
if self.config.is_expected_improvement_pareto_value_clamped:
# Biasing technique to encourage higher cost exploration: Choose a random cost along pareto front, and make the pareto value there the minimum for all observations
# The pareto_groups are sorted, so the 0th and -1th elements represent the low and high values
log_min_cost = math.log(observation_group_cost(pareto_groups[0]))
if len(pareto_groups) > 1:
log_max_cost = math.log(observation_group_cost(pareto_groups[-1]))
else:
log_max_cost = log_min_cost
if self.config.max_suggestion_cost is not None:
log_max_cost = min(
log_max_cost, math.log(self.config.max_suggestion_cost)
)
# Choose a random cost along the curve
cost_threshold = math.exp(random.uniform(log_min_cost, log_max_cost))
# Choose the surrogate value at that cost
pareto_surrogate_at_threshold = (
surrogate_model.get_pareto_surrogate_for_cost(cost_threshold)
)
# Modify the pareto surrogate to be at least that value. This effectively cuts off exploration of low expected value points, which biases the search toward the right side of the pareto curve.
pareto_surrogate = torch.clamp(
pareto_surrogate, min=pareto_surrogate_at_threshold
)
if self.config.is_expected_improvement_value_always_max:
# NB: Only used for ablations
best_pareto_group_output = observation_group_output(pareto_groups[-1])
best_output_in_surrogate = surrogate_model._target_to_surrogate(
torch.tensor([best_pareto_group_output])
).item()
pareto_surrogate = (
torch.ones_like(pareto_surrogate) * best_output_in_surrogate
)
max_cost_masking = torch.ones_like(surrogate_model_outputs.cost_estimate)
if self.config.max_suggestion_cost is not None:
max_cost_masking = cast(
torch.BoolTensor,
surrogate_model_outputs.cost_estimate < self.config.max_suggestion_cost,
).to(torch.float)
assert (
max_cost_masking.max().item() > 0.5
), f"No candidates below max cost bound {self.config.max_suggestion_cost}"
ei_value = expected_improvement(
surrogate_model_outputs.surrogate_output,
surrogate_model_outputs.surrogate_var,
best_mu=pareto_surrogate,
exploration_bias=self.config.exploration_bias,
)
# Central equation: bias points by expected improvement, prior probability, and success probability, excluding points that are too expensive
acquisition_function_value = (
ei_value
* probabilities
* surrogate_model_outputs.success_probability
* max_cost_masking
)
best_idx = int(torch.argmax(acquisition_function_value).item())
# A single point is chosen by that argmax, so log the info for that point
log_info = dict(
surrogate_output=surrogate_model_outputs.surrogate_output[best_idx].item(),
pareto_surrogate=surrogate_model_outputs.pareto_surrogate[best_idx].item(),
surrogate_var=surrogate_model_outputs.surrogate_var[best_idx].item(),
pareto_estimate=surrogate_model_outputs.pareto_estimate[best_idx].item(),
cost_estimate=surrogate_model_outputs.cost_estimate[best_idx].item(),
target_estimate=surrogate_model_outputs.target_estimate[best_idx].item(),
target_var=surrogate_model_outputs.target_var[best_idx].item(),
probabilities=probabilities[best_idx].item(),
expected_improvement=ei_value[best_idx].item(),
success_probability=surrogate_model_outputs.success_probability[
best_idx
].item(),
)
return SuggestionInBasic(
real_number_input=samples_in_basic[best_idx], log_info=log_info
)
def get_surrogate_model(self) -> SurrogateModel:
params = SurrogateModelParams(
real_dims=self.real_dim,
better_direction_sign=self.config.better_direction_sign,
outstanding_suggestion_estimator=self.config.outstanding_suggestion_estimator,
device=self.device,
scale_length=self.search_radius_in_basic.item(),
)
return SurrogateModel(params)
def _get_random_suggestion(
self,
suggestion_id: Optional[str] = None,
is_suggestion_remembered: bool = True,
num_sampling_attempts: int = 8,
) -> SuggestOutput:
for i in range(num_sampling_attempts):
# Might as well try to grab 32 points since we get some parallelism for free. sample_search_space will crank up oversampling if needed.
suggestions = self.sample_search_space(32)
if len(suggestions) > 0:
suggestion_in_param = self._basic_space_to_param_space(
suggestions[0].real_number_input
)
if is_suggestion_remembered:
self._remember_suggestion(
suggestion_in_param, suggestions[0], suggestion_id=suggestion_id
)
return SuggestOutput(suggestion=suggestion_in_param)
raise Exception("Cannot get a random sample :(")
def _observation_group_output_pos_better(self, group: Sequence[ObservationInBasic]):
return observation_group_output(group) * self.config.better_direction_sign
def _crank_oversampling_up(self) -> None:
self.overgenerate_candidate_factor = min(
self.overgenerate_candidate_factor * 2, 2**self.real_dim
)
def _get_pareto_groups(
self, is_conservative: bool = False
) -> Tuple[ObservationGroup, ...]:
"""
Get pareto groups from success observations.
:param is_conservative: see get_pareto_groups_conservative for description
:return:
"""
grouped_observations = group_observations(self.success_observations)
if is_conservative:
pareto_groups = get_pareto_groups_conservative(
grouped_observations,
self.config.min_pareto_cost_fraction,
self.config.better_direction_sign,
)
else:
pareto_groups = get_pareto_groups(
grouped_observations,
self.config.min_pareto_cost_fraction,
self.config.better_direction_sign,
)
return pareto_groups
def _get_pareto_set(
self, is_conservative: bool = False
) -> Tuple[ObservationInBasic, ...]:
pareto_set: List[ObservationInBasic] = []
for group in self._get_pareto_groups(is_conservative):
pareto_set.extend(group)
return tuple(pareto_set)
def _get_resample_suggestion(self) -> SuggestionInBasic:
# we specifically want the non-conservative version so we can resample!
pareto_groups = list(self._get_pareto_groups(is_conservative=False))
# This ordering will determine which obs we select:
# we select the first one which has the minimum number of observations/suggestions already
pareto_groups.sort(key=observation_group_cost)
pareto_obs_with_counts: List[Tuple[ObservationInBasic, int]] = []
for group in pareto_groups:
# Take the first obs because they all have the same sample point
first_obs = group[0]
count = len(group)
# Add to the count of the group the number of outstanding suggestions that are already out to resample this group.
for suggestion in self.outstanding_suggestions.values():
if torch.all(
torch.isclose(
suggestion.real_number_input, first_obs.real_number_input
)
):
count += 1
pareto_obs_with_counts.append((first_obs, count))
# Resample one of the observation groups with the minimum count along the pareto front
min_count = min(x[1] for x in pareto_obs_with_counts)
selected_obs = next(x[0] for x in pareto_obs_with_counts if x[1] == min_count)
resample_suggestion = SuggestionInBasic(selected_obs.real_number_input)
self.resample_count += 1
return resample_suggestion
def _init_wandb(self) -> None:
if self.config.is_wandb_logging_enabled:
wandb_params = self.config.wandb_params
if wandb_params.run_id is None:
self.wandb_run = wandb.init(
config=self.config.to_dict(),
project=wandb_params.project_name,
group=wandb_params.group_name,
name=wandb_params.run_name,
dir=wandb_params.root_dir,
)
else:
self.wandb_run = wandb.init(
config=self.config.to_dict(),
project=wandb_params.project_name,
group=wandb_params.group_name,
name=wandb_params.run_name,
dir=wandb_params.root_dir,
reinit=True,
resume="allow",
id=wandb_params.run_id,
)
@property
def cumulative_cost(self) -> float:
# We can't add cost of failed observations, they may not be valid
return sum(x.cost for x in self.success_observations)
@property
def observation_count(self) -> int:
return len(self.success_observations) + len(self.failure_observations)
def _get_observation_log(self, observation: ObservationInParam) -> Dict[str, Any]:
log_dict: Dict[str, Any] = {
"observation_count": self.observation_count,
"cumulative_cost": self.cumulative_cost,
"failed_observation_count": len(self.failure_observations),
}
log_dict.update(add_dict_key_prefix(observation.input, "observation/"))
log_dict["observation/is_failure"] = 1 if observation.is_failure else 0
if not observation.is_failure:
log_dict["observation/output"] = observation.output
log_dict["observation/cost"] = observation.cost
if len(self.success_observations) > 0:
best_observation_in_basic = max(
self.success_observations,
key=lambda x: x.output * self.config.better_direction_sign,
)
best_observation_in_param = self._basic_space_to_param_space(
best_observation_in_basic.real_number_input
)
log_dict.update(
add_dict_key_prefix(best_observation_in_param, "best_observation/")
)
log_dict["best_observation/output"] = best_observation_in_basic.output
log_dict["best_observation/cost"] = best_observation_in_basic.cost
grouped_observations = group_observations(self.success_observations)
pareto_groups = self._get_pareto_groups(
self.config.is_pareto_group_selection_conservative
)
log_dict["pareto_group_count"] = len(pareto_groups)
log_dict["pareto_area"] = pareto_area_from_groups(pareto_groups)
resampled_groups = [x for x in grouped_observations if len(x) > 1]
if len(resampled_groups) > 0:
best_resampled_observations = max(
resampled_groups,
key=lambda x: observation_group_output(x)
* self.config.better_direction_sign,
)
resampled_observation_in_param = self._basic_space_to_param_space(
best_resampled_observations[0].real_number_input
)
best_resampled_observation_outputs = [
x.output for x in best_resampled_observations
]
log_dict.update(
add_dict_key_prefix(
resampled_observation_in_param, "best_resampled_observation/"
)
)
log_dict["best_resampled_observation/output_mean"] = np.mean(
best_resampled_observation_outputs
)
log_dict["best_resampled_observation/output_std_dev"] = np.std(
best_resampled_observation_outputs
)
log_dict["best_resampled_observation/cost_mean"] = (
observation_group_cost(best_resampled_observations)
)
log_dict["best_resampled_observation/sample_count"] = len(
best_resampled_observations
)
if self.config.is_wandb_logging_enabled and len(pareto_groups) > 0:
plot_path = get_pareto_curve_plot(
self.success_observations,
pareto_groups,
self.config.wandb_params.root_dir,
obs_count=self.observation_count,
)
if plot_path is not None:
log_dict["pareto_curve"] = wandb.Image(plot_path)
if (
self.config.is_wandb_logging_enabled
and self.config.wandb_params.is_observation_logged
and self.wandb_run
):
wandb.log(log_dict)
return log_dict
def _get_suggestion_log(
self, suggestion: ParamDictType, observation_in_surrogate: SuggestionInBasic
) -> Dict[str, Any]:
log_dict: Dict[str, Any] = {
"observation_count": self.observation_count,
"cumulative_cost": self.cumulative_cost,
"failed_observation_count": len(self.failure_observations),
}
log_dict.update(add_dict_key_prefix(suggestion, "suggestion/"))
log_dict.update(
add_dict_key_prefix(observation_in_surrogate.log_info, "suggestion_meta/")
)
if (
self.config.is_wandb_logging_enabled
and self.config.wandb_params.is_suggestion_logged
and self.wandb_run
):
wandb.log(log_dict)
return log_dict
def _autosave(self) -> None:
filename = f"{CARBS_CHECKPOINT_PREFIX}{self.observation_count}{CARBS_CHECKPOINT_SUFFIX}"
self.save_to_file(
filename, upload_to_wandb=self.config.is_wandb_logging_enabled
)
@staticmethod
def load_from_file(
f: Union[str, io.BytesIO],
is_wandb_logging_enabled: bool = False,
override_params: Optional[CARBSParams] = None,
) -> "CARBS":
state = torch.load(f)
if override_params is not None:
state["config"] = override_params
if not is_wandb_logging_enabled:
state["config"] = evolve(state["config"], is_wandb_logging_enabled=False)
optimizer = CARBS.load_state_dict(state)
return optimizer
@classmethod
def load_from_string(
cls, state: str, is_wandb_logging_enabled: bool = False
) -> "CARBS":
return cls.load_from_file(
io.BytesIO(base64.b64decode(state)), is_wandb_logging_enabled
)
def get_state_dict(self) -> Dict[str, Any]:
outstanding_suggestions: Dict[str, SuggestionInBasic] = {}
for key, suggestion in self.outstanding_suggestions.items():
# TODO: don't carry around these enormous real_number_inputs in the first place
outstanding_suggestions[key] = evolve(
suggestion, real_number_input=suggestion.real_number_input.clone()
)
return {
"config": self.config,
"params": self.params,
"success_observations": self.success_observations,
"failure_observations": self.failure_observations,
"outstanding_suggestions": outstanding_suggestions,
"resample_count": self.resample_count,
}
@classmethod
def load_state_dict(cls, state: Dict[str, Any]) -> "CARBS":
optimizer = CARBS(state["config"], state["params"])
optimizer.success_observations = state["success_observations"]
optimizer.failure_observations = state["failure_observations"]
optimizer.outstanding_suggestions = state["outstanding_suggestions"]
optimizer.resample_count = state["resample_count"]
return optimizer
def save_to_file(self, filename: str, upload_to_wandb: bool = False) -> None:
checkpoint_path = (
Path(self.config.checkpoint_dir) / self.experiment_name / filename
)
checkpoint_path.parent.mkdir(exist_ok=True, parents=True)
torch.save(self.get_state_dict(), checkpoint_path)
if upload_to_wandb:
wandb.save(checkpoint_path)
def serialize(self) -> str:
buf = io.BytesIO()
torch.save(self.get_state_dict(), buf)
buf.seek(0)
return base64.b64encode(buf.read()).decode("utf-8")
def warm_start_from_wandb(
self,
run_name: str,
is_prior_observation_valid: bool = False,
added_parameters: Optional[ParamDictType] = None,
) -> None:
filename = load_latest_checkpoint_from_wandb_run(run_name)
self.warm_start(filename, is_prior_observation_valid, added_parameters)
def warm_start(
self,
filename: str,
is_prior_observation_valid: bool = False,
added_parameters: Optional[ParamDictType] = None,
) -> None:
prior_carbs = CARBS.load_from_file(filename)
# Copying over observations
if is_prior_observation_valid:
prior_keys = set(prior_carbs.param_space_by_name.keys())
current_keys = set(self.param_space_by_name.keys())
added_keys = (
set() if added_parameters is None else set(added_parameters.keys())
)
assert_empty(
prior_keys - current_keys,
"Prior observations used params that are not included in search",
)
assert_empty(
current_keys - (prior_keys | added_keys), "Pass in added parameters"
)
assert_empty(added_keys & prior_keys, "Cannot add param already in prior")
for prior_observation_in_basic in prior_carbs.success_observations:
prior_observation_in_params = prior_carbs._basic_space_to_param_space(
prior_observation_in_basic.real_number_input
)
if added_parameters is not None:
prior_observation_in_params.update(added_parameters)
try:
self._add_observation(
ObservationInParam(
input=prior_observation_in_params,
output=prior_observation_in_basic.output,
cost=prior_observation_in_basic.cost,
)
)
except ValueError as e:
logger.warning(
f"Observation {prior_observation_in_params} not valid in current space: {e}; skipping"
)
logger.info(
f"Loaded {len(self.success_observations)} observations from prior run"
)
for prior_observation_in_basic in prior_carbs.failure_observations:
prior_observation_in_params = prior_carbs._basic_space_to_param_space(
prior_observation_in_basic.real_number_input
)
if added_parameters is not None:
prior_observation_in_params.update(added_parameters)
try:
self._add_observation(
ObservationInParam(
input=prior_observation_in_params,
output=prior_observation_in_basic.output,
cost=prior_observation_in_basic.cost,
is_failure=True,
)
)
except ValueError as e:
logger.warning(
f"Observation {prior_observation_in_params} not valid in current space: {e}; skipping"
)
logger.info(
f"Loaded {len(self.failure_observations)} failed observations from prior run"
)
self.resample_count = prior_carbs.resample_count
if added_parameters is not None:
for param_name, param_value in added_parameters.items():
if param_name in self._real_number_space_by_name:
param_idx = ordered_dict_index(
self._real_number_space_by_name, param_name
)
param_value_in_basic = self._real_number_space_by_name[
param_name
].basic_from_param(param_value)
self.search_center_in_basic[param_idx] = param_value_in_basic
================================================
FILE: carbs/model.py
================================================
import math
from typing import List
from typing import Optional
import attr
import numpy as np
import pyro
import torch
from pyro.contrib import gp as gp
from pyro.contrib.gp.kernels import Kernel
from pyro.contrib.gp.models import GPRegression
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import QuantileTransformer
from torch import Tensor
from torch.distributions import Normal
from carbs.utils import ObservationInBasic
from carbs.utils import OutstandingSuggestionEstimatorEnum
from carbs.utils import SuggestionInBasic
from carbs.utils import SurrogateModelParams
@attr.s(auto_attribs=True, collect_by_mro=True)
class SurrogateObservationOutputs:
surrogate_output: Tensor
surrogate_var: Tensor
cost_estimate: Tensor
target_estimate: Tensor
target_var: Tensor
success_probability: Tensor
pareto_surrogate: Optional[Tensor]
pareto_estimate: Optional[Tensor]
class SurrogateModel:
def __init__(
self,
params: SurrogateModelParams,
) -> None:
self.params = params
self.output_transformer: Optional[QuantileTransformer] = None
self.cost_transformer: Optional[MinMaxScaler] = None
self.output_model: Optional[GPRegression] = None
self.cost_model: Optional[GPRegression] = None
self.pareto_model: Optional[GPRegression] = None
self.min_logcost: float = float("-inf")
self.max_logcost: float = float("inf")
self.min_pareto_logcost: float = float("-inf")
self.max_pareto_logcost: float = float("inf")
self.success_model: Optional[GPRegression] = None
def _get_kernel(self) -> Kernel:
matern_kernel = gp.kernels.Matern32(
input_dim=self.params.real_dims,
lengthscale=self.params.scale_length * torch.ones((self.params.real_dims,)),
)
linear_kernel = gp.kernels.Linear(input_dim=self.params.real_dims)
return gp.kernels.Sum(linear_kernel, matern_kernel)
def _get_model(self, inputs: Tensor, outputs: Tensor, kernel: Optional[Kernel] = None) -> GPRegression:
# Heavily inspired by the HEBO paper. Some choices that appear arbitrary, such as the model noise, are copied from that paper.
# https://arxiv.org/abs/2012.03826
if kernel is None:
assert inputs.shape[-1] == self.params.real_dims, f"Must provide kernel for input shape {inputs.shape}"
kernel = self._get_kernel()
# Gaussian Process Regression
model = gp.models.GPRegression(
inputs.to(self.params.device),
outputs.to(self.params.device),
kernel=kernel,
jitter=1.0e-4,
).to(self.params.device)
model.noise = pyro.nn.PyroSample(pyro.distributions.LogNormal(math.log(1e-2), 0.5))
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)
gp.util.train(model, optimizer)
model.eval()
return model
def fit_observations(self, success_observations: List[ObservationInBasic]) -> None:
self._fit_target_transformers(success_observations)
inputs_in_basic = torch.stack([x.real_number_input for x in success_observations]).detach()
outputs_in_surrogate = self._target_to_surrogate(torch.tensor([x.output for x in success_observations]))
self.output_model = self._get_model(inputs_in_basic, outputs_in_surrogate)
costs_after_transform = self._cost_to_logcost(torch.tensor([x.cost for x in success_observations]))
self.cost_model = self._get_model(inputs_in_basic, costs_after_transform)
def _fit_target_transformers(self, success_observations: List[ObservationInBasic]) -> None:
raw_outputs = np.array([x.output for x in success_observations])
# Using n_quantiles < len(observations_in_basic) preserves more distance information within the quantiles
n_quantiles = int(np.sqrt(len(success_observations)))
self.output_transformer = QuantileTransformer(output_distribution="normal", n_quantiles=n_quantiles)
self.output_transformer.fit(raw_outputs.reshape(-1, 1))
log_costs = np.log(np.array([x.cost for x in success_observations]))
self.cost_transformer = MinMaxScaler(feature_range=(-1, 1))
self.cost_transformer.fit(log_costs.reshape(-1, 1))
transformed_costs = self.cost_transformer.transform(log_costs.reshape(-1, 1))
self.min_logcost = transformed_costs.min()
self.max_logcost = transformed_costs.max()
# Target space is the space of the score function that we're optimizing.
# Surrogate space is a well-behaved Gaussian space. It's the output analog of basic space.s
def _target_to_surrogate(self, x: Tensor) -> Tensor:
# Goes from target function space to the surrogate function space we model with GP
assert self.output_transformer is not None, "Fit output_transformer before calling target_to_surrogate!"
x_shape = x.shape
x = x.view(-1, 1).cpu()
transformed_x = torch.from_numpy(
self.output_transformer.transform(x.numpy()) * self.params.better_direction_sign
).to(self.params.device)
return transformed_x.view(*x_shape)
def _surrogate_to_target(self, x: Tensor) -> Tensor:
# TODO: a good unit test would be that this inverts the above
assert self.output_transformer is not None, "Fit output_transformer before calling surrogate_to_target!"
x_shape = x.shape
x = x.view(-1, 1).cpu()
transformed_x = torch.from_numpy(
self.output_transformer.inverse_transform(x.numpy() * self.params.better_direction_sign)
).to(self.params.device)
return transformed_x.view(*x_shape)
# Cost space is in seconds or dollars or something like that.
# Logcost space is the log of that, scaled a min-max range of -1 to 1.
def _cost_to_logcost(self, x: Tensor) -> Tensor:
# Goes from cost space to the logcost function space we model with GP
assert self.cost_transformer is not None, "Fit cost_transformer before calling cost_to_logcost!"
x_shape = x.shape
x = torch.log(x.view(-1, 1)).cpu()
transformed_x = torch.from_numpy(self.cost_transformer.transform(x.numpy())).to(self.params.device)
return transformed_x.view(*x_shape)
def _logcost_to_cost(self, x: Tensor) -> Tensor:
# TODO: a good unit test would be that this inverts the above
assert self.cost_transformer is not None, "Fit cost_transformer before calling logcost_to_cost!"
x_shape = x.shape
transformed_x = torch.from_numpy(self.cost_transformer.inverse_transform(x.view(-1, 1).cpu().numpy())).to(
self.params.device
)
return torch.exp(transformed_x).view(*x_shape)
def fit_suggestions(self, outstanding_suggestions: List[SuggestionInBasic]) -> None:
# Note that although we are fitting, no target outputs are passed in, and they have to be estimated using the surrogate model.
if len(outstanding_suggestions) == 0:
return
assert self.output_model is not None, "Fit observations before suggestions!"
assert self.cost_model is not None, "Fit observations before suggestions!"
inputs_in_basic = (
torch.stack([x.real_number_input for x in outstanding_suggestions]).detach().to(self.params.device)
)
# In both mean and Thompson sampling, the output model, which was trained on all real observations, is used to estimate the outputs of the suggestions. This is necessary to do as we await the real outputs.
if self.params.outstanding_suggestion_estimator == OutstandingSuggestionEstimatorEnum.MEAN:
output_predictions, _unused_output_vars = self.output_model(
inputs_in_basic, full_cov=False, noiseless=True
)
elif self.params.outstanding_suggestion_estimator == OutstandingSuggestionEstimatorEnum.THOMPSON:
sampler = self.output_model.iter_sample(noiseless=True)
outputs = []
for input_in_basic in inputs_in_basic:
outputs.append(sampler(input_in_basic.unsqueeze(0)))
output_predictions = torch.cat(outputs, dim=0)
else:
raise Exception(f"Invalid estimator {self.params.outstanding_suggestion_estimator}")
# With these new guesses at the outputs, we retrain the output model, which tells the Bayesian optimizer not to look for uncertainty near these points.
combined_inputs = torch.cat([self.output_model.X, inputs_in_basic])
combined_outputs = torch.cat([self.output_model.y, output_predictions])
self.output_model = self._get_model(combined_inputs, combined_outputs)
def fit_pareto_set(self, pareto_observations: List[ObservationInBasic]) -> None:
costs_after_transform = self._cost_to_logcost(torch.tensor([x.cost for x in pareto_observations]))
outputs_in_surrogate = self._target_to_surrogate(torch.tensor([x.output for x in pareto_observations]))
# TODO(research): This model is not guaranteed to be monotonic, which would be nice
self.pareto_model = self._get_model(
costs_after_transform, outputs_in_surrogate, kernel=gp.kernels.RBF(input_dim=1)
)
self.min_pareto_logcost = costs_after_transform.min().item()
self.max_pareto_logcost = costs_after_transform.max().item()
def get_pareto_surrogate_for_cost(self, cost: float) -> float:
assert self.pareto_model is not None
costs_after_transform = self._cost_to_logcost(torch.tensor([cost]))
costs_after_transform = torch.clamp(
costs_after_transform, min=self.min_pareto_logcost, max=self.max_pareto_logcost
)
pareto_output, pareto_var = self.pareto_model(costs_after_transform)
return float(pareto_output.item())
def fit_failures(
self, success_observations: List[ObservationInBasic], failure_observations: List[ObservationInBasic]
) -> None:
num_success, num_failure = len(success_observations), len(failure_observations)
if num_failure == 0 or num_success == 0:
self.success_predictor = None
return
all_observations = success_observations + failure_observations
inputs_in_basic = torch.stack(([x.real_number_input for x in all_observations])).detach()
# Use this labeling so we can just get the CDF at zero as probability of success
labels = torch.tensor([-1] * num_success + [1] * num_failure)
self.success_model = self._get_model(inputs_in_basic, labels)
def _get_success_prob(self, samples_in_natural: Tensor):
if self.success_model is None:
return torch.ones((samples_in_natural.shape[0],), device=self.params.device)
success_pred, success_var = self.success_model(samples_in_natural)
prior = Normal(success_pred, success_var)
success_pred = prior.cdf(torch.zeros((samples_in_natural.shape[0],), device=self.params.device))
return success_pred
@torch.no_grad()
def observe_surrogate(self, samples_in_basic: Tensor) -> SurrogateObservationOutputs:
assert self.output_model is not None
surrogate_output, surrogate_var = self.output_model(samples_in_basic.to(self.params.device))
assert self.cost_model is not None
logcost, logcost_var = self.cost_model(samples_in_basic.to(self.params.device))
if self.pareto_model is not None:
pareto_logcost = torch.clamp(logcost, min=self.min_pareto_logcost, max=self.max_pareto_logcost)
pareto_surrogate, pareto_var = self.pareto_model(pareto_logcost)
pareto_estimate = self._surrogate_to_target(pareto_surrogate)
else:
pareto_surrogate = None
pareto_estimate = None
success_probability = self._get_success_prob(samples_in_basic.to(self.params.device))
cost_estimate = self._logcost_to_cost(logcost)
target_estimate = self._surrogate_to_target(surrogate_output)
# target_var is only used for logging, but we want it to be correct-ish
# self._surrogate_to_target will get clamped at the top or bottom, so we take half the distance between
# +1 and -1 std dev as the new std dev. Return variance which is std dev ** 2
target_var = torch.square(
(
self._surrogate_to_target(surrogate_output + torch.sqrt(surrogate_var))
- self._surrogate_to_target(surrogate_output - torch.sqrt(surrogate_var))
)
/ 2.0
)
return SurrogateObservationOutputs(
surrogate_output=surrogate_output,
surrogate_var=surrogate_var,
cost_estimate=cost_estimate,
pareto_surrogate=pareto_surrogate,
pareto_estimate=pareto_estimate,
target_estimate=target_estimate,
target_var=target_var,
success_probability=success_probability,
)
================================================
FILE: carbs/serialization.py
================================================
from __future__ import (
annotations, # using this to get Postponed Evaluation of Annotations -- https://www.python.org/dev/peps/pep-0563/
)
import multiprocessing
from contextlib import contextmanager
from copy import deepcopy
from datetime import datetime
from enum import Enum
from typing import Any
from typing import ContextManager
from typing import Dict
from typing import List
from typing import Set
from typing import Type
from typing import TypeVar
from typing import Union
from uuid import UUID
import attr
FREEZE_KEY = "$_frozen"
TC = TypeVar("TC", bound="Serializable")
_CLASS_KEY = "$type"
lock = multiprocessing.Lock()
@attr.s(hash=True, collect_by_mro=True)
class Serializable:
# noinspection PyDataclass
def __attrs_post_init__(self) -> None:
self.__dict__[FREEZE_KEY] = True
def __setattr__(self, item: str, value: Any):
if self.__dict__.get(FREEZE_KEY):
raise ValueError("instance is frozen; see: .mutable_clone()")
else:
self.__dict__[item] = value
def mutable_clone(self: TC) -> ContextManager[TC]:
freeze_list: List[Serializable] = []
def thaw(attr_object: Serializable) -> Serializable:
kwargs: Dict[str, Any] = {}
for attribute in attr.fields(attr_object.__class__):
k = attribute.name
v = getattr(attr_object, k)
if isinstance(v, Serializable):
kwargs[k] = thaw(v)
elif isinstance(v, tuple) and len(v) > 0 and isinstance(v[0], Serializable):
kwargs[k] = tuple([thaw(x) for x in v])
else:
kwargs[k] = deepcopy(v)
# noinspection PyArgumentList
attr_cloned = attr_object.__class__(**kwargs)
attr_cloned.__dict__[FREEZE_KEY] = False
freeze_list.append(attr_cloned)
return attr_cloned
@contextmanager
def context():
try:
yield thaw(self)
finally:
for attr_cloned in freeze_list:
attr_cloned.__dict__[FREEZE_KEY] = True
return context()
def to_dict(self) -> dict:
dump: Dict[str, Any] = {_CLASS_KEY: get_qualname_from_serializable_type(type(self))}
for attribute in attr.fields(self.__class__):
k = attribute.name
v = getattr(self, k)
dump[k] = _to_dict(v)
return dump
@classmethod
def from_dict(cls: Type[TC], dump: dict, is_upgrade_allowed: bool = False) -> TC:
"""
Using is_upgrade_allowed is dangerous:
- Silently ignores keys that exist in the serialization but no longer in the object
Could definitely lose data.
- Silently ignores missing attributes, assuming they will have default values.
This will likely blow up if there are no default values.
The most likely case that this will be problematic is if you RENAMED an attribute
There's no way for us to really detect that.
I'd be ok with someone adding explicit "rename" support when they actually need it
"""
kwargs = {}
remaining_dict_keys = set(dump.keys())
for attribute in attr.fields(cls):
k = attribute.name
if is_upgrade_allowed and k not in dump:
# this enables us to load older ubjects when new default attributes have been added
# if there is no default value, the later construction will fail, there is
# no easy way around that without getting into more complex migration schemes
continue
if k not in remaining_dict_keys:
raise Exception(f"Serialized object {cls} is missing key={k}")
remaining_dict_keys.remove(k)
kwargs[k] = _from_value(dump[k], is_upgrade_allowed)
if _CLASS_KEY in remaining_dict_keys:
remaining_dict_keys.remove(_CLASS_KEY)
if len(remaining_dict_keys) > 0:
# silently drops keys here!
if not is_upgrade_allowed:
bad_keys = sorted(remaining_dict_keys)
raise ParamTypeError(f"Tried to load data for {cls} but got unexpected keys: {bad_keys}")
# noinspection PyArgumentList
return cls(**kwargs)
def _to_dict(v: Any) -> Any:
# better to check for the existance of this attribute than to check isinstance
# even works in jupyter notebooks when doing code reloading then..
if hasattr(v, "to_dict"):
result = v.to_dict()
elif isinstance(v, tuple):
if len(v) > 0:
result = tuple([_to_dict(x) for x in v])
else:
result = v
elif isinstance(v, Enum):
result = {"value": v.value, _CLASS_KEY: get_qualname_from_serializable_type(type(v))}
elif isinstance(v, UUID):
result = dict(value=str(v))
result[_CLASS_KEY] = UUID.__name__
elif isinstance(v, datetime):
result = dict(value=v.isoformat())
result[_CLASS_KEY] = datetime.__name__
else:
assert isinstance(v, (float, int, bool, str, type(None))), "Unexpected type: " + str(v)
result = v
return result
def _from_value(v: Dict[str, Any], is_upgrade_allowed: bool) -> Any:
result: Any
if isinstance(v, Dict) and _CLASS_KEY in v:
class_name = v[_CLASS_KEY]
if class_name == UUID.__name__:
result = UUID(v["value"])
elif class_name == datetime.__name__:
result = datetime.fromisoformat(v["value"])
else:
t = get_serializable_type_from_qualname(class_name)
if issubclass(t, Enum):
# noinspection PyArgumentList
result = t(value=v["value"]) # type: ignore
else:
result = t.from_dict(v, is_upgrade_allowed=is_upgrade_allowed)
# TAKS c00f9a38-bd11-4e14-97d8-02b9bf843ca6: delete this after we get rid of OldSerializable
elif isinstance(v, Dict) and "_serializable_type" in v:
t = get_serializable_type_from_qualname(v["_serializable_type"])
result = t.from_dict(v, is_upgrade_allowed=is_upgrade_allowed)
elif isinstance(v, (list, tuple)):
if len(v) == 0:
result = tuple()
else:
inner_objects = []
for inner_value in v:
inner_objects.append(_from_value(inner_value, is_upgrade_allowed))
result = tuple(inner_objects)
else:
assert isinstance(v, (float, int, bool, str, type(None))), "Unexpected type: " + str(v)
result = v
return result
qualname_to_serializable_type: Dict[str, Type[Serializable]] = {}
serializable_type_to_qualname: Dict[Type[Serializable], str] = {}
# this is here because some libraries make enums with the same names, but we dont use those, so, fine
unserializable_types: Set[str] = set()
class ParamTypeError(TypeError):
pass
def _get_all_subclasses(cls):
all_subclasses = []
for subclass in cls.__subclasses__():
all_subclasses.append(subclass)
all_subclasses.extend(_get_all_subclasses(subclass))
return all_subclasses
def _compute_unserializable(qualnames: List[str]) -> None:
global unserializable_types
if not unserializable_types:
already_seen_names = set()
for qualname in qualnames:
if qualname in already_seen_names:
unserializable_types.add(qualname)
already_seen_names.add(qualname)
def _dedupe_subclasses_by_qualname(subclasses: List[type]) -> List[type]:
qualname_to_serializable_type = {x.__qualname__: x for x in subclasses}
return list(qualname_to_serializable_type.values())
def _dedupe_subclasses_by_id(subclasses: List[type]) -> List[type]:
id_to_serializable_type = {id(x): x for x in subclasses}
return list(id_to_serializable_type.values())
def get_all_serializable_classes() -> List[type]:
# we don't reload the Enum class (since it's standard library) so we end up having duplicate subclasses for Enums
enum_subclasses = _dedupe_subclasses_by_qualname(_get_all_subclasses(Enum))
# we want to make sure we have classes with different names which is why we need to dedupe with the ids
serializable_subclasses = _dedupe_subclasses_by_id(_get_all_subclasses(Serializable))
return serializable_subclasses + enum_subclasses
def get_serializable_type_from_qualname(qualname: str) -> Type[Serializable]:
global qualname_to_serializable_type
global unserializable_types
if not qualname_to_serializable_type:
# makes our globals thread safe and hopefully free from weird race conditions
with lock:
if not qualname_to_serializable_type:
all_serializable_classes = get_all_serializable_classes()
qualname_to_serializable_type = {x.__qualname__: x for x in all_serializable_classes}
_compute_unserializable([x.__qualname__ for x in all_serializable_classes])
if qualname in unserializable_types:
raise Exception(f"{qualname} cannot be deserialized because there are multiple definitions")
return qualname_to_serializable_type[qualname]
def get_qualname_from_serializable_type(serializable_type: type) -> str:
global serializable_type_to_qualname
global unserializable_types
if not serializable_type_to_qualname:
# makes our globals thread safe and hopefully free from weird race conditions
with lock:
if not serializable_type_to_qualname:
all_serializable_classes = get_all_serializable_classes()
serializable_type_to_qualname = {x: x.__qualname__ for x in all_serializable_classes}
_compute_unserializable(list(serializable_type_to_qualname.values()))
result = serializable_type_to_qualname.get(serializable_type, serializable_type.__qualname__)
if result in unserializable_types:
raise Exception(f"{result} cannot be serialized because there are multiple definitions")
return result
T = TypeVar("T", bool, int, float, str, tuple)
DictNest = Dict[str, Union[T, Dict[str, Any]]]
DictFlat = Dict[str, T]
def flatten_dict(d: DictNest, prefix: str = "") -> DictFlat:
flattened_dict = {}
for k, v in d.items():
if isinstance(v, dict):
flattened_dict.update(flatten_dict(v, f"{prefix}{k}."))
else:
flattened_dict[f"{prefix}{k}"] = v
return flattened_dict
def inflate_dict(d: DictFlat) -> DictNest:
inflated_dict: DictNest = {}
for key, value in d.items():
parts = key.split(".")
d = inflated_dict
for part in parts[:-1]:
d = d.setdefault(part, {})
d[parts[-1]] = value
return inflated_dict
================================================
FILE: carbs/test_carbs.py
================================================
import os
from typing import List
import pytest
import wandb
from carbs import LogitSpace
from carbs import ObservationInParam
from carbs.carbs import CARBS
from carbs.utils import CARBSParams
from carbs.utils import LinearSpace
from carbs.utils import LogSpace
from carbs.utils import Param
# Set wandb to dryrun mode for testing
os.environ["WANDB_MODE"] = "dryrun"
# Initialize wandb
wandb.init(project="my_project", job_type="train")
@pytest.fixture
def carbs_config() -> CARBSParams:
return CARBSParams(is_wandb_logging_enabled=False, is_saved_on_every_observation=False)
@pytest.fixture
def params() -> List[Param]:
return [
Param("p1", LogSpace(scale=1), 1e-2),
Param("p2", LinearSpace(scale=2), 0),
Param("p3", LogitSpace(scale=0.5), 0.5),
]
@pytest.fixture
def carbs_instance(carbs_config: CARBSParams, params: List[Param]) -> CARBS:
return CARBS(carbs_config, params)
def test_suggest_one(carbs_instance: CARBS) -> None:
start_suggestions = len(carbs_instance.outstanding_suggestions)
suggestion = carbs_instance.suggest()
assert len(carbs_instance.outstanding_suggestions) == start_suggestions + 1
assert suggestion is not None
assert "suggestion_uuid" in suggestion.suggestion
for param in carbs_instance.params:
assert param.name in suggestion.suggestion
def test_suggest_observe_ten(carbs_instance: CARBS) -> None:
num_to_suggest = 10
start_suggestions = len(carbs_instance.outstanding_suggestions)
start_success_observations = len(carbs_instance.success_observations)
for i in range(num_to_suggest):
suggestion = carbs_instance.suggest()
observation = ObservationInParam(input=suggestion.suggestion, output=i, cost=i + 1)
carbs_instance.observe(observation)
assert len(carbs_instance.outstanding_suggestions) == start_suggestions
assert len(carbs_instance.success_observations) == start_success_observations + num_to_suggest
def test_observe(carbs_instance: CARBS) -> None:
start_success_obs = len(carbs_instance.success_observations)
start_failure_obs = len(carbs_instance.failure_observations)
obs_success = ObservationInParam(input={x.name: x.search_center for x in carbs_instance.params}, output=1, cost=1)
obs_failure = ObservationInParam(
input={x.name: x.search_center for x in carbs_instance.params}, output=1, cost=1, is_failure=True
)
obs_success_output = carbs_instance.observe(obs_success)
obs_failure_output = carbs_instance.observe(obs_failure)
assert len(carbs_instance.success_observations) == start_success_obs + 1
assert len(carbs_instance.failure_observations) == start_failure_obs + 1
def test_forget(carbs_instance: CARBS) -> None:
start_suggestions = len(carbs_instance.outstanding_suggestions)
suggestion = carbs_instance.suggest()
carbs_instance.forget_suggestion(suggestion.suggestion)
assert len(carbs_instance.outstanding_suggestions) == start_suggestions
================================================
FILE: carbs/utils.py
================================================
import math
import os
from collections import OrderedDict
from enum import Enum
from pathlib import Path
from typing import Any
from typing import Dict
from typing import List
from typing import Optional
from typing import Sequence
from typing import Sized
from typing import Tuple
from typing import Union
from pathlib import Path
from typing import Optional
import attr
import numpy as np
import seaborn as sns
import torch
import wandb
from loguru import logger
from matplotlib import pyplot as plt
from scipy.special import wofz
from torch import Tensor
from torch.distributions import Normal
from carbs.serialization import Serializable
ParamType = Union[int, float, str, bool, Enum]
ParamDictType = Dict[str, ParamType]
@attr.s(auto_attribs=True, hash=True)
class ParamSpace(Serializable):
def basic_from_param(self, value: ParamType) -> Any:
raise NotImplementedError()
def param_from_basic(self, value: Any) -> ParamType:
raise NotImplementedError()
def drop_type(self) -> Any:
return self
@attr.s(auto_attribs=True, frozen=True)
class Param:
name: str
space: ParamSpace
search_center: Union[float, int]
@attr.s(auto_attribs=True, hash=True)
class RealNumberSpace(ParamSpace):
min: float = float("-inf")
max: float = float("+inf")
scale: float = 1.0
is_integer: bool = False
rounding_factor: int = 1
def basic_from_param(self, value: ParamType) -> float:
raise NotImplementedError()
def param_from_basic(self, value: float, is_rounded: bool = True) -> ParamType:
raise NotImplementedError()
def round_tensor_in_basic(self, value: Tensor) -> Tensor:
if not self.is_integer:
return value
raise NotImplementedError()
@property
def min_bound(self):
# compensate for floats being bad bounds for integers
if self.is_integer:
return self.min - 0.1
return self.min
@property
def max_bound(self):
# compensate for floats being bad bounds for integers
if self.is_integer:
return self.max + 0.1
return self.max
@property
def plot_scale(self) -> str:
raise NotImplementedError()
@attr.s(auto_attribs=True, hash=True)
class LinearSpace(RealNumberSpace):
min: float = float("-inf")
max: float = float("+inf")
def __attrs_post_init__(self) -> None:
if self.is_integer and self.scale < 3:
logger.info(
"scale<3 on integer LinearSpace, so may not be able to search neighboring integers!"
)
def basic_from_param(self, value: ParamType) -> float:
assert isinstance(value, (int, float))
return value / self.scale
def param_from_basic(self, value: float, is_rounded: bool = True) -> float:
value = value * self.scale
if self.is_integer and is_rounded:
value = round(value / self.rounding_factor) * self.rounding_factor
return value
def round_tensor_in_basic(self, value: Tensor) -> Tensor:
if self.is_integer:
return (
torch.round(value * self.scale / self.rounding_factor)
* self.rounding_factor
/ self.scale
)
else:
return value
@property
def plot_scale(self) -> str:
return "linear"
@attr.s(auto_attribs=True, hash=True)
class LogSpace(RealNumberSpace):
min: float = 0.0
max: float = float("+inf")
base: int = 10
def basic_from_param(self, value: ParamType) -> float:
assert isinstance(value, (int, float))
if value == 0.0:
return float("-inf")
return math.log(value, self.base) / self.scale
def param_from_basic(self, value: float, is_rounded: bool = True) -> float:
value = self.base ** (value * self.scale)
if self.is_integer and is_rounded:
value = round(value / self.rounding_factor) * self.rounding_factor
return value
def round_tensor_in_basic(self, value: Tensor) -> Tensor:
if self.is_integer:
rounded_value = (
torch.round(self.base ** (value * self.scale) / self.rounding_factor)
* self.rounding_factor
)
if self.base == 10:
return torch.log10(rounded_value) / self.scale
else:
return torch.log(rounded_value) / self.scale / math.log(self.base)
else:
return value
@property
def plot_scale(self) -> str:
return "log"
@attr.s(auto_attribs=True, hash=True)
class LogitSpace(RealNumberSpace):
min: float = 0.0
max: float = 1.0
def basic_from_param(self, value: ParamType) -> float:
assert isinstance(value, (int, float))
if value == 0.0:
return float("-inf")
if value == 1.0:
return float("+inf")
return math.log10(value / (1 - value)) / self.scale
def param_from_basic(self, value: float, is_rounded: bool = True) -> float:
value = 1 / (10 ** (-value * self.scale) + 1)
return value
@property
def plot_scale(self) -> str:
return "logit"
CategoricalTuple = Tuple[int, ...]
SUGGESTION_ID_DICT_KEY = "suggestion_uuid"
def log_norm_cdf(z: Tensor):
"""
@MISC {256009,
TITLE = {Approximation of logarithm of standard normal CDF for x<0},
AUTHOR = {Isaac Asher (https://stats.stackexchange.com/users/145180/isaac-asher)},
HOWPUBLISHED = {Cross Validated},
URL = {https://stats.stackexchange.com/q/256009}
}
This contains -z**2/2 which exactly cancels the same term in prior.log_prob for EI... We can then take the log
out of the exp for the simplified form below
"""
return torch.log(wofz(-z * 1j / math.sqrt(2)).real) - z**2 / 2 + math.log(0.5)
# See explanation for expected improvement in https://distill.pub/2020/bayesian-optimization/
def expected_improvement(
mu: Tensor, variance: Tensor, best_mu: Tensor, exploration_bias: float = 0.5
) -> Tensor:
prior = Normal(0, 1)
sigma = variance.sqrt()
z = (mu - best_mu - exploration_bias) / sigma
# original form:
# ei: Tensor = sigma * torch.exp(prior.log_prob(z)) * (1 + z * torch.exp(log_norm_cdf(z) - prior.log_prob(z)))
# simplified form:
wofz_output = wofz(-z.cpu() * 1j / math.sqrt(2)).real.to(z.device)
ei: Tensor = (
sigma
* torch.exp(prior.log_prob(z))
* (1 + z * wofz_output * math.sqrt(math.pi / 2))
)
return ei
def probability_of_improvement(
mu: Tensor,
variance: Tensor,
best_mu: Tensor,
better_direction_sign: int,
exploration_bias: float = 0.0,
) -> Tensor:
prior = Normal(0, 1)
mu_improvement = (mu - best_mu) * better_direction_sign - exploration_bias
sigma = variance.sqrt()
poi: Tensor = prior.cdf(mu_improvement / sigma)
return poi
def aggregate_logical_and_across_dim(x: Tensor, dim: int = -1) -> Tensor:
"""
Takes in BoolTensor x, aggregates across dimension dim.
"""
return torch.min(torch.where(x, 1, 0), dim=dim).values > 0
def add_dict_key_prefix(input_dict: Dict[str, Any], prefix: str):
return {f"{prefix}{k}": v for k, v in input_dict.items()}
@attr.s(auto_attribs=True, collect_by_mro=True)
class ObservationInParam(Serializable):
input: ParamDictType
output: float
cost: float = 1.0
is_failure: bool = False
@attr.s(auto_attribs=True, collect_by_mro=True)
class ObservationInBasic(Serializable):
real_number_input: Tensor
output: float
cost: float = 1.0
suggestion_id: Optional[str] = None
@attr.s(auto_attribs=True, collect_by_mro=True)
class SuggestionInBasic(Serializable):
real_number_input: Tensor
log_info: Dict[str, float] = attr.Factory(dict)
class OutstandingSuggestionEstimatorEnum(Enum):
MEAN = "MEAN"
CONSTANT = "CONSTANT"
THOMPSON = "THOMPSON"
class SuggestionRedistributionMethodEnum(Enum):
NONE = "NONE"
LOG_COST_CLAMPING = "LOG_COST_CLAMPING"
@attr.s(auto_attribs=True, collect_by_mro=True)
class WandbLoggingParams(Serializable):
project_name: Optional[str] = None
group_name: Optional[str] = None
run_name: Optional[str] = None
run_id: Optional[str] = None
is_suggestion_logged: bool = True
is_observation_logged: bool = True
is_search_space_logged: bool = True
root_dir: str = "/mnt/private"
@attr.s(auto_attribs=True, collect_by_mro=True)
class CARBSParams(Serializable):
"""
Set `better_direction_sign`, `max_suggestion_cost` and `wandb_params` for your run.
I'm not aware of any situation in which we should change the other parameters -- they are mostly for testing.
"""
better_direction_sign: int = attr.field(
validator=attr.validators.in_([-1, 1]), default=1
) # 1 for maximizing, -1 for minimizing
seed: int = 0
# will do random suggestions until this many observations are made
num_random_samples: int = attr.field(validator=attr.validators.ge(1), default=4)
is_wandb_logging_enabled: bool = True
wandb_params: WandbLoggingParams = WandbLoggingParams()
checkpoint_dir: str = "checkpoints/"
s3_checkpoint_path: str = "s3://int8/checkpoints"
is_saved_on_every_observation: bool = True
initial_search_radius: float = attr.field(
validator=attr.validators.gt(0), default=0.3
)
exploration_bias: float = (
1.0 # hyperparameter biasing BO acquisition function toward exploration
)
num_candidates_for_suggestion_per_dim: int = 100
resample_frequency: int = (
5 # resample a pareto point every n observations, set 0 to disable
)
max_suggestion_cost: Optional[float] = (
None # Will not make suggestions with predicted cost above this value
)
# takes minimum cost for pareto set to be this percentile of cost data
min_pareto_cost_fraction: float = attr.field(
validator=attr.validators.ge(0), default=0.2
)
is_pareto_group_selection_conservative: bool = True
is_expected_improvement_pareto_value_clamped: bool = True
is_expected_improvement_value_always_max: bool = False
outstanding_suggestion_estimator: OutstandingSuggestionEstimatorEnum = (
OutstandingSuggestionEstimatorEnum.THOMPSON
)
@attr.s(auto_attribs=True, collect_by_mro=True)
class SurrogateModelParams(Serializable):
real_dims: int
better_direction_sign: int # 1 for maximizing, -1 for minimizing
device: str = "cpu"
min_category_observations: int = 3
scale_length: float = 1
outstanding_suggestion_estimator: OutstandingSuggestionEstimatorEnum = (
OutstandingSuggestionEstimatorEnum.MEAN
)
@attr.s(auto_attribs=True, collect_by_mro=True)
class SuggestOutput:
suggestion: ParamDictType
log: Dict[str, Any] = attr.Factory(dict)
@attr.s(auto_attribs=True, collect_by_mro=True)
class ObserveOutput:
logs: Dict[str, Any] = attr.Factory(dict)
def load_observations_from_wandb_run(
run_name: str,
prefix: str = "observation/",
add_params: Optional[ParamDictType] = None,
) -> List[ObservationInParam]:
api = wandb.Api()
run = api.run(run_name)
history_df = run.history()
observations: List[ObservationInParam] = []
for idx, row in (
history_df[[x for x in history_df.keys() if x.startswith(prefix)]]
.dropna()
.iterrows()
):
input: Dict[str, ParamType] = {}
if add_params is not None:
input.update(add_params)
output = float("inf")
for k, v in row.to_dict().items():
if k == f"{prefix}output":
output = v
else:
input[k[len(prefix) :]] = v
observations.append(ObservationInParam(input=input, output=output))
return observations
CARBS_CHECKPOINT_PREFIX = "carbs_"
CARBS_CHECKPOINT_SUFFIX = "obs.pt"
def get_checkpoint_obs_count(checkpoint_name: str) -> int:
return int(
checkpoint_name.removeprefix(CARBS_CHECKPOINT_PREFIX).removesuffix(
CARBS_CHECKPOINT_SUFFIX
)
)
def load_latest_checkpoint_from_wandb_run(
run_path: str, temp_dir: Optional[str] = None
) -> str:
api = wandb.Api()
run = api.run(run_path)
checkpoint_filenames = [
file.name
for file in run.files()
if file.name.startswith(CARBS_CHECKPOINT_PREFIX)
and file.name.endswith(CARBS_CHECKPOINT_SUFFIX)
]
checkpoint_filenames = sorted(checkpoint_filenames, key=get_checkpoint_obs_count)
latest_checkpoint_filename = checkpoint_filenames[-1]
return load_checkpoint_from_wandb_run(
run_path, latest_checkpoint_filename, temp_dir
)
def load_checkpoint_from_wandb_run(
run_path: str, checkpoint_filename: str, temp_dir: Optional[str] = None
) -> str:
if temp_dir is None:
temp_dir = f"/tmp/carbs/{run_path}"
os.makedirs(temp_dir, exist_ok=True)
checkpoint_path = wandb.restore(
checkpoint_filename, run_path=run_path, replace=True, root=temp_dir
)
assert checkpoint_path is not None, "Could not load checkpoint"
return checkpoint_path.name
def assert_empty(x: Sized, message: str = "unexpected elements") -> None:
assert len(x) == 0, f"{message}: {x}"
def ordered_dict_index(od: OrderedDict, value: Any) -> int:
for i, k in enumerate(od.keys()):
if k == value:
return i
raise KeyError(f"{value} not found in {od}")
# All members of an ObservationGroup have the same parameters
ObservationGroup = Tuple[ObservationInBasic, ...]
def group_observations(
observations_in_basic: List[ObservationInBasic],
) -> Tuple[ObservationGroup, ...]:
"""
Gets observations grouped by matching input params
"""
outputs: List[ObservationGroup] = []
observations = observations_in_basic.copy()
observations.sort(key=lambda x: tuple(v.item() for v in x.real_number_input))
while len(observations) > 0:
obs = observations.pop()
nearby_obs = [obs]
for i in range(len(observations) - 1, -1, -1):
if torch.all(
torch.isclose(observations[i].real_number_input, obs.real_number_input)
):
nearby_obs.append(observations.pop(i))
outputs.append(tuple(nearby_obs))
return tuple(outputs)
def observation_group_cost(group: Sequence[ObservationInBasic]) -> float:
return sum(obs.cost for obs in group) / len(group)
def observation_group_output(group: Sequence[ObservationInBasic]) -> float:
return sum(obs.output for obs in group) / len(group)
def pareto_area_from_groups(obs_groups: Tuple[ObservationGroup, ...]) -> float:
if len(obs_groups) < 2:
return 0
last_cost = observation_group_cost(obs_groups[0])
last_output = observation_group_output(obs_groups[0])
total_area = 0.0
for group in obs_groups[1:]:
new_cost = observation_group_cost(group)
new_output = observation_group_output(group)
total_area += (new_cost - last_cost) * last_output
last_cost = new_cost
last_output = new_output
return total_area
def get_pareto_groups(
grouped_observations: Tuple[ObservationGroup, ...],
min_pareto_cost_fraction: float,
better_direction_sign: int,
) -> Tuple[ObservationGroup, ...]:
min_pareto_cost = np.quantile(
np.array([x.cost for group in grouped_observations for x in group]),
min_pareto_cost_fraction,
)
observations_below_min_threshold = [
group
for group in grouped_observations
if observation_group_cost(group) <= min_pareto_cost
]
group_output_pos_better = (
lambda x: observation_group_output(x) * better_direction_sign
)
first_pareto_group = max(
observations_below_min_threshold, key=group_output_pos_better
)
remaining_observations = [
group
for group in grouped_observations
if observation_group_cost(group) > min_pareto_cost
]
remaining_observations.sort(key=observation_group_cost)
pareto_groups: List[ObservationGroup] = [first_pareto_group]
best_output = group_output_pos_better(first_pareto_group)
for obs_group in remaining_observations:
mean_output = group_output_pos_better(obs_group)
if mean_output > best_output:
pareto_groups.append(obs_group)
best_output = mean_output
return tuple(pareto_groups)
def get_pareto_groups_conservative(
grouped_observations: Tuple[ObservationGroup, ...],
min_pareto_cost_fraction: float,
better_direction_sign: int,
) -> Tuple[ObservationGroup, ...]:
"""
Just like get_pareto_groups but prefers groups with multiple samples. A single sample can only be "better" if
it is better than the max of the previous group. However, a multiple sample group is better if the mean is higher.
For example, with better_direction_sign=1:
Group A: ((output=0,cost=1),(output=1,cost=1)) # mean=0.5
Group B: ((output=0.6,cost=2)) # mean=0.6
Group C: ((output=0.55,cost=3),(output=0.75,cost=3)) # mean=0.65
Group D: ((output=0.8,cost=4)) # mean=0.8
With the normal pareto algorithm, A, B, C and D would all be in the pareto front. With this algorithm, groups
A, C and D will be in the pareto front. B will not be included because it is not greater than the max of group A.
The purpose of this is to reduce thrash in which groups are used as search centers in noisy areas of search space.
"""
min_pareto_cost = np.quantile(
np.array([x.cost for group in grouped_observations for x in group]),
min_pareto_cost_fraction,
)
observations_below_min_threshold = [
group
for group in grouped_observations
if observation_group_cost(group) <= min_pareto_cost
]
resampled_observations_below_min_threshold = [
group for group in observations_below_min_threshold if len(group) > 1
]
# Only use resampled observations if there are any
if len(resampled_observations_below_min_threshold) > 0:
observations_below_min_threshold = resampled_observations_below_min_threshold
group_output_pos_better = (
lambda x: observation_group_output(x) * better_direction_sign
)
max_group_output_pos_better = lambda x: max(
[obs.output * better_direction_sign for obs in x]
)
first_pareto_group = max(
observations_below_min_threshold, key=group_output_pos_better
)
remaining_observations = [
group
for group in grouped_observations
if observation_group_cost(group) > min_pareto_cost
]
remaining_observations.sort(key=observation_group_cost)
pareto_groups: List[ObservationGroup] = [first_pareto_group]
best_output = group_output_pos_better(first_pareto_group)
best_output_max = max_group_output_pos_better(first_pareto_group)
for obs_group in remaining_observations:
mean_output = group_output_pos_better(obs_group)
# If only one sample, it must be better than the max of the last group
# Otherwise, only must have better mean
if (len(obs_group) > 1 and mean_output > best_output) or (
len(obs_group) == 1 and mean_output > best_output_max
):
pareto_groups.append(obs_group)
best_output = mean_output
best_output_max = max_group_output_pos_better(obs_group)
return tuple(pareto_groups)
def get_pareto_curve_plot(
observations: List[ObservationInBasic],
pareto_groups: Tuple[ObservationGroup, ...],
save_dir: Optional[str] = None,
obs_count: Optional[int] = None,
) -> Optional[str]:
sns.set_theme(style="whitegrid")
plt.clf()
pareto_set = [obs for group in pareto_groups for obs in group]
plt.scatter(
[x.cost for x in pareto_set],
[x.output for x in pareto_set],
c="black",
s=100,
# marker="o",
)
plt.plot(
[observation_group_cost(g) for g in pareto_groups],
[observation_group_output(g) for g in pareto_groups],
linewidth=4,
color="black",
)
plt.scatter(
[x.cost for x in observations],
[x.output for x in observations],
c=list(range(len(observations))),
cmap="plasma",
)
plt.xscale("log")
if obs_count is None:
obs_count = len(observations)
plt.title(f"Pareto curve at {obs_count} obs")
plt.ylabel("performance")
plt.xlabel("cost")
if save_dir is not None:
img_path = Path(save_dir) / f"pareto_curve_{obs_count}.png"
plt.savefig(img_path)
return str(img_path)
return None
================================================
FILE: excluded.txt
================================================
setup.py
================================================
FILE: notebooks/analyze_carbs.sync.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"name": ""
}
},
"outputs": [],
"source": [
"import math\n",
"import warnings\n",
"from functools import partial\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import seaborn as sns\n",
"import torch\n",
"import wandb\n",
"\n",
"%matplotlib inline\n",
"from carbs import CARBS\n",
"from carbs import ObservationInBasic\n",
"from carbs import get_pareto_curve_plot\n",
"from carbs import load_latest_checkpoint_from_wandb_run\n",
"from carbs import observation_group_cost\n",
"from carbs import observation_group_output\n",
"from matplotlib import MatplotlibDeprecationWarning\n",
"from matplotlib.ticker import LogFormatter\n",
"from scipy.interpolate import interp1d\n",
"from sklearn.linear_model import LinearRegression\n",
"\n",
"from research.quarantine.abe.plot_helpers import set_axes_style"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"api = wandb.Api()\n",
"run_path = \"sourceress/abe__bones/2i8gnlf9\"\n",
"run = api.run(run_path)\n",
"history_df = run.history()\n",
"history_df = history_df.replace(\"Infinity\", float(\"-inf\"))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"carbs_checkpoint_path = load_latest_checkpoint_from_wandb_run(run_path)\n",
"carbs = CARBS.load_from_file(carbs_checkpoint_path)\n",
"search_vars = list(carbs._real_number_space_by_name.keys())\n",
"search_space_scale = {k: v.plot_scale for k, v in carbs._real_number_space_by_name.items()}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"# Performance\n",
"\n",
"is_best_shown = True\n",
"is_resampled_shown = True\n",
"is_search_space_shown = False\n",
"performance_min, performance_max = (3, 6)\n",
"\n",
"# sns.set(rc={\"figure.figsize\": (12, 4)})\n",
"# sns.set_theme(style=\"whitegrid\")\n",
"cmap = sns.color_palette(\"viridis\" if carbs.params.better_direction_sign > 0 else \"viridis_r\", as_cmap=True)\n",
"\n",
"output_observation_df = history_df[[\"observation_count\", f\"observation/output\"]].dropna()\n",
"observation_x, observation_y = output_observation_df.to_numpy().T\n",
"\n",
"if is_resampled_shown:\n",
" resampled_df = history_df[\n",
" [\"observation_count\", \"best_resampled_observation/output_mean\", \"best_resampled_observation/output_std_dev\"]\n",
" ].dropna()\n",
" resampled_x, resampled_mean, resampled_std = resampled_df.to_numpy().T\n",
" plt.plot(resampled_x, resampled_mean, color=\"green\", linewidth=2, label=\"best parameters mean\", linestyle=\"dotted\")\n",
" plt.fill_between(\n",
" resampled_x,\n",
" resampled_mean - resampled_std,\n",
" resampled_mean + resampled_std,\n",
" color=\"green\",\n",
" alpha=0.1,\n",
" label=\"best parameters variance\",\n",
" )\n",
"if is_best_shown:\n",
" output_best_observation_df = history_df[[\"observation_count\", f\"best_observation/output\"]].dropna()\n",
" best_observation_x, best_observation_y = output_best_observation_df.to_numpy().T\n",
" plt.plot(best_observation_x, best_observation_y, linestyle=\"dashed\", linewidth=2, label=\"best single observation\")\n",
"plt.scatter(\n",
" observation_x,\n",
" observation_y,\n",
" c=observation_y,\n",
" s=20,\n",
" label=\"observation\",\n",
" cmap=cmap,\n",
" vmin=performance_min,\n",
" vmax=performance_max,\n",
")\n",
"plt.title(\"Combined performance metric\")\n",
"plt.xlabel(\"Observation count\")\n",
"plt.ylabel(\"Performance\")\n",
"plt.ylim(performance_min, performance_max)\n",
"plt.legend()\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"\n",
"\"\"\"\n",
"Convergence by parameter\n",
"\n",
"Color matches above plot: lighter yellow = better performance.\n",
"\"Best\" observation shows input parameter for the best output performance so far\n",
"\"\"\"\n",
"for search_var in search_vars[:]:\n",
" search_var_observation_df = history_df[\n",
" [\"observation_count\", f\"observation/{search_var}\", \"observation/output\"]\n",
" ].dropna()\n",
" observation_x, observation_y, observation_z = search_var_observation_df.to_numpy().T\n",
"\n",
" if is_best_shown:\n",
" search_var_best_observation_df = history_df[[\"observation_count\", f\"best_observation/{search_var}\"]].dropna()\n",
" best_observation_x, best_observation_y = search_var_best_observation_df.to_numpy().T\n",
" plt.plot(\n",
" best_observation_x, best_observation_y, linestyle=\"dashed\", linewidth=2, label=\"best single observation\"\n",
" )\n",
" if is_resampled_shown:\n",
" search_var_resampled_observation_df = history_df[\n",
" [\"observation_count\", f\"best_resampled_observation/{search_var}\"]\n",
" ].dropna()\n",
" resampled_observation_x, resampled_observation_y = search_var_resampled_observation_df.to_numpy().T\n",
" plt.plot(\n",
" resampled_observation_x,\n",
" resampled_observation_y,\n",
" linestyle=\"dotted\",\n",
" color=\"green\",\n",
" linewidth=2,\n",
" label=\"best parameter value\",\n",
" )\n",
" search_var_name = search_var.replace(\"_\", \" \").replace(\"pdrop\", \"dropout\")\n",
"\n",
" plt.scatter(\n",
" observation_x,\n",
" observation_y,\n",
" c=observation_z,\n",
" s=20,\n",
" label=\"observation\",\n",
" cmap=cmap,\n",
" vmin=performance_min,\n",
" vmax=performance_max,\n",
" )\n",
" plt.title(f\"{search_var_name} convergence\")\n",
" plt.xlabel(\"Observation count\")\n",
" plt.ylabel(search_var)\n",
" plt.yscale(search_space_scale[search_var])\n",
" plt.legend()\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"# Pareto curve plot\n",
"\n",
"pareto_groups = carbs._get_pareto_groups(True)\n",
"\n",
"get_pareto_curve_plot(carbs.observations_in_basic, pareto_groups, obs_count=carbs.observation_count)\n",
"plt.ylim(performance_min, performance_max)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"\n",
"surrogate_model = carbs.get_surrogate_model()\n",
"surrogate_model.fit_observations(carbs.observations_in_basic)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"# Get loguniform inputs by interpolating the pareto points from the random sampling\n",
"num_uniform_inputs = 30\n",
"num_contour_levels = 10\n",
"\n",
"pareto_costs = [observation_group_cost(x) for x in pareto_groups]\n",
"pareto_logcosts = [math.log(x) for x in pareto_costs]\n",
"pareto_outputs = [observation_group_output(x) for x in pareto_groups]\n",
"uniform_logcosts = np.linspace(min(pareto_logcosts), max(pareto_logcosts), num=num_uniform_inputs)\n",
"pareto_inputs = torch.stack([x[0].real_number_input for x in pareto_groups], dim=0)\n",
"\n",
"reg = LinearRegression()\n",
"reg.fit(np.array(pareto_logcosts)[:, None], pareto_inputs)\n",
"\n",
"uniform_pareto_inputs = torch.from_numpy(reg.predict(np.array(uniform_logcosts)[:, None])).float()\n",
"\n",
"# Then evaluate those on the surrogate\n",
"uniform_surrogate_outputs = surrogate_model.observe_surrogate(uniform_pareto_inputs)\n",
"# print(f\"Outputs: {uniform_surrogate_outputs.target_estimate}\")\n",
"# print(f\"Cost: {uniform_surrogate_outputs.cost_estimate}\")\n",
"\n",
"# Filter observations to those in the range of the pareto front\n",
"observations_in_basic = [\n",
" x for x in carbs.observations_in_basic if x.cost >= min(pareto_costs) and x.cost <= max(pareto_costs)\n",
"]\n",
"obs_cost = [x.cost for x in observations_in_basic]\n",
"obs_output = [x.output for x in observations_in_basic]\n",
"contour_levels = np.linspace(min(pareto_outputs), max(pareto_outputs), num_contour_levels)\n",
"vmin, vmax = min(pareto_outputs), max(pareto_outputs)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"\n",
"import matplotlib as mpl\n",
"import matplotlib.pyplot as plt\n",
"from matplotlib import cm\n",
"\n",
"scalar_map = cm.ScalarMappable(norm=mpl.colors.Normalize(vmin=vmin, vmax=vmax), cmap=cmap)\n",
"\n",
"interp_pareto_value = interp1d(uniform_logcosts, uniform_pareto_inputs, axis=0, fill_value=\"extrapolate\")\n",
"observation_pareto_distance = [\n",
" torch.norm(torch.from_numpy(interp_pareto_value(np.log(x.cost))).float() - x.real_number_input).item()\n",
" for x in observations_in_basic\n",
"]\n",
"search_radius = carbs.params.initial_search_radius\n",
"rescaled_observation_pareto_distance = [search_radius / (search_radius + x) for x in observation_pareto_distance]\n",
"observation_marker_size = [200 * x for x in rescaled_observation_pareto_distance]\n",
"obs_color = [\n",
" scalar_map.to_rgba(output)[:3] + (min(2 * alpha, 1),)\n",
" for output, alpha in zip(obs_output, rescaled_observation_pareto_distance)\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"pareto_set = set()\n",
"\n",
"\n",
"def obs_to_key(obs: ObservationInBasic):\n",
" return tuple(obs.real_number_input.tolist())\n",
"\n",
"\n",
"for group in pareto_groups:\n",
" for obs in group:\n",
" pareto_set.add(obs_to_key(obs))\n",
"\n",
"obs_is_in_pareto_set = [obs_to_key(obs) in pareto_set for obs in observations_in_basic]\n",
"edgecolors = [\"black\" if x else \"none\" for x in obs_is_in_pareto_set]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"\n",
"class CustomLogFormatter(LogFormatter):\n",
" def _num_to_string(self, x, vmin, vmax) -> str:\n",
" return f\"{int(x)}\""
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": false
},
"outputs": [],
"source": [
"\n",
"base_two_search_vars = {\n",
" \"model.n_layers\",\n",
" \"model.n_heads\",\n",
" \"model.kv_size\",\n",
" \"model.ffw_size\",\n",
"}\n",
"\n",
"# Parameter variation along pareto front\n",
"# TODO: add failed observations, predict cost from GP model, make red x\n",
"sel_search_vars = search_vars\n",
"fig, axs = plt.subplots(\n",
" nrows=np.ceil((1 + len(sel_search_vars)) / 2).astype(int),\n",
" ncols=2,\n",
" figsize=(14, 4 * (1 + len(sel_search_vars)) // 2),\n",
" sharex=True,\n",
")\n",
"fig.tight_layout()\n",
"fig.subplots_adjust(hspace=0.2, wspace=0.2)\n",
"axs = axs.flatten()\n",
"warnings.simplefilter(\"ignore\", MatplotlibDeprecationWarning)\n",
"for search_var_idx, search_var in enumerate(sel_search_vars):\n",
" # search_var_idx += 10\n",
" param_from_basic = carbs._real_number_space_by_name[search_var].param_from_basic\n",
" obs_search_var = [param_from_basic(x.real_number_input[search_var_idx].item()) for x in observations_in_basic]\n",
"\n",
" num_search_var_grid_points = 50\n",
" search_var_linspace_in_basic = torch.linspace(\n",
" min([x.real_number_input[search_var_idx].item() for x in observations_in_basic]),\n",
" max([x.real_number_input[search_var_idx].item() for x in observations_in_basic]),\n",
" steps=num_search_var_grid_points,\n",
" )\n",
" # search_var_linspace_in_basic\n",
" input_grid = uniform_pareto_inputs.repeat(num_search_var_grid_points, 1, 1)\n",
" for i in range(num_uniform_inputs):\n",
" input_grid[:, i, search_var_idx] = search_var_linspace_in_basic\n",
" input_grid_flat = input_grid.view(-1, carbs.real_dim)\n",
" surrogate_output_on_flat_grid = surrogate_model.observe_surrogate(input_grid_flat)\n",
" cost_grid = surrogate_output_on_flat_grid.cost_estimate.view(num_search_var_grid_points, num_uniform_inputs).cpu()\n",
" output_grid = surrogate_output_on_flat_grid.target_estimate.view(\n",
" num_search_var_grid_points, num_uniform_inputs\n",
" ).cpu()\n",
" search_var_grid = input_grid[:, :, search_var_idx].cpu()\n",
" search_var_grid.apply_(partial(param_from_basic, is_rounded=False))\n",
"\n",
" ax = axs[search_var_idx]\n",
" contour_plot = ax.contour(\n",
" cost_grid,\n",
" search_var_grid,\n",
" output_grid,\n",
" cmap=cmap,\n",
" vmin=vmin,\n",
" vmax=vmax,\n",
" levels=contour_levels,\n",
" )\n",
" pareto_search_var = [\n",
" param_from_basic(x.item(), is_rounded=False) for x in uniform_pareto_inputs[:, search_var_idx]\n",
" ]\n",
" (pareto_line,) = ax.plot(\n",
" np.exp(uniform_logcosts), pareto_search_var, color=\"black\", linewidth=2, linestyle=\"dashed\"\n",
" )\n",
"\n",
" scatter_plot = ax.scatter(\n",
" obs_cost,\n",
" obs_search_var,\n",
" c=obs_color,\n",
" s=observation_marker_size,\n",
" edgecolors=edgecolors,\n",
" )\n",
" if \".\" in search_var:\n",
" ax.set_ylabel(search_var.split(\".\")[-1])\n",
" else:\n",
" ax.set_ylabel(search_var)\n",
" if search_var in base_two_search_vars:\n",
" ax.set_yscale(\"log\", base=2)\n",
" ax.yaxis.set_major_formatter(CustomLogFormatter(base=2.0, labelOnlyBase=True))\n",
" ax.yaxis.set_minor_formatter(CustomLogFormatter(base=2.0, labelOnlyBase=False, minor_thresholds=(10.0, 0.1)))\n",
" ax.xaxis.set_major_formatter(LogFormatter(labelOnlyBase=True))\n",
" ax.xaxis.set_minor_formatter(LogFormatter(labelOnlyBase=False, minor_thresholds=(10.0, 0.1)))\n",
" else:\n",
" ax.set_yscale(search_space_scale[search_var])\n",
" ax.set_xscale(\"log\")\n",
" ax.set_xlabel(\"Cost\")\n",
" ax.set_xlim(min(pareto_costs), max(pareto_costs))\n",
" set_axes_style(ax, grid=\"both\")\n",
"\n",
"fig.legend([\"Pareto front (fit)\", \"Observations\"])\n",
"# cbar_ax = fig.add_axes([0.96, 0.2, 0.02, 0.6])\n",
"cbar = fig.colorbar(mappable=scalar_map, location=\"bottom\") # , cax=cbar_ax\n",
"for ax in axs[len(sel_search_vars) : len(axs)]:\n",
" fig.delaxes(ax)\n",
"# fig.colorbar()\n",
"# cbar = fig.colorbar(contour_plot)\n",
"cbar.set_label(\"Validation Cross Entropy\")\n",
"plt.savefig(\"/home/user/hyperspace_appendix_plot_2.pdf\", bbox_inches=\"tight\")\n",
"plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
================================================
FILE: notebooks/carbs_demo.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"pycharm": {
"is_executing": true
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running CARBS with params CARBSParams(better_direction_sign=-1, seed=0, num_random_samples=4, is_wandb_logging_enabled=False, wandb_params=WandbLoggingParams(project_name=None, group_name=None, run_name=None, run_id=None, is_suggestion_logged=True, is_observation_logged=True, is_search_space_logged=True, root_dir='/mnt/private'), is_saved_on_every_observation=True, initial_search_radius=0.3, exploration_bias=1.0, num_candidates_for_suggestion_per_dim=100, resample_frequency=-1, max_cost=None, min_pareto_cost_fraction=0.2, is_pareto_group_selection_conservative=True, is_expected_improvement_pareto_value_clamped=True, is_expected_improvement_value_always_max=False, outstanding_suggestion_estimator=<OutstandingSuggestionEstimatorEnum.THOMPSON: 'THOMPSON'>)\n",
"Observation 1\n",
"Observed lr=3.93e-05, epochs=15, output 67.511\n",
"Best lr=3.93e-05, epochs=15, output 67.511\n",
"Observation 2\n",
"Observed lr=2.84e-04, epochs=33, output 4.718\n",
"Best lr=2.84e-04, epochs=33, output 4.718\n",
"Observation 3\n",
"Observed lr=5.43e-05, epochs=6, output 136.616\n",
"Best lr=2.84e-04, epochs=33, output 4.718\n",
"Observation 4\n",
"Observed lr=1.98e-04, epochs=15, output 16.960\n",
"Best lr=2.84e-04, epochs=33, output 4.718\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/venv/lib/python3.9/site-packages/pyro/contrib/gp/models/gpr.py:81: UserWarning: torch.cholesky is deprecated in favor of torch.linalg.cholesky and will be removed in a future PyTorch release.\n",
"L = torch.cholesky(A)\n",
"should be replaced with\n",
"L = torch.linalg.cholesky(A)\n",
"and\n",
"U = torch.cholesky(A, upper=True)\n",
"should be replaced with\n",
"U = torch.linalg.cholesky(A).mH().\n",
"This transform will produce equivalent results for all valid (symmetric positive definite) inputs. (Triggered internally at ../aten/src/ATen/native/BatchLinearAlgebra.cpp:1744.)\n",
" Lff = Kff.cholesky()\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Observation 5\n",
"Observed lr=2.76e-04, epochs=82, output 1.993\n",
"Best lr=2.76e-04, epochs=82, output 1.993\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/venv/lib/python3.9/site-packages/pyro/contrib/gp/util.py:109: UserWarning: torch.triangular_solve is deprecated in favor of torch.linalg.solve_triangularand will be removed in a future PyTorch release.\n",
"torch.linalg.solve_triangular has its arguments reversed and does not return a copy of one of the inputs.\n",
"X = torch.triangular_solve(B, A).solution\n",
"should be replaced with\n",
"X = torch.linalg.solve_triangular(A, B). (Triggered internally at ../aten/src/ATen/native/BatchLinearAlgebra.cpp:2183.)\n",
" Lffinv_pack = pack.triangular_solve(Lff, upper=False)[0]\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Observation 6\n",
"Observed lr=2.64e-04, epochs=163, output 1.118\n",
"Best lr=2.64e-04, epochs=163, output 1.118\n",
"Observation 7\n",
"Observed lr=3.01e-04, epochs=207, output 0.718\n",
"Best lr=3.01e-04, epochs=207, output 0.718\n",
"Observation 8\n",
"Observed lr=2.45e-04, epochs=207, output 1.012\n",
"Best lr=3.01e-04, epochs=207, output 0.718\n",
"Observation 9\n",
"Observed lr=4.58e-04, epochs=173, output 0.437\n",
"Best lr=4.58e-04, epochs=173, output 0.437\n",
"Observation 10\n",
"Observed lr=7.14e-04, epochs=111, output 0.137\n",
"Best lr=7.14e-04, epochs=111, output 0.137\n"
]
}
],
"source": [
"# type: ignore\n",
"import math\n",
"import sys\n",
"from collections import OrderedDict\n",
"\n",
"import numpy as np\n",
"from loguru import logger\n",
"\n",
"from carbs import CARBS\n",
"from carbs import CARBSParams\n",
"from carbs import LogSpace\n",
"from carbs import LogitSpace\n",
"from carbs import ObservationInParam\n",
"from carbs import ParamDictType\n",
"from carbs import Param\n",
"\n",
"logger.remove()\n",
"logger.add(sys.stdout, level=\"DEBUG\", format=\"{message}\")\n",
"\n",
"\n",
"\n",
"def run_test_fn(input_in_param: ParamDictType):\n",
" # A noisy function minimized at lr=1e-3, max hidden_dim\n",
" result = (math.log10(input_in_param[\"learning_rate\"]) + 3) ** 2 * 512 / input_in_param[\n",
" \"epochs\"\n",
" ] + np.random.uniform() * 0.1\n",
" return result\n",
"\n",
"param_spaces = [\n",
" Param(name=\"learning_rate\", space=LogSpace(scale=0.5), search_center=1e-4),\n",
" Param(name=\"momentum\", space=LogitSpace(), search_center=0.9),\n",
" Param(name=\"epochs\", space=LogSpace(is_integer=True, min=2, max=512), search_center=10),\n",
"]\n",
"\n",
"carbs_params = CARBSParams(\n",
" better_direction_sign=-1,\n",
" is_wandb_logging_enabled=False,\n",
" resample_frequency=0,\n",
")\n",
"carbs = CARBS(carbs_params, param_spaces)\n",
"for i in range(10):\n",
" suggestion = carbs.suggest().suggestion\n",
" observed_value = run_test_fn(suggestion)\n",
" obs_out = carbs.observe(ObservationInParam(input=suggestion, output=observed_value, cost=suggestion[\"epochs\"]))\n",
" logger.info(f\"Observation {obs_out.logs['observation_count']}\")\n",
" logger.info(\n",
" f\"Observed lr={obs_out.logs['observation/learning_rate']:.2e}, \"\n",
" f\"epochs={obs_out.logs['observation/epochs']}, \"\n",
" f\"output {obs_out.logs['observation/output']:.3f}\"\n",
" )\n",
" logger.info(\n",
" f\"Best lr={obs_out.logs['best_observation/learning_rate']:.2e}, \"\n",
" f\"epochs={obs_out.logs['best_observation/epochs']}, \"\n",
" f\"output {obs_out.logs['best_observation/output']:.3f}\"\n",
" )"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.5"
}
},
"nbformat": 4,
"nbformat_minor": 1
}
================================================
FILE: notebooks/carbs_simple_2d.sync.ipynb
================================================
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Running CARBS with params CARBSParams(better_direction_sign=-1, seed=0, num_random_samples=4, is_wandb_logging_enabled=False, wandb_params=WandbLoggingParams(project_name=None, group_name=None, run_name=None, run_id=None, is_suggestion_logged=True, is_observation_logged=True, is_search_space_logged=True, root_dir='/mnt/private'), is_saved_on_every_observation=True, initial_search_radius=0.5, exploration_bias=1.0, num_candidates_for_suggestion_per_dim=100, resample_frequency=5, max_cost=None, search_distribution_function=<SearchDistributionFunctionEnum.NORMAL: 'NORMAL'>, min_pareto_cost_fraction=0.2, is_pareto_group_selection_conservative=True, is_expected_improvement_pareto_value_clamped=True, is_expected_improvement_value_always_max=False, outstanding_suggestion_estimator=<OutstandingSuggestionEstimatorEnum.THOMPSON: 'THOMPSON'>)\n"
]
}
],
"source": [
"# type: ignore\n",
"import math\n",
"from collections import OrderedDict\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import torch\n",
"from carbs import CARBS\n",
"from carbs import CARBSParams\n",
"from carbs import LogSpace\n",
"from carbs import ObservationInParam\n",
"from carbs import ParamDictType\n",
"\n",
"from avalon.common.log_utils import configure_local_logger\n",
"\n",
"# logger.remove()\n",
"# logger.add(sys.stdout, level=\"DEBUG\", format=\"{message}\")\n",
"configure_local_logger()\n",
"\n",
"param_space_by_name = OrderedDict(\n",
" [\n",
" (\"learning_rate\", LogSpace()),\n",
" # (\"momentum\", LogitSpace()),\n",
" (\"hidden_dim\", LogSpace(is_integer=True, min=2, max=512)),\n",
" ]\n",
")\n",
"\n",
"\n",
"def run_test_fn(input_in_param: ParamDictType):\n",
" # A noisy function minimized at lr=1e-3, max hidden_dim\n",
" # if input_in_param[\"learning_rate\"] > 1e-3:\n",
" # return float(\"inf\")\n",
" result = ((math.log10(input_in_param[\"learning_rate\"]) + 3) ** 2 + 0.2) * 512 / input_in_param[\n",
" \"hidden_dim\"\n",
" ] + np.random.uniform() * 0.1\n",
" return result\n",
"\n",
"\n",
"initial_params = {\"learning_rate\": 1e-5, \"hidden_dim\": 32} # , \"momentum\": 0.9}\n",
"\n",
"carbs_params = CARBSParams(\n",
" better_direction_sign=-1,\n",
" is_wandb_logging_enabled=False,\n",
" initial_search_radius=0.5,\n",
" # resample_frequency=-1,\n",
")\n",
"carbs = CARBS(carbs_params, param_space_by_name)\n",
"carbs.set_search_center(initial_params)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"\n",
"\n",
"def plot_and_suggest():\n",
" if len(carbs.observations_in_basic) < carbs.params.num_random_samples:\n",
" return carbs.suggest()\n",
" pareto_observations = np.stack([x.real_number_input for x in carbs._get_pareto_set()])\n",
" observations = np.stack([x.real_number_input for x in carbs.observations_in_basic])\n",
" min_lr, min_hid = observations.min(axis=0)\n",
" max_lr, max_hid = observations.max(axis=0)\n",
" dlr, dhid = (max_lr - min_lr) / 20, (max_hid - min_hid) / 20\n",
" lr_range = np.arange(min_lr - dlr * 4, max_lr + dlr * 10.5, dlr)\n",
" hid_range = np.arange(min_hid - dhid * 4, max_hid + dhid * 10.5, dhid)\n",
" nlr, nhid = lr_range.shape[0], hid_range.shape[0]\n",
" assert nlr == nhid, \"Aaargh this is broken for some reason\"\n",
" grid_inputs = np.stack(np.meshgrid(lr_range, hid_range), axis=-1).reshape((-1, 2))\n",
" (\n",
" surrogate_model_outputs,\n",
" probabilities,\n",
" ei_value,\n",
" acquisition_function_value,\n",
" ) = carbs.evaluate_candidates(torch.from_numpy(grid_inputs).to(torch.float), (0,), is_rounding_candidates=True)\n",
"\n",
" fig, axs = plt.subplots(1, 2, constrained_layout=True)\n",
" fig.set_size_inches(12, 6)\n",
" cs = axs[0].contourf(\n",
" grid_inputs[:, 0].reshape((nlr, nhid)),\n",
" grid_inputs[:, 1].reshape((nlr, nhid)),\n",
" surrogate_model_outputs.surrogate_output.cpu().numpy().reshape((nlr, nhid)),\n",
" )\n",
" axs[0].scatter(observations[:, 0], observations[:, 1], color=\"orange\")\n",
" axs[0].scatter(pareto_observations[:, 0], pareto_observations[:, 1], color=\"white\")\n",
" fig.colorbar(cs, ax=axs[0])\n",
" axs[0].set_xlim((lr_range[0], lr_range[-1]))\n",
" axs[0].set_ylim((hid_range[0], hid_range[-1]))\n",
" axs[0].set_title(\"Surrogate function value\")\n",
"\n",
" cs = axs[1].contourf(\n",
" grid_inputs[:, 0].reshape((nlr, nhid)),\n",
" grid_inputs[:, 1].reshape((nlr, nhid)),\n",
" torch.log(torch.clamp(ei_value / surrogate_model_outputs.cost_estimate, min=1e-10))\n",
" .cpu()\n",
" .numpy()\n",
" .reshape((nlr, nhid)),\n",
" )\n",
" prob_max = probabilities.max().item()\n",
" axs[1].contour(\n",
" grid_inputs[:, 0].reshape((nlr, nhid)),\n",
" grid_inputs[:, 1].reshape((nlr, nhid)),\n",
" probabilities.cpu().numpy().reshape((nlr, nhid)),\n",
" levels=[prob_max / 9, prob_max / 1.5],\n",
" colors=\"w\",\n",
" linestyles=(\"dotted\", \"dashed\"),\n",
" )\n",
" # if surrogate_model_outputs.success_probability.min()[0] < 0.5:\n",
" # axs[1].contour(\n",
" # grid_inputs[:, 0].reshape((nlr, nhid)),\n",
" # grid_inputs[:, 1].reshape((nlr, nhid)),\n",
" # surrogate_model_outputs.success_probability.numpy().reshape((nlr, nhid)),\n",
" # levels=[0.5],\n",
" # colors=\"r\",\n",
" # linestyles=\"dashed\",\n",
" # )\n",
" axs[1].scatter(observations[:, 0], observations[:, 1], color=\"orange\")\n",
" axs[1].scatter(pareto_observations[:, 0], pareto_observations[:, 1], color=\"white\")\n",
" fig.colorbar(cs, ax=axs[1])\n",
" axs[1].set_xlim((lr_range[0], lr_range[-1]))\n",
" axs[1].set_ylim((hid_range[0], hid_range[-1]))\n",
" axs[1].set_title(\"Acquisiton function value\")\n",
"\n",
" suggest_out = carbs.suggest()\n",
"\n",
" assert len(carbs.outstanding_suggestions) == 1\n",
"\n",
" suggestion_in_basic = list(carbs.outstanding_suggestions.values())[0].real_number_input\n",
"\n",
" axs[0].scatter(\n",
" [suggestion_in_basic[0]], [suggestion_in_basic[1]], color=\"pink\", marker=\"*\", s=500, edgecolors=\"black\"\n",
" )\n",
" axs[1].scatter(\n",
" [suggestion_in_basic[0]], [suggestion_in_basic[1]], color=\"pink\", marker=\"*\", s=500, edgecolors=\"black\"\n",
" )\n",
" if len(carbs.failed_observations_in_basic) > 0:\n",
" failed_observations = np.stack([x.real_number_input for x in carbs.failed_observations_in_basic])\n",
" axs[0].scatter(failed_observations[:, 0], failed_observations[:, 1], color=\"red\", marker=\"x\")\n",
" axs[1].scatter(failed_observations[:, 0], failed_observations[:, 1], color=\"red\", marker=\"x\")\n",
" plt.show()\n",
" return suggest_out"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/opt/venv/lib/python3.9/site-packages/pyro/contrib/gp/models/gpr.py:81: UserWarning: torch.cholesky is deprecated in favor of torch.linalg.cholesky and will be removed in a future PyTorch release.\n",
"L = torch.cholesky(A)\n",
"should be replaced with\n",
"L = torch.linalg.cholesky(A)\n",
"and\n",
"U = torch.cholesky(A, upper=True)\n",
"should be replaced with\n",
"U = torch.linalg.cholesky(A).mH().\n",
"This transform will produce equivalent results for all valid (symmetric positive definite) inputs. (Triggered internally at ../aten/src/ATen/native/BatchLinearAlgebra.cpp:1744.)\n",
" Lff = Kff.cholesky()\n",
"/opt/venv/lib/python3.9/site-packages/pyro/contrib/gp/util.py:109: UserWarning: torch.triangular_solve is deprecated in favor of torch.linalg.solve_triangularand will be removed in a future PyTorch release.\n",
"torch.linalg.solve_triangular has its arguments reversed and does not return a copy of one of the inputs.\n",
"X = torch.triangular_solve(B, A).solution\n",
"should be replaced with\n",
"X = torch.linalg.solve_triangular(A, B). (Triggered internally at ../aten/src/ATen/native/BatchLinearAlgebra.cpp:2183.)\n",
" Lffinv_pack = pack.triangular_solve(Lff, upper=False)[0]\n"
]
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2cAAAG4CAYAAADBvJ+2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAB7z0lEQVR4nO3dd3xb1f3/8dfH24kdZ09nkUDChrD3KFDKLC20tNCW1Ra+paV0/brooHtSKC2jQGnLboAywgoQCCuQkEn2IMPOcBLvPXR+f0g2iiPbsi3pXknv5+PhR6x7r64+V1Z09NY591xzziEiIiIiIiLeyvC6ABEREREREVE4ExERERER8QWFMxERERERER9QOBMREREREfEBhTMREREREREfUDgTERERERHxAYUzSRtmNs3MFptZjZl9I4GPO8HMas0sM1GP2RMzu8LM3vS6DhERv4v2PdzMLjOzlxJUk5nZP82swszeS8Rjhj3282b2pUQ+Zk/MzJnZVK/rEIkFhbM0Y2YnmtnbZlZlZuVm9paZHeV1XdEws41mdkY/dvE9YI5zrtA5d1us6uqsc53Ouc3OuQLnXFu8HlNEJJ2Y2WuhYJIb78eK9j3cOfegc+6ssBrjGRhOBM4Eip1zR8fpMTCzn5nZA+HLnHOfcM79K16PKZLuFM7SiJkNAp4F/goMBcYBPwea+rAvM7OMTsuyYlFnHE0ElntdhIiI9J2ZTQJOAhxwgbfVeGYisNE5V+d1ISISWwpn6WU/AOfcw865Nudcg3PuJefcUtj7GzIzmxT65i8rdPs1M/uVmb0F1AP7hNZ/zczWAmtD233ZzNaFeuaeNrOxYfs8y8xWh3ru/m5mr5vZNaF1U8zsVTPbbWa7zOxBMxscWvcfYALwTGh4yfdCy48N9QRWmtkSMzs10oGb2avAacDtofvvFzqea8K22WOoX+jYrjWztaH9/83MLGz9l81sZWiY5AozmxGpzgjP49jQ81Ieep6+HLbPn5nZY2b279B+l5vZkV0c0x1m9sdOy54ys2+Ffv++ma0Pq++iLvazR31hf+vw5+aq0LFWmNmLZjYx0r5ERBLgi8A84H5gj+F1ZjbezJ4ws52htuT20PJMM/tjqG3ZEGq3wt+X9xjxEN4eRngPvyK0jxoz+9DMLgtb/mbo97mhXS0JtQWfDS3vrn3sts0J2+5q4B7guNC+f965/Qrb39TQ7/eH9jcrVPe7ZjYlbNsDzWx2qK4dZvZDMzsb+CHw2dDjLAlt29E+mFmGmf3YzDaZWVmo7Srq9Lx9ycw2h577H0X6g5rZMWa23cKGjprZRWbW/vnkaDN7J/S8bDOz280sp4t99dS2Tw871tVm9plI+xHxisJZelkDtJnZv8zsE2Y2pA/7+ALwFaAQ2BRa9kngGOAAMzsd+A3wGWBMaJtHAMxsODAT+AEwDFgNHB+2bwvddyywPzAe+BmAc+4LwGbg/NDwkt+b2ThgFvBLgj2B3wEeN7MRnYt2zp0OvAFcH7r/miiP9zzgKOCQ0DF9PHQsl4Rq+yIwiOC3t7sj1Rlhn48AJaHjvBj4deh5a3dBaJvBwNPA7V3U9jDBRtNCNQ0BzgrdF2A9wW+Xiwj2kD5gZmOiPO4OZnYhwQb6U8AIgs/jw73dj4hIjHwReDD083EzGwXBAEZwdMgmYBLB0SHt74dfJvh+fjhwJMH33l4zs4HAbcAnnHOFBNuwxZ23c86dHPr10FBb8Gh37WOYiG1Op33fC1wLvBPa90+jLP9Sgm3BEGAd8KvQMRUCLwMvEGyXpgKvOOdeAH4NPBp6nEMj7POK0M9pwD5AAXu3WScC04CPAT8xs/0jHNO7QB0Q3hZ+Hngo9HsbcCMwHDgutK//i/K4O4T+frND+x1J8Dn5u5kd0Nt9icSLwlkacc5VE3yTdMA/gJ2hb+5G9WI39zvnljvnWp1zLaFlv3HOlTvnGoDLgPuccwudc00Eg9hxFhyGcg6w3Dn3hHOulWADtz2svnXOudnOuSbn3E7gz8Ap3dRyOfCcc+4551zAOTcbWBB6nFj5rXOu0jm3GZgDHBZafg3we+fcfBe0zjm3qcu9hJjZeOAE4P855xqdc4sJfgP6xbDN3gwdUxvwHyBSgwjBkOQIBjAIfth4xzm3FcA591/n3NbQc/MowZ7NvpybcC3Bv/HK0N/t18Bh6j0TkUQzsxMJDul7zDn3PsEvoT4fWn00wXDxXedcXeg9tr3H5DPAX5xzW5xz5QRDUl8FgIPMLN85t805F+1w+e7ax3ZdtTmx8KRz7r3Q+/iDYfs+D9junPtT6DmrCYWlaFwG/Nk5t8E5V0vwmC61PU9z+HlopM4SYAldt2kPA5+DjsB4TmgZzrn3nXPzQp89NgJ30f3ng66cR3A46D9D+1oEPA5c0od9icSFwlmaCX3AvsI5VwwcRLAh+0svdrGlh2Vj+ahHjdCb9W6C32CODd/WOecI9iABYGajzOwRMys1s2rgAYLfknVlInBJaJhDpZlVEgyfve4d6sb2sN/rCX4rCMFevfV92N9YoNw5VxO2bBPB56erx8yzCOfzhZ6/Rwg1ZgQ/oDzYvt7MvmjB2Snbn5uD6P757MpE4Naw/ZQT7OUc1+29RERi70vAS865XaHbD/HR0MbxwKZQ+Ohsj/aHsHaqN0LneH2W4JdW20LDBKdHeffu2sd2XbU5sRDr9gw6HVPo9ywg/EvfaI/pIeBTFpzk5VPAwvYvPS14KsKzoaGP1QS/JOxre3ZMp88NlwGj+7AvkbhQOEtjzrlVBMfsHxRaVAcMCNsk0puV62HZVoJvfkDHEIJhQCmwDSgOW2fhtwm+2TrgYOfcIII9Y+Hj7Ts/9hbgP865wWE/A51zv41QYyTRHG9XtgBTulgX6TlqtxUYGvpWsN0Egs9PXzwMXBzqxTqG4DeAhG7/A7geGOacGwx8wJ7PZ7v2E8q7ei62AF/t9DznO+fe7mPNIiK9Zmb5BHvATgl9SN9OcKjboWZ2KMH3qgmRvswi2P6MD7s9odP6qNsD59yLzrkzCX4RuIrge200umsf+2uP+s2st+3ZPl2s6649g07HRPB5bQV29OLxgw/k3AqC4e4T7DmkEeAOgs/1vqHPBz8kcnsG3f8ttwCvd2rPCpxz1/W2XpF4UThLI6GTYL9tZsWh2+MJ9rrMC22yGDjZgtd0KSI4PKG3HgauNLPDQt9+/Rp4NzQMYRZwsJl9MtR4fo093zQLgVqgKnQ+2Xc77XsHezYgDwDnm9nHLXiyd56Zndp+fFFYTPBbugEWPGn66l4c5z3Ad8zsCAuaGjbMr3OdHZxzW4C3gd+E6j0k9LgPRNq+J6EhGbtC9bzonKsMrRpIsFHdCWBmV/JRCO+8j50EPxxcHnoer2LP4Hkn8AMzOzC0r6LQOXciIon0SYLnHh1AcEjeYQTPT36D4NDw9wiGsN+a2cDQe+wJofs+BnzDzIpD5+d+v9O+FxMcjpdtwUmYIp6TFhrhcWEoWDURbLMCXdTbuS3orn3sryXAgaF95xE6XztKzwJjzOybZpZrZoVmdkxo3Q5gknWanTnMw8CNZjbZzAr46By1SL2X0XgIuAE4Gfhv2PJCoBqoDfVUdhemFtN12/4ssJ+ZfSH0t842s6MinQcn4hWFs/RSQ7B35V0zqyMYyj4Avg0QOmfrUWAp8D7BN7Fecc69DNxEsAdnG8EP+ZeG1u0iOK779wSHchxA8Byx9qn8fw7MAKoIBrknOu3+N8CPQ0MRvhMKOu2TVewk+I3Yd4n+dX0L0Eyw8fkXYUMCozjO/xI8mfohgs/r/whOSrJXnRHu/jmCJ6tvBZ4Efhp63vrqIeAMwr5lDH0D+SfgHYLHdzDwVjf7+DLB5243cCDBANm+ryeB3wGPhIaTfEDwm00RkUT6EvBPF7zu2Pb2H4ITUFxGsCflfIITWmwmOGz+s6H7/gN4kWCIWcje7ctNBNurCoJt0UNElgF8i+D7dznB8566Cgo/A/4Vags+01372F8uOMnVzQQn9lgLvNn9Pfa4bw3Ba6adT3AI4lqCE3zARwFpt5ktjHD3+wieGz0X+BBoBL7eh0No9zDB5/TVsKGrEJzw6/ME29t/EPys0pUu2/bQsZ5F8HnfSvB4fwfE/Xp5ItGy4GkrIokX+iauBLjMOTfH63pERCQ9hCbh+BDI7kcvj4hIzKnnTBIqNARxcGhIR/uY8Xk93E1EREREJOUpnEmiHUdwVqhdBIdQfDI0Bb+IpKjQFzIzzWyVBS9mflyn9WZmt1nw4rxLzWyGV7WKiIhEYmZnW/DC5evMrPN5q7F7HA1rFBGReDKzfwFvOOfuMbMcYEDY5DWY2TkEz1M5h+B5sbc6546JuDMREZEEs+BF7tcQPD+zBJgPfC50jn9MqedMRETiJjTz68nAvQDOuebwYBZyIfDv0AXd5wGDzSyW1ysUERHpj6OBdaELrjcTvM7shfF4IIUzERGJp8kEZ1P9p5ktMrN7QtOQhxvHnhcILkEXORcREf9IWDsV6UKNCTFkaIYbV+zZw4uIJKXly1p2OedG9PX+J52a5yrKu7osU5/qWU5w+ux2dzvn7g67nUXwEhlfd869a2a3ErzG1E0xK8JD2UX5Lm90kddl+ML4/OHUtzWzu7k6dHsEVS11VLfWe1yZSM8GZTX2vJHEzNYVVcnWliWMZ+loXHEWj88a7tXDi4gkpekTtm3qz/0rygMxfe+dPmFbo3PuyG42KQFKnHPvhm7PZO8LAJcC48NuF4eW+V7e6CKO+PvlXpfhiW9Pv5hsy+S3K4OXnPrO9ItZU1PK06XvdGwz0qviRGLkzNGrvC4hJf34oFnJ1pYlrJ1S15WIiMSNc267mW0xs2nOudXAx4DOJ1A/DVxvZo8QnBCkyjm3LdG1Svc+P/E0jhw6jW8tuhOA3U3VZGVkdqz/46qZXpUmEjezt0/fa5kCW1qaD+xrZpMJhrJLCV4YPeYUzkREJN6+DjwYmqlxA3ClmV0L4Jy7E3iO4EyN64B64EqvCpWPHDNsOpdOPJXvLvoHra6N6pZ6ypoqybQM2lyA+z98yesSRTyhwJZ+nHOtZnY98CKQCdznnFsej8dSOBMRkbhyzi0GOg8XuTNsvQO+lsiaZG9j84dx6YRTeXDTq+xorCADIzcjh+G5g9jeWMGzW9/l2a3v9rwjkTSkwJb6nHPPEfwyMa4UzkRERNJQTkYWp486nPW1W1lbU0oGxhmjD+etXcvZ0VjBO7tX8s7ulV6XKZK0Ogc2hTWJhsKZiIhImhiYmcfgnIGUNuwm0zL45rSL+O/muaytKaWkYRcXzv0pLa7N6zJFUpJ61yQaCmciIiIpLAMjgAPg9iOvZ1dTFd9d/A8a2pq5ct4f2dZY3rGtgplIYql3TTpTOBMREUlRl008nTNGz+DKd/8IwF3rZlHVUtexPjyYiYj31LsmCmciIiIpYsKAEXx6/Encue5ZGtqaKWnYxaKKdeRmZNMUaGGeziETSTrqXUsvCmciIiJJbFz+cOpaG6lsqaUoeyBnjT6Cl7cvZFnVRl4vW8rrZUu9LlFEYkhhLbUpnImIiCSZ9muNDc4u4D/HfY9/fTibf304m2VVG/nUmz+noa3Z6xJFJEE0FDK1KJyJiIgkkT8c9mXKm2v4zYpHqGyp5ZfLH2JZ5Ycd6xXMRES9a8lL4UxERMTHZgyZyhFD9+Uf658HYGHFOupaGzvWv7pjsUeViUiyUO9a8lA4ExER8ZlphcWsrSklgGP6oPF8fMyRPLRxDnVtjTy8aY7X5YlIClDvmj95Fs4q2gYws3pGj9tdPGhhAqoRERHxh+OHH8CvD72Kby+8i/cr1jJzyxs8uvl12lzA69JEJIWpd80ffN9zFk2Aa6cgJyIiyWZAZi7fP+BS3t61nBe2LWBB+Rp+v+IxVlVvAaA50OpxhSKSrhTYEs/34aw3FORERCQZ5GfmMmHACFbXlFDf1sTArDxyMrKBYBh7btt7HlcoIhKZhkPGV0qFs96INsgpxImISKz94IBLOaBoAp9961e0uQDfXnSX1yWJiPSJetdiK23DWbQU4kREpL/2LRzHVft8nF8tf5ja1gb+s/FlsixT55GJSEpSYOs7hbMY6SnEKbyJiKSX3IxssjOyqG1twDnHlIKxjB8wgpXVm1lbU+p1eSIiCbVnYJvlWR1+p3CWIJqZUkQkfeRmZPPQ8T/glR2L+PvaZ1hXu5VL3/oVAZzXpYmIiI8pnPlIdwFOwU1ExN+K84dz8ODJPL9tPk2BFh7a9GrHjIuAgpmIiPRI4SxJKLiJiPjb+eOO44JxxzK3bBl1bY08vuVNr0sSEZEko3CWAhTcRES8MXHASKYWjGVd7VYe2vQqj2yeQ11bo9dliYhIklI4S3FdBTeFNhGR/mtsa6YwewAAVS11HlcjIiLJTuEsTSm0iYj0346mShZVrPO6DBERSREKZ7IHhTYREREREW8onElUIoU2BTYRERERkdhROJM+U2ATEREREYkdhTOJKQU2EREREZG+UTiTuOsc2BTWRERERET2pnAmCaewJiIiIiKytx7DmZmNB/4NjAIccLdz7tZO25wKPAV8GFr0hHPu5phWKilLYU1E4k1tmYiIJINoes5agW875xaaWSHwvpnNds6t6LTdG86582JfoqQbhTURiQO1ZSIi4ns9hjPn3DZgW+j3GjNbCYwDOjdoInGhsCYi/aW2TEREkkGvzjkzs0nA4cC7EVYfZ2ZLgK3Ad5xzyyPc/yvAVwCKxuT3ulgR2DOsKaiJSG/Fsi3LHVkYx0pFRCTdRB3OzKwAeBz4pnOuutPqhcBE51ytmZ0D/A/Yt/M+nHN3A3cDjDtwsOtr0SLt1KsmIr0R67ascNpotWUiIhIzGdFsZGbZBBuzB51zT3Re75yrds7Vhn5/Dsg2s+ExrVQkCjOrZ3T8iIiEU1smIiJ+F81sjQbcC6x0zv25i21GAzucc87MjiYY+nbHtFKRXtLwRxFpp7ZMRESSQTTDGk8AvgAsM7PFoWU/BCYAOOfuBC4GrjOzVqABuNQ5p6Ee4hsKaiJpT22ZiIj4XjSzNb4JWA/b3A7cHquiROJJQU0k/agtExGRZBDVOWciqUrnp4mIiIiIX/RqKn2RVKXeNBERERHxmnrORDpRb5qIiIiIeEHhTKQLmpZfJHbMLNPMFpnZsxHWXWFmO81scejnGi9qFBER6S0z+4WZLQ21Xy+Z2dj+7E/hTCQKCmki/XYDsLKb9Y865w4L/dyTqKJERET66Q/OuUOcc4cBzwI/6c/OFM5EekEhTaT3zKwYOBdQ6BIRkZTinKsOuzkQ6NclWBTORPpAIU2kV/4CfA8IdLPNp0PDQmaa2fjElCUiItJ/ZvYrM9sCXEY/e840W6NIP7QHNM3wKMmiom1AjL9YmDXczBaELbjbOXd3+w0zOw8oc869b2andrGTZ4CHnXNNZvZV4F/A6TEsUkREUogHbdnLwOgId/yRc+4p59yPgB+Z2Q+A64Gf9rUShTORGFBIkzS2yzl3ZDfrTwAuMLNzgDxgkJk94Jy7vH0D59zusO3vAX4fn1IlFjaWjPC6hL1MKt7pdQkikty6bcucc2dEuZ8HgedQOBPxh5nVMxTQRMI4534A/AAg1HP2nfBgFlo+xjm3LXTzArqfOESi5McQFS+xOlaFPBHpLTPb1zm3NnTzQmBVf/ancCYSY+pFE+mZmd0MLHDOPQ18w8wuAFqBcuAKL2vzm3QKWV7rzXOtICciIb81s2kEz6veBFzbn50pnInEiUKayJ6cc68Br4V+/0nY8o7etXSgsJUaevo7KryJpAfn3KdjuT+FM5E401BHkfSg0CXhuns9KLiJSFcUzkQSQL1oIqmpuTlLoUx6ravXjEKbiCiciSSQetFERKQrnUObwppI+lE4E0kw9aKJiEg0IvWwKbCJpDaFMxGPqBdNRER6S71rIqlN4UzEQwpoIiLSHzp/TSS1KJyJeEwBTUREYk09bCLJSeFMxAcU0EREJJ4U1kSSQ4bXBYhIUPtEISIiIvG2sWREx4+I+Id6zkR8RD1oIiKSaOEBzc89ahkYeZk5DMjKJT8zl/zMHBrbmtlcH6z5uOEHkJeRTaZlkGmZZFoGWxt2s7hyPQCfKj4Rh6PNBTp+NtftYEX1ZgBOHHEQbS5AS6CVhrYm6lubKG+upqql3rNjlvSjcCYiIiIiQGKCWqZlMCh7AEXZA8m0DNbXbgPgwnHHM3HgSAbnDKQoO/izrnYbv13xCAAPHv8DxuQP3WNfb5Qt46Zl/wLge/t/hiE5BXusf3Hbgo5w9tWp55Kbmb3H+ie3vMWK6s1kWga/POSKvWp9aOMc7l4/i4KsfJ466Wc0tDUHg1tbE7WtDfyv5G1mb1/IgMxczh93LJXNdVS11FHZUktlcx3lzdU0B1pj8rxJelA4E/EZ9Z6JiIgf9DWoDc4uYFTeYEbkDSY/M4fZ24Nt2o3TPsXxww9gaO4gMi14Zs36mq1c/d6fAThj9OFMGjgqGG6a6yhrrGJbw+6O/T68aQ65Gdk0tDV1hKSyxsqO9Te8/3eAsJ6xNhramjvWf+qNnwd71TIyOnrXGtuaAAg4x9Xv/olMyyA3I5u8zBzyM3MpbdjVsc+HNs0hPyuXAZnBn4KsfNpcAIAReYO5bt/z93ou/rRqJs+UzmNc/nBumPZJyhor2dFYyc6mSsoaK1lTU0pta0PUz62kPoUzER9SQBMRET/pHNSG5hQyYcBIRucP4YVtCwD4+n4Xcv7YY8kJ652qaanvCGc7m6p4v3wtO5uqKG+uobK5lp1NVR3bfuP9v+NwXdbwdOk73da4ub6s2/V1bY1drnO4jh68SBramrh3wwtdrt9Ut4NzXvsRg3MKKMoeyODsgRTlFLC8aiMAA7JyKcoeyNTCcQzNKey43w+W3Mc7u1Zw6OB9+PKUcyip38mWTj+trq3b45LUonAm4lMKaCIi4qUMMyYXDmFjTQVtznHRPgdy1QFHMnnQEAqyczu2e6PsA+raGlletYmWQBs7GivY2VhJWVMlZY0fha8HNr7S7eN1F8ySQX1bE/UNTWwN6+1rt7amlK/OvxWAnIwshucWMTJ3MBtCgdDMaA60csTQ/Th77FEd97vm3T+zrnYrRw3dj+OHH8iW+p1sqN3Gutqt6nFLUQpnIj6mgCYiIokyekAhpxfvw0FDR3PA0JFMGzKC/KxsznzqXtZW7qI1EKC8sZ4FZSV8WF3BhqpyPqwuJ2dIcGjgqzsW8+qOxd4eRBJoDrSytWH3HiFuccV6FlcEz43Lz8xhXP5wxg8YwZbQZCfjB4zkzDEzKMjK77jPjsYKrpr3J+raGhmXP4yWQBtlTZUJPRaJPYUzERERkTQztWgYR40s5vARY3lozWIW79rG/kNG8OvjzqayqYEV5WU8tGYxK8vL2NlQC8AzG1fyzMaVe++sbjjg75kek0lDWzPrareyrnZrx7InSt7kiZI3GZpTyJSCMUwpGMvYAcM6hmpetc/ZfGz04VQ217KmppTV1VtYWrmB+eVrvDoM6SOFMxGfU++ZiIjEwvC8AfzmuLM5YuQ4huYNAGBXQx2vl37I4l3bmLdjCyfOvIOSuuo+7T9ZpuRPZuXNNZSX1+wVuh7eNIellR+yX+E49htUzOcmnsaxw/dn/nvB7S4qPoGqljqWVm5gV1Pf/r6SGApnIiIi4ku5m3O8LmEPTROae97IB/IyszhqVDFHjRzPUSPHsWBnKX9a9AZVzY0UFxQxe8s6FpSV8N6OLWyqqey4X0NrCyWtLTGpoT2oKaQlRueetryMHIblDuq4ffmkj3Xc3tZQztLKDczZsYR5uyP0hIqnFM5EREQk7vwWtPqir8eQyFB3+8kXcOaEfcnNzKItEGB5+Q52NwQvotwSCPCJZ/6ZsFpAvWleaQw0d1wGAOAzb/2SqQVjOXjwZA4ePJmjhk1je2MF83avJC8jh+v2PY/55WtYWL6W+tDlBcQbCmciSUBDG0XEr1IhdMVbT89RX8LboJxcThwziZPHTmZ8YRGXvfQoACV11fx71ULe2LqR98tKqWv1T2+fetO80+YCrK4pYXVNCTO3vAFAtmUCMGHgCM4YPYMLi4+nNdDG8qpNvLd7FS9uX6AhkB5QOBMREZFuKYDFV1fPb6TQdtb4ffnKQUdz+PCxZGZkUN3cyBtbN5KbkUlToI3fvv9anKvtP4U0f2gJXT9tTU0pF8z9CQcWTeLoYdM4eug0vjz1HN4rX82upmqmFRYzNn8Y83av3OOi3hIfCmciIiKiAOZDuZtzmD5qBGfvvy//XfQBpVXV5E3OIisjg9uXvcPrpR+yZNdW2lxyXh9MIc0/2lyApZUbWFq5gXvWP8+QnAIqm+sAOHP0DC6ecDJNbS3ML1/N62VLeXvnim4v6i1951k4q27NY/b26R23zxy9yqtSRJKChjaKSKwoiPnbQWNG8fH99+Xj++/LxKGDaQsEWLNzF6VV1bz4+npefH19x7ZtE5IzmIXbWDJCAc1nKpprO37/+9pneH3nMk4ZcTCnjDyEE0ccxLaGcj739q8ByLJMWkO9cNJ/vuk5Cw9q8aIAKCIi6UQhLHkMyc+joqGRwfl5PHrVpTjneOfDzdz91nxeWbOeivqGiPcL/xsny2ySkagXzb8COJZVfsiyyg/529pn2H/QeIaGZn40jAeO+39srt/J62VLeaNsGdWt9R5XnNx8E84SoT8BUMFORET8TmEsuYwZVMiFh+zPJw85gB3VNXzpgcepbGjk2kf+x5LS7VQ39m7WvM5//2QMawpp/uZwrKje3HE7NyObV3Ys4pSRh/Dd/S/hm9Mu4u1dK3ho46usrinxsNLklVbhrD96G+wU5iQeNLRRRNopiCWvk6dM4spjj+DYyePJMOPdjVt4YumKjvVvrN8Uk8dpf40ka0hTQPO/xkAz/1j/PP9Y/zxTC8Zy1pgjOGP0DIqyBwIwIreIwuwBbKjd5nGlyUPhLE6iCXMKcCIi0hOFsNQwsnAglfWNNLe1MWnYEMYPKeL2ufN4aukKSirjO115sg59VC9acllXu5V1a7dy17pZuNAkNRcVn8DnJ53OssoPeXTTa7y9awUBkv88yXhSOPNQdwFOwU1EJD0ofMVX4abEfBCsmWgRl08dMYyrjzuC8w6azs+fe5WZiz/g4QVL+M97izz5iJqMQU29aMmlzQU6fn9082uUN9fw6fEn8ctDr2RL/U4e2TSH1z2sz+8UznxKwU1EJLkpdMVWokJWX3Wub9KYoXzlguM466hpNDS18MRrS1n21iYKdzngo5ntugp1iZBMQU29aMmpqqWemVve4MmStzhpxMFcOvFUDhm8j9dl+ZrCWRKKFNwU2EREEs+aTSEsRvwevnrr51edzaQxQ7n32Xk8NHshVXWRrwnV03EnKrwlS1BTL1pyanMBXitbwmtlS8jNyPa6HF9TOEsRCmwiIpIMUi2EtRs7fBBfOvso/vbkW1TXNXLzP19kV3UdVbX9u1BvV89XPEOb32d9VC9acmsKtHhdgq8pnKWwzoFNYU1ERBIpVYNYuFFDC7n63GO44IQDaXOOuUs28NayD1m/dXdcH7fzc5uIsObHkKaAJqlG4SyNKKyJiEi8JFMQK1rfu+uHRZKRYXzt+jM559xDAXjm6UU8/NA77N5dS1E396uaktvvx44kEWHNjyFNvWiSahTO0pjCmoiI9JWfw1gswldXzMA5CAQcQ4cO5MUXlvLQg+9QVhbddPg91Rar8Bb+94l1UPNrSFNAk1SgcCYdwsOagpqIiITzYxiLZwiL5MST9uPqa07lph/NpKSknJt//iQuxk9L52OKRVhr/9ulekhTQJNUoHAmEalXTURE/BbIEh3G2k2ePIL/u/4MZsyYxIYNZeQPCIaSWAezSGIZ1uLVm+ankKaAJslO4Uyiol41EZH04JdA5lUQA8hZVdLx+1dv/jTnX3UqdVX13P79R3j+gbcItAXo7QUUmqcXx6S28OclFkEt1iFNAU2kfxTOpNfag5pCmohIakjHQBYewLrT0tzGs/+cywN/mkVtZX3MH68/oS0WQS3WIc0vvWgKaJKsFM6kz9SbJiKSnNItjEUbxAAGDsrnm3+6jKfve51l76zlvl/+L36FEbm2vgS2/ga1eIQ0BTSR3lM4k5hQb1piXDxoodcliEgS8ksYA38Gsnbj9hnJz/51LaMnDmfBnBUse2dtHCrrWXvtfe1Va3+O+xPSoP9BLfxi1l4FNU21L8lG4UxiSiFNRMKZWR4wF8gl2ObMdM79tNM2ucC/gSOA3cBnnXMbE1xqyvBTEGvnx+GKnR120jR+ePc1BNra+P7Ft7L8vfUxrqz3claVeDrsMR5BzcuQpoAm8WJmXwe+BrQBs5xz3+vrvhTOJC4U0kQkpAk43TlXa2bZwJtm9rxzbl7YNlcDFc65qWZ2KfA74LNeFJts/BjEwJvJPPoaygD2O3QCv3zoa2xeu52fffFOykrKY1hZ//S3F61df2d9jNWwRy9DmgKaxIOZnQZcCBzqnGsys5H92Z/CmcSVQppIenPOOaA2dDM79NM5UVwI/Cz0+0zgdjOz0H0ljF/DGCRfIAu3dukW7vvlUzz/wJs01Hk3S2R3YhXS2vW1Vy2WIc2rgAYa5igxdR3wW+dcE4Bzrqw/O8uISUkiPZi9ffpe104TkfRgZplmthgoA2Y7597ttMk4YAuAc64VqAKGJbRInyrc5Pb48aOi9U0JPY8s/Kc/CgYP4Pt3XsWo8cNwzvHEXa/4NpiFi8Wxd9b+N+zN3zEWr8vczTl7nJeWSO0hTSQG9gNOMrN3zex1MzuqPztTz5kklHrSRLxV3ZoX4y9KZg03swVhC+52zt0dvoVzrg04zMwGA0+a2UHOuQ9iWETK8GsA68zPk3pEo3jqKH72r2sZOW4Ic596nx1bdsflceKp83PjZY9af3vTvOxFUw9ackp0W2ZmLwOjI9zxRwTz1FDgWOAo4DEz26evoz8UzsQTs7dPV0DrJc3UKD61yzl3ZDQbOucqzWwOcDYQHs5KgfFAiZllAUUEJwZJeckSxiB+gSxeAawrM06Zzg/uuprWlja+f8ltrJi/IaGPHy/hz6NXQa0/Ic2rc9EU0CSk27bMOXdGV+vM7DrgiVAYe8/MAsBwoE8vLIUz8Yx60URSn5mNAFpCwSwfOJPghB/hnga+BLwDXAy8mqrnmyVTGIP4BLJEh7FwR51+ID/997VsWrWVn19xl68m/oil7p7jREzPX7jJ9asXDRIb0nQemvTT/4DTgDlmth+QA+zq684UzsRz6kUTSWljgH+ZWSbB85wfc849a2Y3Awucc08D9wL/MbN1QDlwqXflxlayhTFInR6ySJa+s5Yn7nyFh/78PI31/j+/LB76e9HrovVNcQ9o4M1QR/WiSR/dB9xnZh8AzcCX+vMFo8KZ+IJ60URSk3NuKXB4hOU/Cfu9EbgkkXXFSzKGsXapGsqysjO59IazefyOl2moa+K+X/7P03r8qLdDIqPtRUvGc9EU0KS3nHPNwOWx2l+PszWa2Xgzm2NmK8xsuZndEGEbM7PbzGydmS01sxmxKlDSi2Z0FJF4iGdb5vfZFLvTl1n6ohWPWQX74v/dcSWXffscjvrYgV6XkhR6MyNmtK+bZJvRUTM5ipei6TlrBb7tnFtoZoXA+2Y22zm3ImybTwD7hn6OAe4I/SvSa+pFE5E4iEtblpn4Ceb6Ld4zLfohkLX72CVHc+K5h3PvL55k7tOaVKkverq+Wqr2ouk8NPFKjz1nzrltzrmFod9rgJUEr0kT7kLg3y5oHjDYzMbEvFqRNKWZGkX6R21Z/K9H5peesnZDRxXx1Zsv4YN31/H4Ha94XU7Si2UvWl97mtt70RLZk7axZIR60iShenXOmZlNInjuQJcXEA0pCS3b1p/iJL1pohARiYd0assScT0yPwWycFf9+JNk52bx528+QIpO/plwsepFaxeLqfchMTM7qidNEiXqcGZmBcDjwDedc9V9eTAz+wrwFYDckYV92YWkGQU0EYmlWLdlOQOHxLC62EnnUNbunp8/wev/W8C2jfowHWs5q0q6nTikr9dHA/8HNYU0ibcehzUCmFk2wcbsQefcExE2ab+AaLvi0LI9OOfuds4d6Zw7MnvwgL7UK2lIk4SISCzEoy3Lyh8Yn2L7KN5DF8F/wxc7G1CYR0aGUbmrhvmvLPe6nJQV7Wugt5PO9HdynUQNe9RwR4mXaGZrNILXoFnpnPtzF5s9DXwxNNPVsUCVcy5ph4GIiEhqSeW2LJ4zLobzeyhr972/XcEvHvqa12Wkhd6+JvoS0voa1BIZ0kRiKZphjScAXwCWmdni0LIfAhMAnHN3As8B5wDrgHrgyphXKmlNwxtFpJ9Sri1LxNBF8P/wxXBnXHIMx5x5MHf9ZKbXpaSVnoY5dpZq56bp2mgSSz2GM+fcm0C3/xtCV8HW11QSV+ka0DRTo0j/pVJblqhQBskVzIaNLuKrv7iYD95dx1P3vOZ1OWmntwEN+n5umh+n49e5aBIrvZqtUcRr7QHtgPJSTtu+mqKWBqqy85kzehorhnaeFVtEJHUolHXv67//HFk5yTU746mHr+GKc+YxYkgtOysKuP+5Y3lt0X5el9VnnV838epNi0VPmnrRxK8UziTpHFBeyrkly8hxbQAMbmng3JJlAApoIpJyFMp6VjSsgOIpo7j/108lzeyMpx6+hhs+8xp5Oa0AjBpayw2feQ0gqQNauEivp54CWyJDmnrRxI8UziTpHFe6viOYtctxbZy2fbXCmYikhEQGMkjeUNauanctXzvj1zQ3tnpdStSuOGdeRzBrl5fTyhXnzEuZcBZJ+Gstmun44xnS1IsmfhTVVPoifjIyUBNxeVFLQ4IrERGJrUTMuhguWWZg7M64fUZSOGQgTQ0tSTOcEWDEkNpeLU9F0bz2EjENfzxnddRsjtJb6jmTpFOWUcjoCAGtKjvfg2pERPov0T1lkPy9Ze2u/92lDCjM44azf+91Kb2ys6KAUUP3DmI7Kwo8qMY77a/DeAx3hOh70zoHtFj2poUHNPWkSU/UcyZJ5x8FJ9HY6XuFZstkzuhpHlUkItJ3CmZ9l5mVwfQjJrPivfVel9Jr9z93LI3Ne7Zljc1Z3P/csR5V5K3eXtQ6Wn29Xlr7ddJi3aumi1dLT9RzJknnlfz9Afhy7RuMCtRotkYRSUpehDJInWAGMOWg8eTl57D83Q1el9Jr7eeVpdJsjf3Vm+n4e3utNKDLgNZT71o8rpWm3jTpisKZJKVX8vfnlfz90/K6ZyKS/BTMYuPAY6YAsHx+8vWcQTCgpXMYi6SvF7SG3gW1cJ1DW3dhTUFN4k3hTEREJEG8CmWQesEM4MCjp7Bt404qyqq9LkViqC8XtIbI/7/6Etiinf0xHtPxaxp+UTgTERGJMy9DGaRmMAP456+fYtioIq/LkDjoa0DrrKv/e9FOLOJFQAP1pqUzhTNJarO3T9fQRhHxNQWz+CldX0bp+jKvy5A4ifaaaH3R3f/L8OAWTS9aIq6X1k5BLfUpnImIiMSB16EMUjuYHXj0FMZMHM6cJ+fT1hrwuhyJs0iv5VgHtnZF65v26lmLNqTFK6C1U1BLfQpnIj43s3oGFw9a6HUZIhIlP4QySO1gBnDWpcdxzMcP5uX/vut1KeKR7l7j/Q1uXc0G2dNQx0QEtHYKaqlJ4UxEujSy5RwGj7oJBoyH+i1U7vgFZdnPeV2WiG8pmCXOAUfvw4r3km8KfUmM8P8D1904jnOvvZyMwmICNSXMuvMB7riltMv7hge7vvSixXuYYyQKaqlDF6EWkYhGtpzD4Am3YgMnYpaBDZzI4Am3MrLlHK9LE/Gd3l4YN57SIZgVDSugeMooViTpFPqSONfdOI7zv3UDmYMmYJZB5qAJnP+tG7juxq6vjdr5/1BX/797urB1rC9gHa32C13rgtfJSeFMkt7s7dO9LiElDR51E5Y1cI9lljUw2JMmIgBkNjrfhDJIj2AGYBbssWhuavW4EvG7c6+9PGJbdu61l3d7v0j/l5IpoIVTWEsuGtYokgQ8Oe9swPiul5cnthQR6Vm6BDOAyl011FTUMWbScK9LEZ/LKIx87llXy/vCT+ehRaOrgKbhkP6gcCYikdVvgYETIy8XEfHY1Sf8nJqKOq/LEJ8L1JSQOWhCxOWJ5LeAFolCmz8onIkkiUT3nlXu+EXwnLOw4SCutY7KHb+A7ISVISJRSKdes3YKZhKNWXc+wPnfumGvtmzWnQ90e794TNPvxUQhsRDtUEiFuNhQOBORiMqyn4PNaLZGEfGlaYdP4rwrT+aum/5LbVWD1+WITwVnZby1V7M19kVPQxvDJUMvWl/ofLbYUDiTlDB7+3TOHL3K6zLiLtG9Z2XZz1FW/txH55ipx0zEd9Kx1wxg0NCBnHHJMTz/nzdZMV9T6kvX7rillDtu+V3U23fVa9Z5Sv3+SNWAJv2n2RpFkszM6hlelyAi4rkta7cDUDx1lMeViPSNH2ZyFP9ROBNJQgpoIgLp22sGUFZSTlNDMxP2He11KSJAz9PqR6KAJp0pnImIiEjSCQQcJevLGK9wJjEUj4lAeqKAJuEUzkSSlHrPRCTdbVhegmVENwmDSE+8CGbtFNCkncKZpIzZ26d7XULCKaCJpK90HtLY7s/f/A8/uezvXpchKSBWwawvQxvbKaAJKJyJJD0FNBFJdwVF+V6XIEnMyx6zznI35yikpTmFM5EUoIAmIunqC987j9tf/iF5A2I3zbmkh+bpxVEHs1hOox8NBbT0pXAmKSUdhza2m1k9QyFNRNLO+3NWMKp4KF/8f+d5XYokid6EMuhdMIv2QtTRUEBLTwpnknLSOaCBQloq099VZG8r5m/gmX++zoXXnMr0IyZ7XY74WF9CmVfBrJ0CWvrJ8roAEYmP9g/yFw9a6HEl0lcKYyLR+eevnuKYsw7mxj9fxvVn/paW5lavS5IYS/R5YYkextid3M05NE1o9roMSRCFM0lJs7dP58zRq7wuwxcU0pKHwphI3zTUNXHbdx/m//39SiZOH8O6pVu8Lkk68dOkGz3pSzCLR69ZOAW09KFwJilLAW1P4R/8FdT8IR3CmJndB5wHlDnnDoqw/lTgKeDD0KInnHM3J6xASRnvz1nBFUffRH1No9elpLVkCmGR+KnHrLP2IY4Kaf5iZocCdwIFwEbgMudcdV/3p3AmkobUm+aNdAhjEdwP3A78u5tt3nDOaTYH6bf6mkbMjJMvnMEbzywi0BbwuqSUluxBDGITxuLda9aZQprv3AN8xzn3upldBXwXuKmvO1M4k5Sm3rPuqTctvtI0jO3BOTfXzCZ5XUcqap5erAtRR3DoCfvx/TuuYuS4//Hfv832upyUkQpBrJ2fe8d6QyHNN/YD5oZ+nw28iMKZiPRX5yChsNY7yRLEmpuz2FgyIpa7HG5mC8Ju3+2cu7uX+zjOzJYAWwl++7g8duVJuln85mrenLWIy79zLm+/sITS9WVel5S0UiWQxTOMJbrXLJJ0DGk+a8uWAxcC/wMuAcb3pxCFM0l56j3rG/WqdS1ZgliC7HLOHdmP+y8EJjrnas3sHIKN274xqUzS1t9/+BiHvj6Nb/7pMr530V9wznldUlJJtlCWKj1h/ZWOIS2Gum3LzOxlYHSEVT8CrgJuM7ObgKeBfv0BFM4kLSig9U+kMJIOgU0hLP7CT5p2zj1nZn83s+HOuV1e1iXJraKsmrt/+jjfvvULXHjNqfzvH3O8Lsn3/BzI/Bq+/NBrFolCWuw5587oYZOzAMxsP+Dc/jyWwpmI9ElXwSXZQpsCmLfMbDSwwznnzOxoIAPY7XFZSUPnnXXt5cfmccjx+7Lq/Q973jjN+TWY+TWUJYvwC1grqMWPmY10zpWZWQbwY4IzN/aZwpmkDfWeJUa0YSeeIU6Byz/M7GHgVILj+UuAnwLZAM65O4GLgevMrBVoAC51GoPWKwpoXfvzN//T8Xvx1FGUrNvhYTX+49dQBgpmsabetLj6nJl9LfT7E8A/+7MzhTNJK7O3TwdQSPMBBaj04Jz7XA/rbyc41b5I3Jxy4RF8929X8Iev3c/rT73vdTm+4NdgplAWXwppseecuxW4NVb7y4jVjkSSSXtIExFJBX79oO0X815axor5G/ju7V/i9IuP9rocTzVPL/bt60XBLHFyN+d0/Ii/KJxJ2lJAE5FU4tcP3H7Q1NDMTy77O8veWcd3//olzr/yFK9LSji/hzIFM+8oqPmLwpmkNQU0EUklfv3w7QeN9U385At/5+0XlvB/v/4Mk/Yf63VJCePn14VCmb8oqHlP55xJ2tNEISKSSjRBSNdamlr51TX3cNhJ09i4cqvX5cSdn0MZKJj5XeeApvPUEkM9ZyKoB01EJF0E2gIsfG0lAAcfty9f+81nycjw5/Wq+srPQxhBwxiTVXivmnrW4kc9ZyIh6kETkVSh3rPoHHj0Ppx3xckUDSvgD9f/i5bmVq9L6jM/h7FwqRLK/HoB6kSKFNDUu9Z/CmciYRTQRCRVKKD17JFbX6SpsYWv/OzT5Bfk8cur/0FTQ3J9uFQoEz/pqkdNoS16GtYo0sns7dM1zFFEUkKyfHD30pN3vcotNz7A4SdP51ePXE/egOQIEX4fuhhOwUw0JDJ6nvWcNTdnsbFkRMftScU7vSpFJCL1oolIKmj/AK9etK699Mg71Nc0csRp+/u65yxZwlg4BTOR3vHNsMbwoAYKa+IP7T1oCmkikuw0zLF7b85axJuzFgEwYb/RfObrH+feXzxJRVm1x5UlZygDBTORvvBNOOtMYU38RCFNRFKBetGis++hEzj5/MM57uMH858/zuKZ+16nrTWQ8DqSNZSlg8JNTpOCSFwkzTlnG0tG7PEj4gWdjyYiqSCZzlfywiv/fY+vnvorlr+3ga/+/GJun/0DDj5uakJrSPa/Tzr0mhVucl6XICkoacJZZwpr4iWFNBFJBckeAOJp28ad/OTyv/PzK+4if2AuR552YEIeV8E5uSigSaz5dlhjb2kYpHhBwx1FJNlpqGP35r24lEVzV3bcPuzEaUw+cBxP3/taTIc6plIgS4des3Aa4iixlDLhrDPNBCmJpJAmIskuPBwoqO2pqaGl4/fjPnEoF1x1Cmddehx3/vi/LHlrTZ/3m0qBrF26BbN2CmgSK0k7rLE3NPxREkXDHUUkFWhoXdfu+NFjHUMdfzvzBn71yPVMOXh81Pdvf271/KYeDXGUWEjZnrOuqEdNEiE8oKk3TUSSlXrTIpv34lIWvr6Sc790Ep/9xseZevB41i/b0uX26RLE0rXXLJx60KS/0i6chVNQk0TQkEcRSQUKantqbmzhybte5cUH36YxdOHqc790EtMOn8Q/n1zIjh1VHlcoXlFAk/5I63AWTkFN4k29aSKSKjr3BKVzWKuvbex4PvKnjuaUi47k1E8dxbPPLOLBB96ioqLe4woTQ71me1JAk75SOIugPagppEm8KKiJSCpJp1617oYoPvTgO7z04gd84YsncMGFMzj7E4fw11tf4sUXlyWwwsRTMJNoGTA4P8/rMnxN4awb6k2TRFBQE5FU0lV4ScbQ1pdzxXbtquGWP7/AY4++yxVXnkxFZbDnbOSoQRxy8HjeeGM1TU2tsS5VfEi9Zx8ZNnAAu+vqccDnjzqMd70uyMd6DGdmdh9wHlDmnDsowvpTgaeAD0OLnnDO3RzDGn1BQU0SofNMjwprIrGhtsx7fg5t8Ziwo7S0gl/98qmO26edtj9f+erpfKOuidfmrOTFF5ey/IPSmD9uoqnXrHvtMzimY0gblJfLeQdO46JDD2TysCGceMvdNLa28uSS5V6X5mvR9JzdD9wO/Lubbd5wzp0Xk4qSgIKaJIrCmkjM3I/aMl/qTTDqbZDz0yyJjz36LitWbOXssw/h9I8dwLnnHcaGDWV89cv3EQhoCvZUl04hbd8Rw/i/k47hY9OmkJuVxeodO/nr3HfIsOCxb62q8bhCf+sxnDnn5prZpATUkpR0fpokUqRrqCmwifRMbVlq8FPY6i3nYNnSLSxbuoW/3vYSp5wynZGjijqC2VevPZ0PP9zJ22+tpba20eNqo6Nes95L1aGOB48ZRV1zMxt2V5CTmclxkyfw6MJlPLlkBSu2l3ldXlKJ1Tlnx5nZEmAr8B3nXMT+SjP7CvAVgMxhg2P00P6g3jTxinrXRGKm121Zbu7gxFUnKaOxsWWPSUIGDMjhpJOn8ZnPHkNraxuLFm3ijbmrefON1VRVNXhYqcRDKvSiGXB48VjO2n8qZ06fSvHgIh5duIyfzHqZ5dvLOOmWu2kJBLwuMynFIpwtBCY652rN7Bzgf8C+kTZ0zt0N3A2QO7k4ZfvwFdTES5F610ChTaQHfWrLBhWmblsmiVNf38zln7+DadPGcNLJ0zj5lOl869ufAGDWs4sZMCCH/Pwcdu+u9bjSj6jXrP+SuRdt5jWf56Axo2hubeWtDZu5/fV5vLpmfcd6BbO+63c4c85Vh/3+nJn93cyGO+d29XffqUDDHsUvFNpEuqa2TPxg9eptrF69jXv+8Rr7TBlJ2Y7gy/L0jx3Ijd86m+UflPDG3NXMm7eOLVvKPa5WYsHvvWj52VkcNbGYM6ZN5cAxI7n4nodwwCPvL6WhpYXX1n5IbVOz12WmlH6HMzMbDexwzjkzOxrIAHb3u7IUo9408auuQls7hTdv7f33meVJHalObZn4zYb1H52n8/6CD7nv3tc56eRpXPt/H+Pa//sYW7dWcM1V99DU1IpZ8Jy2RFGvWez5LaQdO2k8XznhKI6aMI6crCzqmpp5bd2HFOTmUtPUxH8XfeB1iSkrmqn0HwZOBYabWQnwUyAbwDl3J3AxcJ2ZtQINwKXOJfItIvmoN02SSU/hrZ1CXM+ifS4l9tSWSTLbtq2SBx94mwcfeJvRo4s48qh9GD9+aMf10n7+i08zaFA+C+Z/yPz3NrBmzbaEhjWJnfaQBokLaoPz8zhu8gROmjKRf7+3mFU7dpKfnc2IgoE8MH8Jb27YyPxNpTS3tSWknnQXzWyNn+th/e0EpyeWXlJvmqSS/gYPP4c7harkp7ZMUsX27VU8+8yiPZatWrmNE0/ajyuvOpkrrzqZqqp6Hp85nwcfeDvmj69es8SJZ2/aoLxcrj7uSI6bPJ6Dx44mw4zKhkZeX7eRVTt2MmftBuas3RDzx5WexWq2Rukn9aZJulMAEhHpm4cefJuHHnyboqJ8jjhyMkcdvQ+NjS0A5OVl88BD17F61TaWLdvCsmVbWLN6Oy0t6gVJFv0NaaMKCzhiwliOHD+ODbsreGD+YhpbWvnC0YexpmwXf5s7jzfWb2TZ1h0E1OXqOYUzn1FvmoiIiPRFVVUDr76ygldfWdGxLD8/m7ffWstBBxdz7HFTAWhubuX3v5vFnFdXMHBgLqNHF7F58+4eA5t6zbwVzZDHgTk51DUHJ+i4+dwzOGnKRMYWDQKgrqmZxxYFL+HQ3NbGMX+8kxYNVfQdhTMfU1ATEfG3tjyjakouReubvC5FJKKKinr+/KfnASgqyuegg4o58KBiPtwQnHBkxhGT+NnPP0VbW4AtW3bz4YadbNhQxvPPLaWioq5jPwpm/lK4ybHP2GEcvM8YJuw/nP1GDmfayOFUNTZy9t//BYBzjoVbtnL/vIUs2FzKqh07aQvrGVMw8yeFsyShoCYi4l/tH1wV0sTPqqoaeOuttbz11tqOZR8sK+EXN/+PyZNHMHmfEUzffyynnX4Ac+ashAr4xDmH8PGzD2FteQVbyirZvKOCkrJKPtxWriFwCTCsaCATRg5m/MjBFI8czNjhRfz4H88BcPlZR3DBiQfR0NTC+tJdvLJmPSu3f/QZ8afPveJV2dIPCmdJSEFNRMSfwnsXFNQkGVRU1PHanJW8Nmdlx7IBA3JoaAgOjWtqaqU11zhtxlSGFA7o2OaE626jqaWVT59yCNMmjGRLWSVbyirZtquKHRW1VNY2JPxYklFOViYjBhcwamhhRwD79wvzqalv4urzjuG6T57QsW1raxulu6ooKsijqraRe2e9y33PvUfpzso9ZucsxD9T8kvvKZwlOU0kIiLiT+pNk2RVXx8MZlVTcnly43qe/P16AArycxk/cjCjhhbS1BKcxn/ciCJOPXwqQwd9FNx2VdVx9rfvAuC6Tx7P2OFFlFXUUlZRw46KWkp3VrK2JPWv756bncWY4YMYUTSQkUMLGTWkgJFDCnl49kI27ajg40dP51dfOWeP+7S2tvHygjWs3lzG28s2UlPXxOayYK/l9t3VtAU+SmGlO6u6fGwvpuSX2FA4SxHqTRMR8Sf1pkkyinSOWW1DEys37WDlph0dy26b+Qa3zXyjI7iNHlZITlZmx/ohhQM4eJ8xjBxSQE528GPn8g+386VfPQTAvd+/lNFDC6msbaCytoGqukaWb9jGg7MXAvCxI/bFzGhoaqG+sZmG5hbKq+spq6gFIMMsbsMrMzOMkUMKGZCbTX5eDvk5WQzIy2H91t2UlFUyYnABnztjBoML8xg8MJ+ignwGF+Rz28y5vL54PQdPGcOd37lkj31W1jQwZ+FaNu2oYPWWMu548i12VNSws7KWkrIqtpd/FMA6P9d9paCWXBTOUlB4UAOFNRERv1BvmiSDvkz+ESm4Afz6Py93/D64IJ9RQwrICgtvc5esZ8LIIQwuyKOoIJ/9ikfQHOqVA/ju505j+OCCPfb50nur+OHdwfOuXr3t/8jJyqS1LUBbIEBbm+Ppt5Zz28y5APzv11fRFgjQ2hbAOcjIMJ55azn/eXEBA/NyePyXV5CZmUFmRgYZGUZWRgb3PDuP+5+fz/DBBTzzu2v2OtY/PTKHh19exIC8bD5z+mHBUBkKl6s3V1Nd1wjAupJd/OjuWeysrGNHRQ27Kus6ehwBNm4r595Z7/b6ue4PBTX/UzhLA+pVExHxF/WmiV/Fc1bG9t6xcP96fn639/nSrx9mYF4O+bnZDMjNZkBeDrur6zvW//uF+QzIyyEzIxiwsjIzWL05OBOlGSzdsI2sDCMjFL7a2gLsqgrOQtnS2sbcJRuCoS7gaAsFvJWbgvevrGngZ/e9EOq1a6GhuYWGxma2l9cAsGl7BSf+323dHu+L763u/ROVIOFBDRTW/ELhLM0oqImI+It608Qv/Dhd/o5QEOrKfbPe63Kdc/CTe57vcn1za9sePXudNbW08uzbK7pcn2oU1vwhw+sCxDsbS0Z0/IiIxIuZnW1mq81snZl9P8L6XDN7NLT+XTOb5EGZnquaktvxI5Joet1JZ4Wb3B4/EpmZXWJmy80sYGZHhi0/08zeN7NloX9Pj2Z/6jkTQOepiUh8mFkm8DfgTKAEmG9mTzvnwr+OvhqocM5NNbNLgd8Bn018tf6hYY+SSApmEo1IAU29awB8AHwKuKvT8l3A+c65rWZ2EPAiMK6nnSmcSUQa/igiMXI0sM45twHAzB4BLgTCw9mFwM9Cv88Ebjczc05XuAUFNYkvBTPpj6561NIptDnnVgKYWefli8JuLgfyzSzXOdftG7nCmfRIvWoiqcOajdzNObHc5XAzWxB2+27n3N1ht8cBW8JulwDHdNpHxzbOuVYzqwKGEfzWUcJ0/iCtsCZ9pVAm8RTv0OZBW9ZfnwYW9hTMQOFM+kBhTUTC7HLOHdnzZhIP6lWT3lAgE6/5+Ny1btsyM3sZGB1h1Y+cc091t2MzO5DgcP2zoilE4Uz6TWFNRLpRCowPu10cWhZpmxIzywKKgN2JKS91qFdNuqJQJtI/zrkz+nI/MysGngS+6JxbH819FM4k5hTWRCTMfGBfM5tMMIRdCny+0zZPA18C3gEuBl7V+Wb9p7AmCmUi3jGzwcAs4PvOubeivZ/CmcRdpKn6FdhE0kPoHLLrCc5SlQnc55xbbmY3Awucc08D9wL/MbN1QDnBACcxFumDugJb6lEgE0ksM7sI+CswAphlZoudcx8HrgemAj8xs5+ENj/LOVfW3f4UzsQTCmwi6cM59xzwXKdlPwn7vRG4JNF1iXrXUoHCmIi3nHNPEhy62Hn5L4Ff9nZ/CmfiGwpsIiLeUu9aclAgE0ldCmfiawpsIiLe6ioIKLQlhoKYSHpROJOkEymwgUKbiEgiKbTFnoKYiCicScpQaBMR8V5PASPdw5sCmIh0R+FMUl5XoQ0U3EREEq034STZgpyCl4j0l8KZpLXughsovImIeElhR0TSjcKZSDcU3kREREQkURTORPqhp/AGCnAiIiIiEh2FM5E4U4ATERERkWgonIn4QDQBrp2CnIiIiEhqUjgTSTK9CXLtFOhERERE/E/hTCQN9CXQhVO4ExEREYk/hTMR6VF/w113FPxEREREghTORMRT8Qx+IiIiIskkw+sCREREREREROFMRERERETEFxTOREREREREfEDhTERERERExAcUzkRERERERHxA4UxERERERMQHFM5ERERERER8QOFMRERERETEB3QRahERkT5qy4GaidZxu3CT87AaERFJdgpnIiIiMRIe1CC2YS0zM4P8nGzyc7PJyswgMzODmvpGqmobyczMYOq44WRlZpBhRmZmBpkZRunOKraX15CVmcGUccMJBAK0BRxtbQFaAwEqaxqoa2zGDDIyMmhrC8SsXhER6T2FMxERkTgJD2tDBuRTXJlH0cDQT0E+O8preG/lZgB+/dVzKcjPJT8ni/zcYAh78b3V3P30O+RkZfL2nTfstf/7Zr3L3598i4L8HB78yeV7rf/bE2/yz+feY8Tggojr//DQqzz66mL2GTOMR2/+Ei2tbTQ0tdDQ1EJjcwu3zXyD1xevZ+LoIVx74fFU1zVSFfqprmvkvRWb2FFRS252FgPysqmua6QtoN5DEZG+UjgTERGJgRP3mciEIUWMLCxgZOFARhYWsG7nbn47ey4Az371CwwvGLjHfV6av7ojnI0bXoRzjoamFsoqa2lsamHrrioAmlvbuOPJt2hobqGxqYXm1jYCAcf60l0A1DU08+3bn6K1LUBbIEBbW4CAg9KdlQBU1NTzrb/+L9SjFuxVy8zIYPnG7QBU1jZwx5NvkRcKhcEeuiyq6xoBGJCbw9Ti4QwamEfRgDyysjIB+OZtT7KjopajD5jALV//JABVdY3srKhlZ2Utf/nvXNaX7mLs8EHsWzyCnZW17Kyso7y6TiFORCQChTMREZE+Gj+kqOP3G08/gYPGjKI1EGBXbR1lNXW0hA0T/O3suQSco7KhgaqGJiobGqiob6Au1Lv2qQce7nYY5L2z3u1yXWtbgNcXr+9yfWNzK3OXbOhy/e7q+m73v3LTDi656V8dtwfm5VBUkEdFTQMAG0p38/sHX6WoII8hhQMYMbiAkUMKaAsEj/+4Ayfxgy+c0XH/tkCAsoparvvTTErKKtlv/AgmjR5K6a4qSsoqqQqFQhGRdKNwJiIi0kfVDU0MDv3+zcdn0dDcQnl9AwG3d8h65oNVPe4vnuesxVJdYzN1jc0dt0t3VfHYnMVdbv/Ce6tYvnE7IwYXMKJoICOGFDJu+CAqqusBOPOoaVx5ztEd29fWN1Gys5Jrfvcojc2tTB4zlKzMDDZtr6C5tS1uxyUi4jWFMxERkT6qamzsCGdbKqpivv9kCWs9qWtoZtWmMlZtKou4/p5n5/HCvJWMG1FE8cjBjBtexIghBTQ2twJw1bnH8Ilj96ctEKB0ZxWrN5exeG0pj766OIFHISISfwpnIiIiSSJVp+1vam5l/dbdrN+6O+L6u59+h7lL1jN5zDCmjB3GQfuMYcKoIR3h7FufPYX6xhbeW7mZpeu30qpZJ0UkSSmciYiIJKFU6VWLxpaySraUVe6xLD83u+P3yWOGcdT+E7jm/GNpaGph4ZoSnnlrOS8vWJPgSkVE+kfhTEREJAWkU1gDaGhq6fj96395goH5ORw5bTxH7z+BYw6YyD5jhwHBEPeDL5zB/JWbeXfFJsoqar0qWUSkRwpnIiIiKShVh0B2pa6hmdcXr++YtTLDgsc/fuRgjt5/Auccuz8QnHnytYXreOrND9hVVedZvSIikSiciYiIpLjOvWqQ+oGtfcbMNVt2cva372LKuOGccPAkTj18KtdddAJvLt3Arqo6Jo8ZysD8XJZ/uI0Ik2yKiCSUwpmIiEgaSrdhkOtLd7G+dBf/fmEBw4oGsjvUa3bZWUfwyZMOpqyiltcXr2POwnW8v6aENk0qIiIeUDgTERGRiL1rkJqhbXfYcMZb/zuX91eXcNrhUznv+AO55LTDWF+6i8/+9N8eVigi6UrhTERERLrUVWiD1AhuNfVNPD9vJc/PW0ludhbHHjixYybIDDP+euOneG/lZp6ft1KTiYhI3CmciYiISJ90F9zaJVOAa2pp7ZhQBGBIYT652Vl8/dMn8bWLTuS9lZt49u0VvLZoXccFskVEYknhTEREROImmgDXH/EMf7ur67nmd49SPHIw5x53AOcetz+//PI53HDrk7y17EOyszJpaW2L2+OLSPpROBMREZGkFe/wB7CSKlYufoc/LX6HIyeMY2HNVtomGt86/ThOmTqZhxYs4Zllq6hvael5Z72QTL2OIhIbCmciIuIJM7sE+BmwP3C0c25BF9ttBGqANqDVOXdkomoUCeeA+ZtLO26v2rGTk6ZM4uZzz+C7HzuJp5at5OH3l7Ju5+6YPF4y9zqKpIuu2jIzmwSsBFaHNp3nnLu2p/0pnImIiFc+AD4F3BXFtqc553bFuR6RXnlu+RqeW76Gw8aN4XNHHsJnDj+IIQPy+dYTzwGQaUabjy+eFsvwp6Anaay7tmy9c+6w3uysx3BmZvcB5wFlzrmDIqw34FbgHKAeuMI5t7A3RYiISPpxzq0ECDYj8aW2TOJpcek2Fpdu47cvvc6AnBwApo4Yxr8u/zQzFy/n4feXsL06tWd67G/QU7iTZBXrtiyanrP7gduBri748Qlg39DPMcAdoX9FRMRnMptj/iFouJmFD0e82zl3dywfgOBospfMzAF39XH/96O2TOKsoqGRioZGAIxgaPvy8Udy9XFH8OwHq7nnnQUxG/KYavoa7hTq0lMStWWTzWwRUA382Dn3Rk936DGcOefmhsZMduVC4N/OOQfMM7PBZjbGObct2qpFRCRp7eruHDAzexkYHWHVj5xzT0X5GCc650rNbCQw28xWOefm9qZItWWSaGt37uZrjz3D2KJCrjhmBpccfjBn7T+Vk265m7rm2E4cks56G+oU5qQL8WjLtgETnHO7zewI4H9mdqBzrrq7QmJxztk4YEvY7ZLQMjVoIiJpzjl3Rgz2URr6t8zMngSOBnoVzqKgtkziYmtVDb9+6XX+/sa7HDx2VEcw+8nZp/Haug+Zu26jtwWmGYU56Yu+tGXOuSagKfT7+2a2HtgPiDj5VbuETghiZl8BvgKQOWxwIh9aRESSkJkNBDKcczWh388Cbva4po62LKtoiJelSBKpbGjkjfWbABhRMJDT9tuHy446jFU7dnLP2wt4bvlqX08ekq56E+YU5CScmY0Ayp1zbWa2D8Fh8xt6ul8swlkpMD7sdnFo2V5CYzfvBsidXKxXsEgPcjfneF1C0mua0Ox1CdIFM7sI+CswAphlZoudcx83s7HAPc65c4BRwJOhE62zgIeccy/EoZw+tWV548arLZNe21lbx5m3/5NzD5rGl48/ij9e9Am+edrx/N+jT7O6TJOSJisFufTUVVsGnAzcbGYtQAC41jlX3tP+YhHOngauN7NHCJ48XaUx+pKuFKb8J5F/EwXB3nHOPQk8GWH5VoKzJuKc2wAcmoBy1JZJQrUGAjy1dCVPL13Jqfvuw6cPO5CN5RUAjBlUyPbqGvTxPXUpyKWObtqyx4HHe7u/aKbSfxg4leAsJiXAT4Hs0IPeCTxHsBFdR3D64St7W4SIHyloSW/F+jWjsBc7asvErxwwZ+0G5qwNjnbKzsjggS9eQmVjI7e8+hZvbtjkbYHiuWiDnEJcaohmtsbP9bDeAV+LWUUicabQJclCr9XYUVsmyaLNOW6b+w7fOPk47r3sU7y7cQt/evVNlpRu97o08Tn1xqWGhE4IIhJv+jArIiLJLOAcTy1dyXPL1/DZGQdz3YlH89hVn+Pz9z/K+1u2el2epAj1xvmXwpkkFYUvERFJBy1tbTwwfzFPLF7O+QdP7whmh40bw4rtZTS3tXlcoaSDvl4cPBIFvegonImvKHyJiIh8pL6lhUcXLgOgIDeHey+7iLKaOn74zEssKtGcNZI8Yhn0UlmG1wVI+sjdnNPjj4iIiERW29TM1//7LLlZWTx0xWf5wZmnkJel79lFUon+R0tMKWCJiIjEz9sfbub8u/7Nt08/kSuOncFp++3DRf94kLpmze4qkgoUzqRPFMJERMDl6BwKSby65hZufmEOz69YwzGTxncEs0wz2pxekyLJTOFMuqUQJiLSvaYJzXqvFE/M31zK/M2lABw4eiS3fPpcfvbcK7z94WaPKxORvlI4E0AhTESkP9ovGK73UvGKmdEWCPDPyz/NYwuX8buX51LbpKGOIslGE4KkIU3EISISH+0hTSTRPti2g0/+4wH+8fZ8Pn3Ygcy69oscPbHY67JEpJfUc5biFLxERBJLwxyTS7IF6u5eW02tbfzxlTd5ceVa/vDJszlm0nje21SSwOpEpL8UzlKMPhCIiHhPAc0fki14RSOaY1rAFs594X6a2lppm+CYMWIsFeua2FpVk4AKRaQ/FM6SmBp+ERH/UkBLjFQMYLFQ39oCQIYZfzjhHIZ/bCD/7+3neWHzmh7vq9etiHcUzpKI3ixFRJKLAlp8KJBFL+AcV74yk7+efAF3nnYRV78yk1dK1nd7n2ieX72uReJD4czH9MYnIiISpEDWd5trKrn4+QeYfeE1fP3Q43sMZ9GI9PfQ5xaR/lM48xm9sYmIiAQpkMVOSyDAXR+8y2+OP5tjRo3n3R1bYv4YCmwi/adw5gN64xIRkVgq3OQiLq+ZaAmupO8UzGLv8fUfsLuxnvfiEMy6osAm0jsKZx7RG5OIiMRKV2Es2u38FtoUzOKjOdDGS1vWel2GAptINxTOEkhvPCIiEgvRhrHe7M8vAU3BLP6uPuBIpg0ewffeft7rUjp0/rvrM5OkK4WzONObi4iIxFKsg1nn/XoZ0hTMEqMwO5fP7HsI96yYz5rKXV6XE5F61yRdZXhdQKrJ3Zyzx4+IiKS3WAaOeAWzRD9GZ00TmhXMEuj+Ve9T19LMtQcd43UpvdL+OtHrRVKZwlkMKIyJiEi8JTI0FW5yCXs8fchOvMqmRh5as5gLJh9AcUGR1+X0mcKapCLPwpk1+2Nse1+od0xERBLJi96sRDyuPlB7554V8wk4x1cPPNrrUmKmc1jT60uSkafnnEUKNn77j6TwJSIiXvIqmHV+/Fifi+a39j7d7Kiv5eb5L7N8d5nXpcSVJhqRZOO7CUG6+08Tzzdy/WcVERG/8TqYxYuCmT88sHoxAENz86lqbqTNpebrLZzCmvid78JZd/QfSEREklHThOZet2F9DWZF65siLq+aktun/bWL1XT7Cmb+8+cTz6O4oIg/LprLC5vXeF1OQmlWSPGbpApnItI7qfKtu1+uvyTiV10Fsu626W9Y6wsFM396eO0SvnP4Sdx52kUs2bWNPyycy5vbNnpdlmcU2MRLCmciSSBVQlZf9ff4Fe7ED/rSe9adaAJZNPdPVEhTMPOvFzevYfaWtXxqnwP55mEn8sBZn+V7bz3PY+uWel2abyiwSaIonIl4KN1DV6L09nlWmBO/628w67yveAc0BTP/CzjHzPUf8PSHK/nsvofw/KZVAMwYMZbaluY+X6x6YuFgCrJzGZidzcCsHAZm51BSW8XiXdsw4OoDjiIrIyP4YxlkZmTwflkpr5VuICcjk+/OOJm2QIBWF6At4Gh1AeZt38z8shLys7K5ZOrBtAbaqG9toa6lmdqWZjZUl7OjvpYMMwZmZVPX2kIgDufTKbBJPCicicSJglfyiuZvpwAn8dTdazCWwSx8n/EKaApmyaU50MZ/Vi/quP3DI09jxohxLN21DYDMjAwWlJXw8/deAeDxT1zOqAEFZFoGWRlGXmY2L21Zy7ffnAXA8+dfyYDsPQPLQ2sWszi0vx8fdfoe61oDAe5Z/h6vlW4gLzOLz+13aEdoy87IBOBPi95gflkJg3PyuPmYM/c6hpvfe4X7Vi5g8qChvPLJawBoaG2hprmJ8qYG/rRoLrO3rGPUgAI+O/UQdjfWs7uxnvKmenY31FNaV01jW2ufnj8FNukvhTORflAAS189/e0V3iSSWA9t9DOFstRw9SuP85UDj+aQ4WM6erAqGhs61i/dvY3C6lxaAwECztHY1sriXVs71n/nredoDQSobWmmrjXYs1XeWA+AAw566BZaAgHaAgHaXIDwd9bqliYOeugve9STaR+9t+5oqOXwR24jOyNzj565LbWVAFQ01vOL+a8yMCubgdk5FObkMixvALUtwdfm5MIhfOvwk/Y65q+8+gQvbVnLjBFjuf6Q49laV83W2mpK66opratieXkZDa0tUT+HXf1fSJf3AukdhTORKCiESW8pvEk8xKPXLHzfseo9S5ZgNql4Z8Iea2PJiIQ9VixVNTfyh0Vzu1zf3oPWlec2re52fXtQilb4dP8B56hoCgXFhr23LW9q4N4V87vc17wdW9j3P39gSO4AhuXlMyxvIEPz8lm0MxguB2TlMDJ/IIcNH8PQvAEd9zvvmfv5oHwHZ43fl8/tdygbqstZX7WbDVXlrK8qZ2djXVTHotAmkSiciXSiICaJ0NXrTKEt9fW19yzaYJazqiTi8ubpxb1+zL7wSzBLZPCKRrT1JGuIS1YtgQBlDbWUNdQCe/6N3ty2kfOe3QhAflY2YwcUMrZgEOurywHIzcxiZH4Bx4wav8fQzaMevZ2djXV8rHgK04eMZFVFGcvLy9heXxNVTQpt6U3hTNKagpj4TXevSQU3/8nJaWVS8c6Ef6DuKoD15T7hoa2/vWdeBDO/hbD+6ul4FN680dDawvrq8o5gBvDMxpU8s3ElBowZWMg+g4YxadCQjp6zE8dO4sr9j+zYfndjPYt3buXqVx8HoCgnj+rmRqL9JNLd/y8Ft9ShcCZpQ0FMkp1ew/7V24DW296z8F6zvgSz7uSsKolpQIu3VAtjvdXV8Su0eccBW+tq2FpXs8f14X7+3iv8ceEbTB8yggOHjeLAoaPIz/roo/edp36Sg4ePZmX5TpaX72DRzq0sKCuhtK661zX09MWIwlvyUDjr5KYjcrj05PPJHFhMW10Jj8x9hl+8748hGtI7+iArIonkRQ9aJNfdOI5zr72cjMJiAjUlzLrzAe64pdTrsvos3cNYtBTa/KmutZn3d5by/s69/w8+tGYJR1Tu4sCho7hk6sFcsf8RzC39kC++/BgAF+1zIGsrd7GyomyPc+36QuEteSichbnpiBwuO+sKLGsgAFkFE7jsrCuA+xXQfE5BTET8oDcBLdres970ml134zjO/9YNHe1Y5qAJnP+tG4Bbuw1oseg9i+WQRgWy2In0XCqw+UP7sEiADDOmDx5BVkYGAAXZOfzpxHPJMKOupZkPdm9n8a5tPLdpNUtClyGIpWj//yrExZ/CWZhLTz6/o0FrZ1kDufTk8/nF+497VJVEojAmIn7V/mE4mg/A7R+IOn/gqZloHe9z7SGpaH1TR4DqKqSde+3lEduxc6+9nDtu+V3E+3SeKCRSKOvufMdYhDKFscTq/HwrrHkv4BwrKso6bte2NHPCzDs4cmQxM0aO5dBhY7hi/yMoqa1iya5tjB5QyDcOPZ53t2/h3R1bop5spL96+/9dYa73FM7CZA6MPJNVV8slMRTERCQZhX8A7unDb6RetPZAFB7S2nvRupp5MaOw6+U9zdaY6FCmQOYf6l3zp231NXv0rmVnZJBpwZ61yYOGcN6k/fn8focBsKmmgne3b+GvS99mS22VVyXvxS+ztyYThbMwbXUlZBVMiLhcEkdhTCQ9mNkvgAuBAFAGXOGc2xphuy8BPw7d/KVz7l+JqzI2oulN664XDYLvjeG9aJG01ZaQVRihHavtuh1LVChTGEsu6l3zn5ZAgBYCALyzfTOHPXIr+w8ZybGjx3PMqPGcNWFfbln8JgDnTZrOMaPGM6d0A+9s39yri2ZL75jZJcDPgP2Bo51zC0LLs4F7gBkEM9e/nXO/6Wl/CmdhHpn7zB7nnAG41joemfuMh1WlNgUxkbT2B+fcTQBm9g3gJ8C14RuY2VDgp8CRBCdFe9/MnnbOVSS62Fjob0jrPNSxsyeef4ZLPrV3O/bE889EdQ5ZrEOZAlnq6E1PsCRGwDmWl+9gefkO7l2xAIOOafknFAzmU1MO4gvTZ9DU1sq727fwasl67l/1vpclp6oPgE8Bd3VafgmQ65w72MwGACvM7GHn3MbudqZwFiY46cf9mq0xjhTGRKSdcy58vuiBEPFyPx8HZjvnygHMbDZwNvBw/CuMn2g+6EYKaZ2HOnb2+5eagPv51CfOJ7OgmLbaEp54/pnQ8q51FcoUyCQS9ar5U/i7wt8/mMc9K+Zz1KhiTh23D6eNm8KnphzYEc4u2+8wSmqrmLdjC01trd4UnCKccysBzPZ6H3XAQDPLAvKBZqDH6yQonHXyi/ebNflHDCmMiUh3zOxXwBeBKuC0CJuMA7aE3S4JLUsZPfWmhQek9qDWXQ/XTaubuWl1p3asFxcw720gUxgThTV/ag608da2Tby1bRO/WjCHgVnB949MM759+EkMzRtAXUszr5SsY9bG1bxWukFBLbZmEhy6vw0YANzY/kVjdxTOJGYUxET8L7PRdXm+Uh8NN7MFYbfvds7d3X7DzF4GRke434+cc085534E/MjMfgBcT3AIY1rqa29aLCiQSSxpCKQ/1bUG/5+3OcdxM+/guNETOGv8vnx84n5cMPkA/rL4Tf6y5C2yLIPMjAxfBzW/tWVd7PNooA0YCwwB3jCzl51zG7orROFM+kWBTCTt7XLOHdnVSufcGVHu50HgOfYOZ6XAqWG3i4HXelFfUurpw22iZ0BTGJO+UlDzp6a2Vl4r3cBrpRu46d2XOGbUBDbVBE/lPWXcZG47+YJ061GLVVsW7vPAC865FqDMzN4ieP60wpnEjsKYiMSKme3rnFsbunkhsCrCZi8CvzazIaHbZwE/SER90RiU1ciZo/cue/b26TF7jEQOGVMIk3hSUPOnNud4e/umjtsltVU8tWEFZ4d61Gpbmnhp81p+9u7LVLfEtLcq1W0GTgf+Y2YDgWOBv/R0J4Uz6ZbCmIjE0W/NbBrBqfQ3EZqp0cyOBK51zl3jnCsPTbk/P3Sfm6MZs++1SIGtXX+DmwKUpAIFNf9aXbmLH857kZvefYljR0/gvEn7c8iw0dSEgtkpYyezvrqcEh9dT81LZnYR8FdgBDDLzBY75z4O/A34p5ktBwz4p3NuaU/7UziTPSiMiUiiOOc+3cXyBcA1YbfvA+5LVF3x1l1w6yyWPXB+0pvnIJZS9flMdgpq/tTmXMeEIu0yzPjjiecwIr+Aeds388+VC3hx89pu9pL6nHNPAk9GWF5LcDr9XlE4S3MKYyIi/hXPHrh48Cp0Raur+vz4XKarScU7FdB8LOAcFzz7by6aciCX7Xc4t550PtMf/LPXZaWUDK8LEO8omImIiIhIb2yrr+Hvy+bx5IYPyMxQlIg1PaMiIiIiItIry8vLeHL9cq/LSDka1igiIiIiIr3y/KbVPL9ptddlpBz1nImIiIiIiPiAwpmIiIiIiPTKjYedyIrP3+h1GSlH4UxERERERHol04zszEyvy0g5CmciIiIiIiI+oHAmIiIiIiLiAwpnIiIiIiIiPqBwJiIiIiIivbJo51YeWL3I6zJSjsKZiIhIEjpz9CqvS9iD3+rpjWSuXcQrr5Ss5+fvvcIBQ0YyKDvX63JShi5CLSIikqQUKmInmudy9vbpCahEJLncdvIFFBcMYtam1TywahGLdm31uqSkpnCWBgo3Oa9LEB8pWt/U731UTdE3ZCKSfnoKcApvko5ueOMZPrffoVw4+QA+PeUgFu3cyh8WzuXt7Zu8Li0pKZylGAWx1BGLEBUv8axNwU9EklVX4U2hTVLZ8vId/HjeS/xmwWt8eupBXLn/EQzOzQOgMDsHM6O62b+fafwmqnBmZmcDtwKZwD3Oud92Wn8F8AegNLTodufcPTGsUyJQEEsefg5aftPb50phTqKltky8Eim0KbBJqqlrbebfqxbyn1ULMTMArtj/SK476Bhmrv+A+1e+z4bqco+r9L8ew5mZZQJ/A84ESoD5Zva0c25Fp00fdc5dH4ca055CmL8peHkr2udfIS69qS0Tv+kc2BTWJFU4wLngZ9eXNq9hfEERn933EL44fQavbFnHvSsWoAGPXYum5+xoYJ1zbgOAmT0CXAh0btCknxTC/EsBLPn19DdUeEt5asvE1xTWJBWtrtzF995+nt8vfJ3Lph3O5dMO58oDjuBhrwvzsWjC2ThgS9jtEuCYCNt92sxOBtYANzrntnTewMy+AnwFIKtoSO+rTREKYf6kAJbeuvv7K7ilhLi0ZUVj8uNQqsieYU1BTZLdrsZ6bl3yFncsm8fgXL1vdidWE4I8AzzsnGsys68C/wJO77yRc+5u4G6AvHHjUzqhKID5kwKY9EVXrxuFtpTT67Zs3IGD9WYvcadeNUkVzYE2yhpqvS7D16IJZ6XA+LDbxXx0sjQAzrndYTfvAX7f/9L8TeHL/xTEJN4U2pKK2jJJGe1hTSFNJPVEE87mA/ua2WSCDdmlwOfDNzCzMc65baGbFwArY1plgil4JReFMPEbhTZfSru2TFKfhj6KpJ4ew5lzrtXMrgdeJDj98H3OueVmdjOwwDn3NPANM7sAaAXKgSviWHOfKHClBgUxSWaRXr8KbImRKm2ZSFdSpTdtY8kIr0tIeZOKd3pdgmZr7EZU55w5554Dnuu07Cdhv/8A+EFsS9ubAlZ6URCTdKDAljh+actE4ilVQprszQ+hSuIvVhOC9Fpms8KWfERBTOQjCmwi0l9njl7ly4CmnrHIFLyknWfhTNKXgphI7+n/jYj0llcBTQEsOgpkEonCmcSNPkyKSKobklnPxYMWel1Gr82snuF1CZKi+hLMcjfnxKESaJrQHJf99ocCmfRE4Uz6TSFMRCS59DdQKtwlj0T2nkUTzOIVxKJ9LC8CmwKZ9IbCmfSKgpiIiPQl3CnQpbaeglkiQ1l32utIVEhTMJPeUjiTiBTC/ClnVYmnj988vdjTxxeR5BVtoFOIi7149551F8wihbJETAhXM9G6XZ+7OSfuAU3BTPpC4SzNKYR5y+uw1Vt9qVeBTkR6I5oQpwDnH10Fs656yhI1U3ekx+kc2BLdiyYSDYWzNKEQlnjJFrziJZrnQQFORHqjpwCn8La3ePSeRQpm3Q1fDA9Msf5cEs3lRgo3uYg9avEIaeo1k75SOEsxCmGJo/AVO909lwpuItJb3YW3dA5usQxovQlmnXux4vFZJXyf3QW1rgIaJGaoo0hPFM6SlEJY4iiEeUvBTURiqavgls6hrbf6GswS9dml8+N0DmvtNXXVi9bfgKZeM+kPhTMfUwBLHAWw5NTV302hTUR6K1JoS8XA1t/es2iDWXe9ZfFqc7t6729/7EghLV4BTaSvFM48pPDlDQWx1Bfpb6zAJiK9dfGghSkZ0PqqL8Gs82edeLbB4fuO9J5ftL5JAU18L8PrAkREREREREThTERERERExBcUzkRERERERHxA4UxERERERMQHFM5ERMRTZvZtM3NmNryL9W1mtjj083Si6xMREemKmf3BzFaZ2VIze9LMBoeWDzOzOWZWa2a3R7s/hTMREfGMmY0HzgI2d7NZg3PusNDPBQkqTUREJBqzgYOcc4cAa4AfhJY3AjcB3+nNzhTORETES7cA3wNcTxuKiIj4jXPuJedca+jmPKA4tLzOOfcmwZAWNYUzERHxhJldCJQ655b0sGmemS0ws3lm9skElCYiItIXVwHP92cHugi1SIq57sZxnHvt5WQUFhOoKWHWnQ9wxy2lXpclPmGNzbG+COxwM1sQdvtu59zdHY9n9jIwOsL9fgT8kOCQxp5MdM6Vmtk+wKtmtsw5t75fVYuIr6ktk+74qS1zzj0V2uZHQCvwYH8KUTgTSSHX3TiO8791A5Y1EIDMQRM4/1s3ALeqUZN42eWcO7Krlc65MyItN7ODgcnAEjOD4DCQhWZ2tHNue6d9lIb+3WBmrwGHAwpnIilKbZl4oE9tWTszuwI4D/iYc65fw/Q1rFEkhZx77eUdjVk7yxrIudde7lFFIpE555Y550Y65yY55yYBJcCMzsHMzIaYWW7o9+HACcCKhBcsIgmjtkySiZmdTfDc6Qucc/X93Z96zkRSSEZhca+Wi/iRmR0JXOucuwbYH7jLzAIEv1D8rXNO4UwkhaktkyRzO5ALzA6NBJnnnLsWwMw2AoOAnNA502f11IYpnImkkEBNCZmDJkRcLuJnod6z9t8XANeEfn8bONijskTEA2rLJJk456Z2s25Sb/enYY0iKWTWnQ/gWuv2WOZa65h15wMeVSQiItI7assknSmciaSQO24p5Zk/30pb9WacC9BWvZln/qwTqEVEJHmoLZN0pmGNIinmjltKueOW33ldhoiISJ+pLZN0pZ4zERERERERH1A4ExERERER8QGFMxERERERER9QOBMREREREfEBhTMREREREREfUDgTERERERHxAU2lLyIi0kcZGQMZkHsizjURcI0410hboIq2QBkAZnk41wQ4bwuVPrt40EIAZlbP8LiS2Dlz9Cpmb5/e6/tNKt7JxpIReyxrmtBM7uacPZbVTDQACjc5qqbkAlC0vgmA5unF5Kwq6UvZ3WqeXtzt+vY6wrXX2VnThOZeP/6k4p29vo9IJApncdb+ZiTeikdDIMml/TXQUwMu0hvZmcWMH/nYHstqG16idNcVAOwzZh5ZmSNxrgXnmnGumeqGpyir+CEAE0c9D2QG19GCc03UNrxIZe0/ARg15A841xQW/ppoaJ5PQ9M7QDaDBlxAwDWF7h/crqV1M61tW4FMsjOLcTR3PHb7Yygs9l57SIPUCGpnjl7V8Xtvglp7CAkPae1hJtqQloj34UhhrHNdnfUmlCVzGAv/23vlda8L8DGFsz5Q4EoeCmXSmUKaxFJL6yY2l12EkYdZLhmWS2vgow9tu6v/SmZGEWY5GDmY5dDYvLRjfXPrRjIsHyMbs5zg75YXWptNQf7HMHIxC+7fLIPd1bfR0PQOGRkFjBn2171q2ln5W8prbiMrczT7jH1nr/U7Km6isvZecrL2ZcLI/4UCWyg80squqt9Q2/ACOdnTGDX4V2HrW4BWymvupLF5ETlZ+zKk8Gqca8PRAqF/q+oeoaV1IzlZUynIPwcI4GgLrW+jpv4Z2gJlZGYMJTNjOC2tG3H0vqfCS+FBDZI/rLV/WO9tSIvUiwaRQ1rhpuAXAp170mKluzAWXkdnyRrI/BCwJD4UzlDYSjUKZBINhTSJhYBrpKHp3S7XV9be2+39t+2+rpu1Lazf2vlDf/ZHjx2oYsO240OhLzf0k0Nr6xYA2gKVbNv9jWAwtJxQAMyloWl+qPYaquufCFuXDZZNW6Aq9AjBD7IZNhDLyMbIBssiwwoByMocSUH+2RhZYFmhfWRR3/hWMJxlT2fE4O/vdVSNzYtoay6jIP8TjB76B5xro6VtC80t62luXU959W20BcpDx9rS7fPnF6nSq9bbkBapFw0ih7TwXjSILkz1R1e9Y+2iCWVehDGFLkmJcKZwJQpk0lcKaZJcwsNKgJbWjV1u6Vwd1fUzu1zf2radssqbulzf3LKKLTsv7nJ9fdNbrN96WJfraxtmsXrLRMwygQyMTLBMAoFaAOoaX2Pr7q+RkzWVnOwp5GRNYUDu8eyuugWAYYO+yZDCq6lvfJvahueobZhNwFV1+Xh+kQpBLdYhDT4Kaj2FpnjrKZTFO5ApfElPPAtnmY1OoUr6TGFMYk0hTSTWHNA+HDJ0llvYqW6tbaXU1D/Z6T7WsVFD83tk1Q9jYN4ZFA44G+daqGuc03E+XzJI9qDW2/PSIg11bNfVkMd4inbIYqwDmQKY9EdK9JxJelAgk0RQSBPx0kfprb7xdeobg9MG5OUcSkH+OWTYgI71o4b8nubW9dTWP09L2+aEV9pb6RLUOged7nrTvBKrMKYQJvGgcCa+pTAmXgp//SmoiXirsXkJjc1LOm6b5ZGXcyiDCy5n5OCf0ti8nLrGl6mu/x/NLas9rDQ66RLUoOewFk+xCGEKYLEzNGcc+xTMoL61CpjldTm+pXAmvqAgJn6m3jQRf3GukU07Pk525gQK8s+mIP9shhZ+jZbWLTS3rCYzYyQF+WdS1/garW2lXpfbrWSf+bFzeOltWPMLhbD4GJO3L4cOOZMpBTMYnDMagOVVcz2uyt8UziThFMQkWak3TcRfWto2U1F7NxW1d5NhhTgCAAzMO5nRQ/8AQFPLGuoa51DX8BoNTe/4ftr+zmENkiuwdRdy+nLh61hQ8EoUY1TeZKYUHMEHVa9R3bKT4bnjOajoVDbWLeGdXY+zvnYhVS07vC7U1xTOJG4UwiSVKaiJ+EvA1XT8Xl0/k8bmxQzMO42BeacxuOBLDC38KutKD6MtUEZezuEY2TQ0LyIZputP9t61dgpJqSc3YwAHFZ3KhIEHM3HgwQzMGgxAVctOlle9xorqN1hePZeAa/W20CSicCb9ogAmsvf/A4U1Ee81t66juXYdFbX/wCyfvJxDaQuUATC08OsUDjibuobXKNn1eY8r7b1k712Tj0T6WyaLmdUzMMvg42OupaZ1N+trF7KxdjEb6hZS11oJQJvz/5cffqNwJlFTEBOJjs5RE/EX5xpoaJrXcXt7+Y1kZBSQlTXGw6pEkl9jWy1/XXMVNa27vC4lZWR4XYCIiIhIIgVcFYFAhddliCS1/QedxAXjvqVgFmMKZyIiIiIi0iuj8iZxQNHJXpeRcjSsUURERNJOdf2TZGYM9boMEZE9KJyJiIhI2qlteMHrEkRE9qJhjSIiIpJ2MjOGk5WpCUFExF8UzkRERCTtjBryK4pHPOR1GSJJqynQQG1LuddlpBwNaxQRERERkV55Z9dM3tk10+syUo56zkRERERERHxA4UxERERERHrloKLTuHj8j7wuI+UonImIiIiISK8Mzy1mauFRXpeRcnTOmYiIiKSdqrpHyMgY7HUZIiJ7UDgTERGRtFPXOMfrEkRE9qJhjSIiIpJ2sjLHkZ01xesyRET2oHAmIiIiaWfk4J8wbvg9XpchkrTq22oob9rqdRkpJ6pwZmZnm9lqM1tnZt+PsD7XzB4NrX/XzCbFvFIREZF+UFsmIhI77+3+H3ev/z+vy0g5PYYzM8sE/gZ8AjgA+JyZHdBps6uBCufcVOAW4HexLlRERKSv1JaJiEgyiKbn7GhgnXNug3OuGXgEuLDTNhcC/wr9PhP4mJlZ7MoUERHpF7VlIiIxdNjgs/j8xF95XUbKiSacjQO2hN0uCS2LuI1zrhWoAobFokAREZEYUFsmIhJDg3NGMWHggV6XkXISOiGImX3FzBaY2YLmlrpEPrSIiPiQmX3dzFaZ2XIz+30X23R7rliihbdlFeUBr8uRPqqovZ9dVRq5KiL9Y2Z/CLVjS83sSTMb3Gn9BDOrNbPvRLO/aMJZKTA+7HZxaFnEbcwsCygCdnfekXPubufckc65I3OyB0ZTn4iIpCgzO43gUMJDnXMHAn+MsE0054pFIy5t2ZChmvQ4WTU0vUNtwwtelyEiyW82cJBz7hBgDfCDTuv/DDwf7c6iaVXmA/ua2WQzywEuBZ7utM3TwJdCv18MvOqcc9EWISIiaek64LfOuSYA51xZhG2iOVcsGmrLZA/ZWVPIzT7I6zJEJMk5514KDYUHmEfwyz8AzOyTwIfA8mj312M4Cz3Y9cCLwErgMefccjO72cwuCG12LzDMzNYB3wI8H3YiIiK+tx9wUmja+tfN7KgI20RzrliP1JZJZ0UDL2XYoK97XYZI0qppLWd7w3qvy/Cbqwj1kplZAfD/gJ/3Zgfm1ZeCZrYTqAN2eVKAt4aTnscN6Xvs6XrckL7HHq/jnuicG9HXO5vZCwRri5U8oDHs9t3OubvDHu9lYHSE+/0I+BUwB/gGcBTwKLBPeG+VmV0MnO2cuyZ0+wvAMc6562N4DH2mtiwtjxvS99jT9bghfY9dbRndt2XOuadC2/wIOBL4lHPOmdkfgfecc4+Z2c+AWufcXsP3O8vqx0H0i3NuhJktcM4d6VUNXknX44b0PfZ0PW5I32P363E7585O8OOd0dU6M7sOeCIUxt4zswDBxnZn2GbRnCvmGbVl6XfckL7Hnq7HDel77H49bj+1ZQBmdgVwHvCxsC8YjwEuDk12NRgImFmjc+727vblWTgTEZG09z/gNGCOme0H5LD3N7Qd54oRDGWXAp9PZJEiIiJdMbOzge8Bpzjn6tuXO+dOCtvmZwR7zroNZpDgqfRFRETC3AfsY2YfEJzo40uhoSBjzew56PpcMc8qFhER2dPtQCEw28wWm9md/dmZ1z1nd/e8SUpK1+OG9D32dD1uSN9jT9fjjlpo9sXLIyzfCpwTdvs54LkEltZb6fq3TtfjhvQ99nQ9bkjfY0/X446ac25qFNv8LNr9eTYhiIiIiIiIiHxEwxpFRERERER8IKHhzMx+ZmalofGYi83snC62u9HMlpvZB2b2sJnlJbLOWOvFcQ82s5lmtsrMVprZcYmuNdaiPfbQtplmtsjMnk1kjfEQzXGb2Xgzm2NmK0Kv9xu8qDXWevF6P9vMVpvZOjNLmetJmdm3zcyZWcQpfs3s96G/90ozu83MLNE1Sv+oLVNbprZsj23UlqktU1sWQ16cc3ZLd3P8m9k4gte8OcA512BmjxGcnev+BNUXL90ed8itwAvOuYvNLAcYkIC6EiGaYwe4geAJ/4PiXE+i9HTcrcC3nXMLzawQeN/MZjvnViSovnjq6f95JvA34EyCFxWeb2ZPJ/uxm9l44CxgcxfrjwdOAA4JLXoTOAV4LRH1SUypLeua2jK1ZWrLkpjaMm/5dVhjFpBvZlkE39S3elxP3JlZEXAycC8ET5R3zlV6WlQCmVkxcC5wj9e1JIpzbptzbmHo9xqCjfk4b6tKmKOBdc65DaFJIR4BLvS4pli4heB0ul2dzOsIXugyB8gFsoEdiSlNPKC2TG1ZylNbprYMtWUx5UU4u97MlprZfWY2pPNK51wp8EeCaX0bUOWceynRRcZBt8cNTCZ44dV/hoZD3GNmAxNcY7z0dOwAfyH4RhBIXFlxF81xA2Bmk4DDgXcTUln89XTs44AtYbdLSPLG3MwuBEqdc0u62sY59w4wh+B72zbgRefcygSVKLGltkxtWSR/QW2Z2rIkprbMezEPZ2b2cmh8feefC4E7gCnAYQT/mH+KcP8hBL91mAyMBQaa2V5TLftNf4+b4DesM4A7nHOHA3VAUoxdjsHf/DygzDn3fkIL76cY/M3b91MAPA580zlXnYja+ytWx55sejjuHwI/6eH+U4H9gWKCDfjpZnZSd/cRb6gtU1umtkxtGWrLurq/2rI4ivk5Z865M6LZzsz+AUQ6WfYM4EPn3M7Qdk8AxwMPxKzIOIjBcZcAJc659m+bZpIkDVoMjv0E4AILnmybBwwyswecc77+IBOD48bMsgk2Zg86556IYXlxFYNjLwXGh90uDi3zta6O28wOJvghfIkFz4kuBhaa2dHOue1hm14EzHPO1Ybu9zxwHPBGXAuXXlNb1j21ZWrLOq1TWxaktkz6LdGzNY4Ju3kR8EGEzTYDx5rZAAu+Mj5GcPxy0ormuEMv+i1mNi206GNAUp9QClEf+w+cc8XOuUkET5h/1e+NWU+iOe7Q6/teYKVz7s+Jqi3eovx/Ph/Y18wmW3DCgEuBpxNRXzw455Y550Y65yaFXsclwIxOjRkE399OMbOs0IeZU0jy97d0pLYMUFumtuyjbdSWqS1TWxZDiT7n7PdmtszMlgKnATcCmNlYM3sOIPRt20xgIbAsVGOyX528x+MO+TrwYGi7w4BfJ7zS2Iv22FNNNMd9AvAFgsMBepyeOYlE8/+8FbgeeJHgG/pjzrnlXhUcT2Z2pJm1Tw4wE1hP8L1tCbDEOfeMZ8VJX6ktU1umtkxtmdoytWVxYc51NRGLiIiIiIiIJIpfp9IXERERERFJKwpnIiIiIiIiPqBwJiIiIiIi4gMKZyIiIiIiIj6gcCYiIiIiIuIDCmciIiIiIiI+oHAmIiIiIiLiAwpnIiIiIiIiPvD/AYXwZQGt1WsdAAAAAElFTkSuQmCC\n",
"text/plain": [
"<Figure size 864x432 with 4 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAA2cAAAG4CAYAAADBvJ+2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAACOqklEQVR4nOzdd3xb1f3G8c+xvGPH2TvOTggJJOw9y95Q9t6lFNpSuoDSQlu6aKG0UCA/yihllBUIexMSyCB77+HYieO9bcmWzu8PycZxPGRb8tV43q+XW+vq6tyvZKKjR+fcc421FhEREREREXFWgtMFiIiIiIiIiMKZiIiIiIhIRFA4ExERERERiQAKZyIiIiIiIhFA4UxERERERCQCKJyJiIiIiIhEAIUziUvGmEnGmGXGmEpjzA978LjZxpgqY4yrp47ZEWPMtcaYuU7XISISSYJ9vzbGXGGM+aiHajLGmGeMMaXGmIU9ccxmx37fGHNNTx6zI8YYa4wZ73QdIqGkcBbHjDFHG2O+NsaUG2NKjDFfGWMOcbquYBhjthljTupGEz8HPrfWZlpr/xGqulpqWae1Nsdam2Gt9YbrmCIiscoY80UgmKSE+1jBvl9ba1+w1p7SrMZwBoajgZOBEdbaQ8N0DIwx9xlj/tt8m7X2dGvtc+E6poj4KZzFKWNMb+Ad4J9AP2A4cD/g7kJbxhiT0GJbYijqDKNRwGqnixARkeAYY0YDxwAWOMfZahwzCthmra12uhARCQ+Fs/g1EcBa+5K11mutrbXWfmStXQF7f2tmjBkd+DYwMXD7C2PMA8aYr4AaYGzg/h8YYzYCGwP73WSM2RQYmZtljBnWrM1TjDHrAyN3/zLGzDbG3Bi4b5wx5jNjTLExpsgY84Ixpk/gvueBbODtwJSTnwe2Hx4YCSwzxiw3xhzf2hM3xnwGnAA8Gnj8xMDzubHZPntM9Qs8t1uMMRsD7T9mjDHN7r/JGLM2ME1yjTHmwNbqbOV1HBZ4XUoCr9NNzdq8zxjzijHmP4F2VxtjDm7jOT1ujPlri21vGWN+Evj9l8aYzc3qO7+Ndvaor9nfuvlrc33guZYaYz40xoxqrS0RkRC7GpgPPAvsMb3OGDPSGPOGMaYw0G88GtjuMsb8NdCPbAn0Uc3fg/eY3dC872vl/fraQBuVxpitxpgrmm2fG/j9y0BTywPv+5cEtrfXF7bbvzTb7wbgKeCIQNv3t+yrmrU3PvD7s4H23g3UvcAYM67ZvlOMMR8H6tptjLnbGHMacDdwSeA4ywP7NvUFxpgEY8yvjDHbjTEFgX4qq8Xrdo0xJifw2t/T2h/UGHOYMSbfNJs6aow53xjT+FnkUGPMvMDrsssY86gxJrmNtjrqx/dp9lzXG2Mubq0dEacpnMWvDYDXGPOcMeZ0Y0zfLrRxFXAzkAlsD2w7DzgM2NcYcyLwR+BiYGhgn5cBjDEDgNeAu4D+wHrgyGZtm8BjhwGTgZHAfQDW2quAHODswJSTvxhjhgPvAr/HPxL4U+B1Y8zAlkVba08E5gC3BR6/IcjnexZwCLB/4DmdGnguFwVquxrojf8b3eLW6mylzZeB3MDzvBD4Q+B1a3ROYJ8+wCzg0TZqewl/R2oCNfUFTgk8FmAz/m+cs/CPkP7XGDM0yOfdxBhzLv5O+wJgIP7X8aXOtiMi0gVXAy8Efk41xgwGfwDDPxNkOzAa/0yQxve+m/C/dx8AHIz/fbbTjDG9gH8Ap1trM/H3V8ta7metPTbw67TA+/7/2usLm2m1f2nR9r+BW4B5gbZ/E2T5l+J/3+8LbAIeCDynTOAT4AP8fdB44FNr7QfAH4D/BY4zrZU2rw38nACMBTLYu386GpgEfAf4tTFmcivPaQFQDTTv9y4HXgz87gXuAAYARwTaujXI590k8Pf7ONDuIPyvyb+MMft2ti2RcFM4i1PW2gr8b5wW+D+gMPBt3uBONPOstXa1tbbBWlsf2PZHa22JtbYWuAJ42lq7xFrrxh/EjjD+qSlnAKuttW9Yaxvwd3r5zerbZK392FrrttYWAg8Bx7VTy5XAe9ba96y1Pmvtx8CiwHFC5U/W2jJrbQ7wOTA9sP1G4C/W2m+s3yZr7fY2WwkwxowEjgJ+Ya2ts9Yuw/+t6NXNdpsbeE5e4HmgtU4S/CHJ4g9g4P8AMs9auxPAWvuqtXZn4LX5H/6Rza6cr3AL/r/x2sDf7Q/AdI2eiUg4GWOOxj+l7xVr7WL8XzhdHrj7UPzh4mfW2urA+2njiMnFwN+ttTustSX4Q1JX+YCpxpg0a+0ua22wU+Pb6wsbtdW/hMJMa+3CwHv2C83aPgvIt9b+LfCaVQbCUjCuAB6y1m6x1lbhf06Xmj1Pabg/MCtnObCctvuvl4DLoCkwnhHYhrV2sbV2fuBzxjbgSdr/LNCWs/BPB30m0NZS4HXgoi60JRJWCmdxLPAB+1pr7QhgKv7O7e+daGJHB9uG8e2IGoE38GL832oOa76vtdbiH0ECwBgz2BjzsjEmzxhTAfwX/zdnbRkFXBSY+lBmjCnDHz47PTrUjvxmv9fg/6YQ/KN6m7vQ3jCgxFpb2WzbdvyvT1vHTDWtnM8XeP1eJtDB4f/Q8kLj/caYq41/dcrG12Yq7b+ebRkFPNKsnRL8o5zD232UiEj3XAN8ZK0tCtx+kW+nNo4EtgfCR0t79DU065M6I3CO1yX4v6DaFZgmuE+QD2+vL2zUVv8SCqHuu6DFcwr8ngg0/4I32Of0InCB8S/ycgGwpPELTuM/7eCdwNTHCvxfCHa17zqsxWeEK4AhXWhLJKwUzgQAa+06/PP4pwY2VQPpzXZp7Q3MdrBtJ/43RKBpWkF/IA/YBYxodp9pfhv/G7AF9rPW9sY/MtZ8Dn7LY+8AnrfW9mn208ta+6dWamxNMM+3LTuAcW3c19pr1Ggn0C/wTWGjbPyvT1e8BFwYGMU6DP+3ggRu/x9wG9DfWtsHWMWer2ejxpPM23otdgDfa/E6p1lrv+5izSIi7TLGpOEfATsu8CE9H/9Ut2nGmGn435eyW/viCn9fM7LZ7ewW9wf93m+t/dBaezL+L/3W4X9fDUZ7fWF37VG/MaazfdfYNu5rr++CFs8J/+vaAOzuxPH9B7J2Df5wdzp7TmkEeBz/az0h8Fngblrvu6D9v+UOYHaLvivDWvv9ztYrEm4KZ3EqcGLsncaYEYHbI/GPuswP7LIMONb4r/OShX/KQme9BFxnjJke+EbsD8CCwNSEd4H9jDHnBTrUH7DnG2kmUAWUB84n+1mLtnezZ6fyX+BsY8ypxn8CeKox5vjG5xeEZfi/uUs3/hOpb+jE83wK+Kkx5iDjN77ZNL+WdTax1u4Avgb+GKh3/8Bx/9va/h0JTNMoCtTzobW2LHBXL/wdbSGAMeY6vg3hLdsoxP+B4crA63g9ewbPJ4C7jDFTAm1lBc65ExEJl/Pwn3u0L/4pedPxn4s8B/808IX4Q9ifjDG9Au+nRwUe+wrwQ2PMiMC5uL9s0fYy/NPxkox/waVWz0kLzOY4NxCs3Pj7J18b9bZ832+vL+yu5cCUQNupBM7NDtI7wFBjzI+NMSnGmExjzGGB+3YDo02LlZibeQm4wxgzxhiTwbfnqLU2ehmMF4EfAccCrzbbnglUAFWBkcr2wtQy2u7H3wEmGmOuCvytk4wxh7R2HpyI0xTO4lcl/tGVBcaYavyhbBVwJ0DgnK3/ASuAxfjf2DrFWvsJcC/+EZxd+D/kXxq4rwj/XO+/4J/esS/+c8Qal/K/HzgQKMcf5N5o0fwfgV8Fpif8NBB0GherKMT/LdnPCP6/8YcBD/4O6TmaTQkM4nm+iv8E6xfxv65v4l+UZK86W3n4ZfhPYN8JzAR+E3jduupF4CSaffMY+Fbyb8A8/M9vP+Crdtq4Cf9rVwxMwR8gG9uaCfwZeDkwxWQV/m87RUTC5RrgGeu/7lh+4w/+BSiuwD+Scjb+BS1y8E+RvyTw2P8DPsQfYpawd19yL/6+qRR/v/MirUsAfoL/vboE/3lPbQWF+4DnAu/7F7fXF3aX9S9o9Vv8C3tsBOa2/4g9HluJ/5ppZ+OfgrgR/wIf8G1AKjbGLGnl4U/jPw/6S2ArUAfc3oWn0Ogl/K/pZ82mroJ/ca/L8fet/4f/c0lb2uzHA8/1FPyv+078z/fPQNivlyfSWcZ/qoqIswLfzuUCV1hrP3e6HhERiT2BRTi2AkndGOUREQkbjZyJYwJTEPsEpnk0ziOf38HDRERERERiksKZOOkI/CtFFeGfVnFeYAl+EYlxxpg7jP/C6quMMS8FzpcRERGJSMaY04z/AuabjDEtz18N3XE0rVFERHpSYJGfucC+1tpaY8wr+K9T+KyzlYmIiOzN+C92vwH/eZq5wDfAZYHz+kNKI2ciIuKERCAtsFprOv6T9EVERCLRocCmwIXXPfivLXtuOA6kcCYiIj3KWpsH/BX/ynq7gHJr7UfOViUiItKm4ex5Qftc9ryQfMi0dsHGHtG3X4IdNsKxw4uIxIw1K+uLrLUDu9PGUcen2rKSti7b1Ol6VuNfWrvRDGvtjMYbgetNnQuMAcqAV40xV1pru3SNv56SlJVmU4dkOV2GSKclGRcA9dZLAoaR6QMpq6+mvL4aFwmMzxxGQV0ZpfVVuDAMTu1HWX0VNV53By2LE1ISkuif0puiunI8toHMxDSGpfVne/Vu6nz19HKl0j+lN7tqi6m3XlwmgQQM9dbrdOlNqjbs7na/dczxqbY0RP3W6g76rZ7kWDpKGdqH058/udX7rur7davbRURkb9NH5W7vbhtlJT5efGdwKMph+qjcOmvtwe3schKwNXDRc4wxbwBH0sULsPeU1CFZHPSvK50uQ6RDl2QfR0FdGZ8XLMdgeP/4B5iVN49/bXwbgD9Ou54Pdi1idsEKACZkDie3pojaZmFsqCOVS2c0/o3SXMlkpw+iX3U+Hl8DB/WbwFWjT+Ke5c9Q7a3jwpHHcNvEczn3y19TXl/DgX3Hs0/vkbyS8yUNDgW22Sf9rdv9VmmJj9ffHRCKctgne1dH/VYeMLLZ7RGBbSEXkUNXz5ce2eZ9Cm4iIlEvBzjcGJMO1ALfwX8RehEJUkZiGlUN/gWO79r3Utzeeh5a/zoApw49mHUVO/i8YDkWywOrXyS35ttrO9+1/Ok92tpYGZbPmNJDar0e1lfmNt1eXLKRxSUbm24vKF5HzRo35fU1ABzUbwLnDj+SF7f7Lyt7xagTmZyVza9WPAtAakIydT5Pzz2B6PANMMEYMwZ/KLsU/wXSQy4iw1l7FNxERKKbtXaBMeY1YAnQACwFHJk+IhIN+if3ZmT6QJaVbQbgd/tdw6DUvnzvm78DUOyuwOP79praNy/8+x4jInMKV/VovRJZdtQUsqOmsOn2/21+n/9s/aTpdr31Utvw7ajpPVMuo29yJrctfhTwj6wWucsp9VT1XNERxlrbYIy5DfgQcAFPW2tXh+NYURfO2qPgJiISHay1vwF+43QdIpGof3JvpvYZ3TTt8OoxJ3HSkAM5a/a9WCwf5i8mMzGtaf8Zm9/b4/FOTVWT6OH21Tf9/krO7D3um12wglRXctPtu/e9jN11pfxy+b8BOKjvBLZW51PiqeyZYiOEtfY94L0Od+ymmApn7VFwExERkUjUJymDwwdM5vPdy3D76jlh8DRum3guF839HYXucl7fMYf3d33TtP9cjYRJGH2ye+ketx9c+wqNV0VOMi7+MO16ZuXN47GNswA4cfB0lpVujruwFi5xE87ao+AmIiIiPSUlIYlD+09iTXkOxZ4KJvYewS/3vYT8uhKWlW7m091LWVK6iSJ3BQA5zaakifS0NRU5Tb83WB+3LX6Umgb/wobD0wbw66lX8tC615mVN4/UhGQO6DeexSUb9phqK8FTOOtAW8FNoU1ERESCNTi1LwbIrytlQEpvfrf/tTyyfiYzc79ieelmrp3/INuqdwNQ6qmK6/N7JHJZ7B4LyOysLeaGBX9r+iLhwH7j+cO06/nJkidYUrqJ3onpJCUkUuypcKrkqKNw1kUabRMREZG2GAy9k9Ipr68mybh49vCf8cGub3hk/Uzyaou59Zt/sr7Sf01bt6++KZiJRBOLZXPVrqbbC4vXc+fSJ1lRthWA04cdyvfGn8F35/6WUk8Vaa5kar1aCbI9CmdhoNE2ERGR+PbwgbfQ4PPy02UzqLdeHlj9IlubfYhdU9HtyzyJRJwG691jGf+5hauoaqhtGgm+bcK57Js1irH8zakSI57CWQ/SaJuIiEhsOmvYYZw29JCm5cdn5c3DZ23T/VrEQ+JRXm0RebXfXmNvfvFatlbnO1hR5FM4ixAabRMREYkeEzKHc2n28Ty07nWqvXXUeN0UeyrISEylqqGOz3Yvc7pEkYija+51TOEswim0iYiIOC8zMY2zhh/OV4WryKkpJNWVzAF9xzEifQDrK3P5bPcyBTIR6TaFsyil0CYiIhJe/ZIzSXUls7O2mKSERG4cdzoV9TXk1BSyqmwb3537Oyy244ZERIKkcBZjFNpERES6zmUS8FofCRj+fdidLCrZwAOrX6TEU8lFc3/XdKFdhTIRCQeFszjRWmhTYBMREfnWbRPOYVLvkdy++DF8WB5c+wq5zS4A3RjMRETCReEsjmmUTURE4tnwtP6cPuxQnt78Ab7A9ZrcvnoSMPiwfF20xukSRSTOKJzJXjTKJiIiscplEkjAUG+9jM0YysXZx/FlwQo2VObx/q5vnC5PROKcwpkERYFNRESiXZ+kDGYc+mNe3P4Zb+Z+zddFa7ho7u8or692ujQREUDhTLpB0yJFRCTS7dt7FINS+/BFwXLK6quYU7iSbdW7AfBan4KZiEQUhTMJOY2yiYhIpLhs1PGMzRjK7IIVWCz/3PCW0yWJiLRJ4Ux6hAKbiIj0hH17j+K2iedy9/KnKauv4h8b3qSqoVZL34tIVFA4E8cosImISCikuZJJTkiivL6ayoYaUhISGZiaRVl9FYXucqfLExEJmsKZRJSWgU1hTURE2pNkXDx/xC/4unAND61/nR01hdyw8CGnyxIR6RKFM4loGl0TEZGWxmYM5cC+43ltxxzqrZdnt3zElqpdTpclItJtCmcSdTS6JiIS304YNI3zRxzFB7u+oaqhjnd2LnC6JBGRkFA4k6in0TURkdiWldSL748/i7fy5rG2IoeXtn/O/3JmU9VQ53RpIiIhpXAmMUmjayIi0c9gsFg8vnqm9x3PyvKtrK3Iocbrdro0EZGwUDiTuKCwJiISXa4YdSIH9pvAnUufpNbr4cp5f6LBep0uS0QkrBTOJC4prImIRJ6MxFSqG9xYLGX1VeTXlZCckIjH16BgJiJxQeFMBIU1ERGnZacP4h8H3crD699gdsEK3t25kHd3LnS6LBGRHqVwJtIKhTURkZ7RODKWW1PIl4UryakucLokERHHKJyJBEFhTUQk9C7JPo6zhx/O9Qv+hsfXwEPrXne6JBERRymciXSBwpqISNclYPBhWV+Ry9C0jSQaFx4anC5LRMRxCmciIaCwJiLSsZSEJP48/UYWFK/jpe2fs6xsM8vKNjtdlohIxFA4EwkDhTURkb25ffXk1RZR6ql0uhQRkYiU0NEOxphUY8xCY8xyY8xqY8z9reyTYoz5nzFmkzFmgTFmdFiqFYlSz5ce2fQjIuGlfiuyTMwczr8O/iH9kjMBeHDtq3ywa5HDVYmIRKZgRs7cwInW2ipjTBIw1xjzvrV2frN9bgBKrbXjjTGXAn8GLglDvSJRT6NqImGnfiuC1Ho9pLmS6ZecSYlGzERE2tXhyJn1qwrcTAr82Ba7nQs8F/j9NeA7xhgTsipFYphG1URCS/2W88ZnDOOS7OMA2FFTyPUL/samqp0OVyUiEvmCOufMGOMCFgPjgcestQta7DIc2AFgrW0wxpQD/YGiENYqEvM0qiYSGuq3nHXK0IM4cfB03t25gKqGOuxe2VhERFoTVDiz1nqB6caYPsBMY8xUa+2qzh7MGHMzcDNA5tD0zj5cJO40D2sKaiLBC0e/lTIoM7RFxpgRaQMAyK0t4qnN7/PfrZ9S1VDncFUiItGlU6s1WmvLjDGfA6cBzTu5PGAkkGuMSQSygOJWHj8DmAEwZEo/fY0m0gkaVRPpvFD2W5mThqjfaoPLJPDXA75HTk0BP1/2f3h8DXh8um6ZiEhndRjOjDEDgfpAB5cGnIz/xOnmZgHXAPOAC4HPrLXqxETCSKNqIq1Tv9VzspLSKa+vwWt9/GHNS+TVaFaoiEh3BDNyNhR4LjB/PwF4xVr7jjHmt8Aia+0s4N/A88aYTUAJcGnYKhaRvWhUTWQP6rd6wMj0gTx60G08uvEtPs5fwoqyLU6XJCIS9ToMZ9baFcABrWz/dbPf64CLQluaiHSVRtUknqnf6hl5NUV8XrCcteU5TpciIhIzOnXOmYhEH42qiUioZCamcdP4M3hi4zvUeN38ff0bTpckIuI4Y8zv8F+ixQcUANdaa7t0/RCFM5E4o7AmIl01JmMoJw8+kNm7V7C4dKPT5YiIRIoHrbX3Ahhjfgj8GrilKw0pnInEOU2BFJGO9EnKoKy+ihVlW7j06wcor69xuiQRkYhhra1odrMXdP3ijgpnItJEo2oi0tKRA/bl3qlXcueSJ1hTkaNgJiLSCmPMA8DVQDlwQlfbUTgTkTZpVE1EVpRt5f2dC9lavdvpUkREmpR603mt4sAQtfbuAGPMomYbZgSuc9nEGPMJMKSVB99jrX3LWnsPcI8x5i7gNuA3XalE4UxEgqJRNZH4kZGYxndHHs3zWz+hqqGWf2x40+mSRETCqchae3B7O1hrTwqyrReA91A4E5GepLAmEruOHjiVK0d/h/lF61hfucPpckREIpoxZoK1tnGVpHOBdV1tS+FMREJCYU06wxjTB3gKmIr/xOnrrbXzHC1Kmnyw6xuWl25mV12J06WIiESDPxljJuFfSn87XVypERTORCRMWoa1RgptEvAI8IG19kJjTDKQ7nRB8W5MryH8ct9LuX/V8+ysLVYwExEJkrX2u6FqS+FMRHpUa6FNgS2+GGOygGOBawGstR7A42RNAj4sLpNAcoI+GoiIOEXvwCLiuLZG2RopvMWcMUAh8IwxZhqwGPiRtbba2bLi06CUPhS4y9hevZubFj6M7frleUREpJsUzkQk4nUU3lqK1TDX9uvwSrfbLvZmdPp1btsrHS1JnAgcCNxurV1gjHkE+CVwb4gKkCDt2zubvx90K39a8zKf7V6mYCYi4jCFMxGJOaELGdJFHS1JnAvkWmsXBG6/hj+cSQ9bX5nLKzmzWVi83ulSREQESHC6ABERiS/W2nxgR2BlK4DvAGscLCnujM0YSkpCEl7r46nN71PVUOt0SSIigsKZiIg443bgBWPMCmA68Adny4kfaa4UHjrge9y5z4VOlyIiIi1oWqOIiPQ4a+0yoL2pjxImtV43f1n7Cluq8p0uRUREWtDImYiISBxIc6UwPmMYAF8XrSFf1zETEYk4CmciIiJx4PsTzuKRg26ld6Ku9y0iEqk0rVFEIp6npp7k9CSnyxCJas9s+ZBvitdT0VDjdCki3ZaRmEa/5AxcxsXWav8U3f37jKV/cibJCUkkJbhITkiisqGGj/OXAHDykAPJTEzH46vH42vA42ug2FPByrKtAIzpNYQG66XEXUm1t65b9XlrPbjSkrv3JCUuKZyJSESrKqzl6bPf5/q3TydjYJrT5YhEnUmZI1lfuYNSTxVzClc5XY5Iu9JcyYxMH8jg1L70S+5NckIir+74EoAfTjyPIwdMoV9KJskJ/o+wOdUFXD3/LwBcP/ZUpvcdt0d76ytym8LZJdnHMT5z+B73Ly7ZyJ1LnwTg9/tfy/D0AQC4vfWUeCqZW7iKxzbOAuCikcfi9tVT4q4gv66UHTWFuH31ez0Hd3EV31zzNIc8dz0p/TNC9dJInFA4E5GItumzPOrrGtj0eR7TLx7vdDkiUWVcxlD+dcjt/GvjLF7fMdfpckSa9EvOJLvXIEakDeCdnf5LHv5w4nlcMPLoPfarbqhrCme768pYVraZEnclJZ4KSjyVlLgrm/Z9cO0rJCYk4vE2joz5/7/RDxY9SnJCIskJSf7/dyXR0Oz+v657jf7JmfRL6U3f5Az6JWdS5C5vuv+GcaeR6tpzNOyVnNn8a+PbAJw57FB21BSx4NPP8NbVU/zVJoadMz00L5jEDYUzEYloOZ8UcNUpZ7Dgk5UKZyKdtLlqFw+te53Pdi91uhQRThg0jQuzjyE7fRCZSd+e+/hV0WpKPVUsKtlAsbuCnJoCdtWWUOKppLy+umm//+V80W77ebXF7d7v9tUHRrpav67f0tJN7T7+zNm/ok9SBv1SMhmW1p/s9EFsqtoJwICU3vxs8sX+HQ+6lapfVrB2y0beca1mdsGKdtsVaU7hTEQiVk1JHTvXFPHwi3cw6vKzqSl1k943xemyRCJev+RMAEo8lbwbGJUQ6QnJCYlMzBzBlKzRTM0axZQ+o/n50v9jU9VO6q0Xt7eBT3YvJae6wP9TU0CppwrwryL6dVHkXo/ea30Ueyoo9lSwsTJvj/uK3ZVcPPf3DCcL16zt/P7WH1NXVUOdy3+O56TMEfx+/+tYXb6NVeXbWF2+nY2VeTRYrxNPRSKYwpmIRKxNX+zkxEMOoX9WH044+GA2f5HHfuePdboskYj3i8mXMDitLzcs+Bte63O6HIlhLpNAckIitV4Pk3tn88hBtzadD7ajppCFReuaAsjcwlXMjdHzHi2WAncZy9/7kgmL3aSemc+ffv0rNh2SytDT96PeellWuokpfUZz/OBpgP+8th8ufoz1lbmkJCRR72vAh3X4mYjTFM5EJGLlfFLA/Sd9D4Arjj2d+z95UuFMJAhPb/mQfimZCmYSFi6TwAF9x3P8oP05ZuB+vJE7l+e2fsz26t28ljOH1YGRobL6KqdL7XHVc7Zx1enXAnDVcafy0w+fhdP3Y0vVLh5Y8xLgH9memjWaqX1Gk1NTAMDlo0/krGGH8WXhSmbvXsGKsi0KanFK4UxEIlJdhYcdy3Zzxq+OAuDMI47ihr/9lroKD6m9tTyxSEsDUnqzX9YYPi9YzvrKHVDZ8WNEOutHE8/nxCHTyUrqRU1DHV8XrWFFYCn6Gq+bGZvfdbhC5zRU1VG8egdn3P9tv3Xd337H6Ko6EjNSm/Yr8VTyZeFKvixc2bRtVdlWRqUP4vShh3D+iKMocVfwcf5SHt/0do8/D3GWwpmIOGL32lJyFu5u8/6SrZUcc8ABZKb3AiAzvRdHT5/O7IeX0290ZpuPyz50MIMn9w15vSKR7uoxJ3PioOksLtmoa5lJSLhMAgf1ncCk3iN5ftsnAPRKTGVh8Xpm717OwpL1e6yGGOsqN+6mbGlOm/fX5JRw9PQ9+62jph/A6idnkz6yX5uP63NANt9MgG9KNpCakMxhA/bh+EHT6Jv87TL8V4w6kbUVOSwv26IR8RincCYijqgurGX+v9YypE9/zj/ueAxmzx2S4brrz95j04M3/IhnPnwb9jwPG4tl5uwvyC8rpv+Y3qBwJnEk3ZVCjdfNYxtmMXPHVwpm0m39kjM5Z/gRnD38cPqn9KaqvpaZuV9R1VDLHwJT8+KRp7iaHc/NZ2iffpx37PEY06Lf6j2c6y/as9966IbbefrDt6Fsz12ttbz55RfsKishPbs/TPBvr/N5mF2wYo8VHnsnpnPl6O+QlpjC7rpS3sqdx7s751Ner3/rsUjhTEQcMfbYYVz+4ol89IvF5BTm8/RPfk2fzLZHxAD2HzeBh2/9yR7byiorufZv99OQ2cDl/zqR/uOywlm2SES5d8oV9E/pzU+WPIHbV8/W6nynS5Iod1j/fXhg/+tIMIYFxet5e91rfFO8nnqtKkj/w8cy7V+Xs/X3H7CtcDfP3XlvUP3W31vpt67+62+pTodpf7icXqP6t9tGRUMN5825j8MHTOac4Ydz8/gzuHbMydy94hkWlWzo9vOSyJLgdAEiEr/6j8vioheOY8egXUy5+RK+XrW8U4//etVyptx0MXmD87noheMUzCTuzC9ey+yCFVo4QLosOSGR04YezBH9JwOwunwbr++Yy1Xz/sxdy//N10VrFMya6TWqP/s+dgkrsiqYfNOlXeq3Jt94CSv7VLLvY5d0GMwauX31zC5YwZ1LZ3DN/Ad5e+d81pb7p1ge3n8yJw6eTqJxdfr5SOTRyJmIOCoxxcVxd01j0+F5nPnrH3PH+Zdzz+XX43K13cl4vV5+/8K/+fubL3Hcr6cx/vjhPVixiHNSEpL44cTz+LpoDV8Vrebj/CVOlyRRamBKFueOOJKzhh1Gn+QMPtu9lHnFa6lqqNMiFB1ISE5k1O3HU3TgJk679yfcecFl/Ory6zrst373wtM8NPNlsu84gQFHju/y8bdX7+afG95qun3msEM5ZtB+FLnLeTtvPm/nzafEoxWBopXCmYhEhPEnDGfwvn157u53+GbTGt6+/+E29z3vtz9lbdUWLn7peDIHp/dglSLO8lof4zOHkVtbxFdFq50uR6LUTeNO59Ls48EYvi5czRu5X7G0dJPTZUWdAUeNJ3PiYB7/44cs3LiWd3/7tzb3Pef+n7OkPIcpj19KyoD2p0J21q9X/odD+0/i/BFHcd3YU7ly9Hf477ZPeW7rxyE9jvQMhTMRiRiZg9OZesUYCv9b3u5+RRVlTL16jIKZxAWXSeD8EUcxK28eHl8DP1j0aNNFfUWCNTAli8r6Wup8HtZX5PLqjjnMzP2K3XWlTpcW1VIGZjLgu/tT9Ob2dvcrrCij/wXTQh7MwL8o1oLidSwoXsfwtAF8d+TRbK3yn3+ampBMWmIypZ74u+ZctHLsnLMKT2rHO4lI3Nn+ST6XH3Nau/tcfsxpbPtECx9IfJiaNZrbJp7L0QOnAiiYSaf0ScrgBxPO4YUj7+KCkf7rb31ZuJInNr2jYBYilXO2ctUxp7S7z1XHnkLlnC1hryWvtoh/bHiz6RpqF4w8mhePvJsbxp5GRmJa2I8v3efogiAf5Uxq80dE4o+33svmuTu54JgTmrat2baF6x68nzXbvu3ULjjmBDbP2Ym3Xh9SJTZlJqZxSL+JACwv28KNCx7is93LnC1KokpGYirXjz2Vl468iwtGHs3H+Yv5VP8NhZyv3kvRgs1ccOye/dY1D/52r36raMFmfD3cb80uWMFXhau4asxJvHTkXVwx6kTSXMk9WoN0TsSu1qjQJhJ/chYUMGnUKIYNGIi1lifefp0jf3w9KzLXc+SPr+fJd97AWsvwgYOYmJ1NzsICp0sWCYtbJpzFb/a7ijRXCgCbqnY6XJFEm3unXMnVY05mXvFarp3/IA+ufVUjZWFQujSHiaNGf9tvzXqdw350A58l5XHYj27gybe/7bcmZI9q9yLW4ZBXW8TvV7/IDQv+xoqyrdw0/gx+NeWKHq1BOifqzjlrL6Cdkr2+BysRkVDb+kk+Vx57JqWVFVzz1/tYsmsd5z9zDP3H9Gb/i8fxu18+xXvffMWzP/0Nlx97Gi988h5jjhrqdNki3ZaRmMYl2cfxdt58CtxlvLDtM17LmUOt1+10aRJFerlSMQaqGup4dONbPLXlfTZW5jldVkyrnLOZ2489hdLKCq568H7m521k8sMXkp7dn+qz9+dXf3iWdxZ9zX9++muuPPYUHp0zm36HjunxOjdX7eKeFc8wNWs0FfXVgP+/F6/1Uefz9Hg90raIHTnrCo22iUQvX4OPzV/kMaRvf6bceDG7hxVy4fPH0n9MbwD6j+3Nhc8fy65hBUy96RKG9O3Pps/z8DX4HK5cpPvSXSlckn0ch/T3T2XcWVusC0pLp4zqNZgnD/0RP5x4PgA7agoVzMLMen0Ufb2Zof36M/mGS1g9oIZ9/3kx6dn+a5f1GtWfyf+8mFX9qph846UM7defoq83Yb3O9VuryreRU1MIwI/3uYB/HXI7Q1P7OVaP7C3qRs66oq2AppE2kcixY3EhVRW13PqvP3PCfdMZd+ywvfZJTHFx7C/2Z/PhO/nB/X+huqKW3CWFZB862IGKRbrnurGn0i85k7+te40CdxmXfP2AVlSTLjliwL78asrluL0e3s6b53Q5caNs+Q5qK2u45bEHGXXnifQ/fNxe+yQkJzLqB8dTdOBmvv/QX6mtrKFsRS59D8h2oOI9fbRrEb+eeiVPHPoj7lv5vC6nECHiIpy1RaFNJHIUri9j7CHD+M7vDiRjUPsrSo07bhiDXurDZ79eSsG6MoUziUouk0CCMRgMFqtgJl1y5ejvcP3YU9lYmcevVjxLobv9S5FI6FRtKWTQ9NGM/vnJpAzIaHffAUeMI/PxQWz7yydUbS6IiHD2TckGbvnmER6Ydj1/nX4Tj26cxczcr5wuK+7FdThri0KbSM87+OpJHHx18NOQMwenc+6TR4WxIpHQyk4fyG+mXsVf1r7C+spcntr8vtMlSZTrm5zBhSOP4dP8pTy47lU8vganS4orIy88GC48OOj9UwZkMukv54exos7Lqy3m1m/+wd1TLuOaMSfz2e6llNfXOF1WXFM464TWQpsCm4iIBKPIXUGN1920AqNIV/VLzqTUU0Wpp4qbF/6dAneZ0yVJFKvxurl3xXMMTetHeX0NBkNmYhoVDQppTlA46yYFNhERacuFI4/hsP778LNl/0eN183tix9zuiSJctP7jOP+/a7mfzmzeXH7ZwpmEhIWy87aYgCuGH0i5w4/gl+teI71lTscriz+KJyFgaZFiogIQK3XTUV9DakJyVquWrrt3OFHcvvEc8mrLeLLghVOlyMxal7RGs4cdhj/POhWHlz3Kh/nL3G6pLiicNaDWoY2hTURkdjSNzmDe6dcyZu5X/Fl4Ure3bmQd3cudLosiXKJxsWPJp3P2cMP5+vCNfx+9QvU6Bp4Eiabq3Zxyzd/5779ruaeKZczPmMYMza/h9fq0jU9QeHMQZoSKSISWyrqa7D4SE5Q9yqhMzFzBKcOPZjnt37CM1s+xId1uiSJceX1Nfx06Qx+MOEcvjvyGD4vWM66Ck1x7AnqPSKMApuISHQZlzGUK0efxB/XvITH18CdS2c4XZLEmDUV27ny6z/p/DLpUV7r4x8b3uT1HXPJqy1yupy4oXAWBTQdUkQkcvVJymD/PmMYkT6QLVW7nC5HYsgdky5gU9VO3s6br2AmjmkMZmcOO5Shaf11GZAwS+hoB2PMSGPM58aYNcaY1caYH7Wyz/HGmHJjzLLAz6/DU66AP6w1/xERkW/1RL81NmMoJwyaBsDi0o1c9vUfFMwkpA7vP5lzRxxJv+RMp0sRAWBMxlAuH3UC+2WNdrqUmBbMyFkDcKe1dokxJhNYbIz52Fq7psV+c6y1Z4W+ROmIpkKKiOwh7P3WdWNOYVzmMOYUrqLBenXxXwmpdFcKd+7zXbZU7eKFbZ85XY4IAP/e/D5HD5jCzyZfzI0LH9L7XgvGmNuBHwBe4F1r7c+70k6H4cxauwvYFfi90hizFhgOtOzkJIJoKqSIxKtw9VspCUlkJaVTXl/DQ+tfx2d9NFhvCCoW2dP3xp9Jv5Te3LvyOf03JhGj1uvhr+te468H3MzVY07W9MZmjDEnAOcC06y1bmPMoK621eG0xhYHHg0cACxo5e4jjDHLjTHvG2OmdLUgCQ9NgxSReBTKfmtYWn9uGHc6AKWeKsrra0JaqwjAiLQBnD38cF7L+VKr40nEWVSygfd2LuTS7OMZnNrX6XIiyfeBP1lr3QDW2oKuNhT0giDGmAzgdeDH1tqKFncvAUZZa6uMMWcAbwITWmnjZuBmgKSBWV2tWbpJo2oiEg9C3W9lDR/Avzd/EN6iJe7l1hZxx5InFMwkYv1r49t8tnsZu+tKnS4lkkwEjjHGPADUAT+11n7TlYaCCmfGmCT8HdwL1to3Wt7fvNOz1r5njPmXMWaAtbaoxX4zgBkA6ROG6SIdEUJhTURiTTj6rcxJQ2x5fXWYK5d41i85kxJPJcvLtjhdikibqhpqWVSyAYDMxDQqG2odqaOiIZWP8/cJUWvvDjDGLGq2YUbg/b+JMeYTYEgrD74Hf6bqBxwOHAK8YowZa63tdN7pMJwZYwzwb2CttfahNvYZAuy21lpjzKH4p0sWd7YYiQzNw5qCmohEG/VbEo0mZY7knwf/gN+t+i9zClc5XU7E2JY7MGRtjR5RGLK2BI4fNI2fT76IW755hJyaqH9ti6y1B7e3g7X2pLbuM8Z8H3gjEMYWGmN8wACg0y9MMCNnRwFXASuNMcsC2+4GsgOFPgFcCHzfGNMA1AKXdiUpSuTRqJqIRCH1WxJVEo2Ln0++iHJPFUtKNjldTtiFMnB197gKbF23vGwzDdbHzyZfzA8X/wtLXL+FvgmcAHxujJkIJANdunJ3MKs1zgVMB/s8CjzalQIkumhUTUQinfotiTaXjz6RcZnDuGv501R765wup0ucClzd1bJuhbXglXqqeHTDW9w95TLOG3EkM3O/crokJz0NPG2MWQV4gGu6+oVf0AuCiLSkoCYiItI9o3sN5qrR3+GT/CXMK4qeqxRFaxjrSOPzUkgLzkf5izlpyAHcPO4M5hWtIT9OFwmx1nqAK0PRlsKZhISCmoiISOdNzBxBmaeaf254y+lSOhSrgaw1zZ+rglr7/rbudf7v0DuYmjU6bsNZKCmcScgpqImIiATno/zFfFGwHI+vwelS2hRPoaw1Cmrt211XyqVfPUCN1+10KTFB4UzCSkFNRESkdQaDxSqYRREFtdYpmIVOgtMFSPz4KGdS04+IiEi8++7Io3n5yHtId6U4XUrMSclJbvUnlLblDlR4DeiXnMmzh/+Mk4cc6HQpUU8jZ+IIjaiJiEi826f3SBKMidhRh2gIHp0NXK3t7872dKsGLSICZZ4qBqZksW/WKD7OX+J0OVFN4Uwcp6AmEp+MMS5gEZBnrT3L6XpEetqk3iNZX7HD6TKiUihHwZq31Z2gFs8hzYdlY2UekzJHOF1K1NO0RokomvYoEld+BKx1uggRJ/RypTIyfSDrK3OdLqVVkTxqFmwwy9xu9/oJVdvtidfpjusrdjA+Yxguo3jRHRo5k4ik0TSR2GaMGQGcCTwA/MThckR63ITewwE0ctYJ7QWnYIJXy/0qR7V+rfrG44RiumM8jaKtr8wl2ZXE6F6D2Vy1y+lyopbCmUQ8BTWRmPR34OdApsN1iDii3FPNzNyvWF8ReSNnkTjq01YwCzaUdfTY1oJaSk6yAlonrCnP4YOd39BgfU6XEtUUziSqNAY1hTRnnJc0iSNG3UhC+gh8NbnM2/4Ub9brbxELKjypoZxSPMAYs6jZ7RnW2hmNN4wxZwEF1trFxpjjQ3VQkWiytTqfR9bPdLqMiNedUJa12c1tl6dz9qXn48oYgbcqlzfef5u/fLT3AiyN7bUMaaEKaBD756Ll15Xwp7X/c7qMqKdJoRKVtCx/zzsvaRJHTfwFrl7ZGJOAq1c2R038Becl6W8geymy1h7c7GdGi/uPAs4xxmwDXgZONMb8t8erFHFQdvrAiDw3J5JGzboSzLI2u5t+brs8nfOuu5HETH+/lZiZzUUXXMs9h7nafHxrbYdq8ZFIem3DaUBKb6dLiGqR964g0kkKaT3jiFE3YhJ77bHNJPbiiFE3OlSRRCtr7V3W2hHW2tHApcBn1torHS5LpMdkJKbxnyN+wSXZxzldSsTqTDBrHsiaO/vS81vtt86+9PxW92/vGApowblh7Gm8eOTdJJq2A7C0T+FMYoZCWnglpLe+PG5b20VEZG/jMoby8IG3ALCsdIvD1ewpUoJDZ4NZW1wZrfdPzbe3FdIU0LqmyF1BckIifZMznC4laumcM4k5WkAkPHw1ubh6Zbe6XaSrrLVfAF84XIZI2LlMAleM/g5Xjz6Jivpq7ln+DGsqtjtdVkRyZ3taDUKVo8xeoal8XEqbAc1blUti5t79lrfq236rfFxKq49tbYGQ7p571iiWzz0bkzGEqoZaitwVTpcStTRyJjFNo2mhM2/7U9iG6j222YZq5m1/yqGKRESiR5JJ5NQhB/FFwXKunf9Xvipa7XRJEa2tINRaaCofl9L009zbL89std96++WZre7f3jEUzIIzMXMEGyvzsHR9Fc14p5EziQsaTeu+N+vXw4Y/a7VGEZEguUwC5ww/gnd3LqDO5+GWbx6hsqHW6bKiRmdG0Bo1D1yPvlgDPLX3ao0LvG0eU8Gs61wmgfEZQ5mZ+5XTpUQ1hTOJOwpqXfdm/Xre3PQzp8sQEYl4Y3oN4Zf7XsKk3iOpaqjl4/wlER/MRo8ojLjzoboS0BqVj0vhgQVeHljwWofHaeuC1KEIZrEeyhq5TAJ/Xfca26rznS4lqimcSVzTddNERCSUDu43kWMH7sdpww6hpqGO36z8D7MLVjhdVlRrDEgtQ1owAa09bQWylsftjngJZgAeXwMf5S92uoyop3AmgkbTRESka0b1GsyU3qN4b9dCAM4ZfgSH9JvIZ7uX8cTGdyirr3K4wtjR2ihaawGro8DWUShrPFZ3xVMwA5iSNYqaBjdbNXLWLQpnIi1oNE1ERNqS5krhoL7jOXTAPhzabx+GpPUFYEHxOoo9Ffx9/RtU1tdQb9s+r0m6rq1pjs0FE746OkZ3xVswA7htwrnU+TzcseQJp0uJagpnIm1QSBMREYDs9IGUeaqpaKjh2IH7cdeUS6lpqGNxyUb+u+0TFhavp9jjXzq8xFPpcLWxr61pjqFsu6viMZQBJBoX4zKG8oYWA+k2hTORDmjKo4hIfEkyLqb1HcfhAyZzRP/JDE8fwN/Xv8GbuV/zddFqfrzkcVaVbaNBo2OOCmYUraPHh1K8BjOAsRlDSHYlsbYix+lSop7CmUgnaDRNRCT6tbYiYXKCC4/PS3piEgsu+gGZySnUNdTzdX4Oj69Ywsc7drG7xv+4lVQA/VptO54/oDsh1AGrK/Q3h316+y/2vb5ih8OVRD+FM5Eu0GiaiEhkC2ZJ+JQEF2eO3ocrJh1AVb2baz55lZqGeh5bOY/1pYXMy8+hztsQ0uPqg3xs0d/Tb3SvwZR6qsivK3W6lKincCbSTRpNExFxXmeuzzW8V2+umDSdSyZMo39qOpvLi3lv+7qm+59YtSAcJQLt16kP+tFDf6s9PbHpHRYW63NQKCiciYSIRtNERMKrOxdINoAxBp+1nDNmMt+bchif5G7iubVL+Dp/e+iK7IZgnl+4QkGkXXw6UimU7Sk5IRGXSaDW62F+8Vqny4kJCmciYaCgJiLSPaEKC5lJyVw4fj+unHQAjyz/illb1/LC+mW8tXUNO6ujb2XFUAc4hbLgKJS17uoxJ/OdwQdw08KHqGqoc7qcmKBwJhJmmvYoIhKcUAaFfilp3LrfEVw2cRq9kpJZUpBHqbsWgIp6NxX17pAdK9IocIWOQlnbJmYO57Ls4/kwf7GCWQgpnIn0EI2miYjsLVxB4tmTLmJKv8G8uXUNz6xZxKqS3WE5jsQeBbKOJRoXv5h8CaX1Vfxr4yyny4kpCmciDlBQE5F4Fo5AluJK5PKJ0/jfxhXUNNRz/8JPKXPXsrmiJOTHktikUBa8K0afyLjMYdy1/GmNmoWYwpmIwxTURCRehCOUJZoELhq/Hz+cdhRDe2VSUlfLW1vXsLgwL+THkraNHlEYVdMpFcS6zmCYkjWaj3ctZl7RGqfLiTkKZyIRREFNRGJRuD60nzNmMndMP5oxvfuxqCCXH895mwW7dRHccGov1HQl8PRUoFMYCx2L5RfLniI5QTEiHPSqikSo5kENFNZEJDqF88P3pROmUdfQwPWfvsZnuZvDdpx41ROBRqEpuhwzcCprK3Ioclfg9tU7XU5MUjgTiRIaVRORaBKOUDYgNZ2fHXgcDy2bw+6aKm6b/Ral7lpsyI8kIi1lpw/i3qlX8ln+Uv609n9OlxOzFM5EopCCmohEsnAEs8smTOOXBx1PamIis/O28N729ZQElsYXkfBKwPCLfS+m1uvmyc3vOl1OTFM4E4lymv4oIpEiHKEsOcHFX446nfPGTmHeru3cM/8jtmgFRpEe4zIJ/GzyxUzJGs3vV71AqafK6ZJimsKZSIxRWBMRJ4Tr3LIf7H8E542dwl+WzOZfK+eH5RjSum25A3VOmHBJ9vGcNvRg/r35Az7ZvdTpcmKewplIjGsZ1kCBTURCJ9yr7T25agEri/P5ZMemPban5CSH7ZjubE/Y2haJNm/mfkWhu4yP85c4XUpcUDgTiUMKbNGltb+XSCyb0m8Qd0w/hh9+OYuahno+2bEprGGspfaOFW/BTaNn8Wlan7FcMfpE7l3xHDVet4JZD1I4ExGg7QCg0BYeClwSC8IxanbcsDE8dvy5VNS4GV7ah5zS8pAfoztaBrd4CGsKaPHlnOFH8MOJ55FXW0RWUi8K3GVOlxRXFM5EpF3thQgFt28pbEm8CUcwO3/sFB486gw2FBTxvZfepKCqusttZW7fc4H9ylGmu+W1qnlYi4egJrHLZRL44cTzOHfEkcwrWsvvV71AtbfO6bLijsKZiHRZZwJJNAQ5BSyR4IQjmJ2XMZW/HX0G87fu4LZXZ1Ht6fgCty0DWHf2DUV4i+VRNY2exb4fT7qAs4cfzovbPuOpze/j0xUEg2aMmQY8AWQA24ArrLUVXWlL4UxEeoSCj0hsCEcwS8lJZmWf3by1Yi33vfcpdQ0N7e7fmVAWrMY2QznC1hjWYiWkKaDFtpe2f87S0k18tnuZ06VEo6eAn1prZxtjrgd+BtzblYYSQlqWiIiIxKxwBLOD60digNyycn4568N2g1nmdhuWYNbaMUJ5nJSc5KYfkUhy7MD9+Nk+FwGws7ZYwazrJgJfBn7/GPhuVxvSyJmIiIh0KNTBLCUnmdMmT+BvF5zBg5/M4dkFba8G19mglLXZ3er28nEpnWqn+XFDNaIW7eeoafQsNrhMAteMOZmrx5zM6vJtpLtSqPG2/u8mUnk8iaF8XxpgjFnU7PYMa+2MTjx+NXAu8CZwETCyq4UonImIiEi7whHMTt5nPH+74AyW5+7ilSUr29y3o2DWVhALdt9gA5umPX5LAS26DU8bwD1TLmPfrFG8t3Mhf1//Bh5f+1OJ40CRtfbg9nYwxnwCDGnlrnuA64F/GGPuBWYBXf5HrXAmIiIibQpHMDtqbDYPX3AGK/LyuemlmdTU7734R3uhrDOBrCPN2womqIUzpEH0BTWJLi6TwIMH3ERmYhr3r3yezwuWO11S1LDWntTBLqcAGGMmAmd29TgdnnNmjBlpjPncGLPGGLPaGPOjVvYxxph/GGM2GWNWGGMO7GpBIiIi3aF+K3RCGcwaz7lKMIZfn3Yi20vKuOmlmXutytjR+V6dCWbJ63JJXpcb9P5Zm91Btx+u89+i5dy0cJx/KOHhMgmcNPgAEo0Lr/Xxx9Uvcd2CvyqYhZAxZlDg/xOAX+FfubFLghk5awDutNYuMcZkAouNMR9ba9c02+d0YELg5zDg8cD/i4iI9DT1WxHMZy0/ffN9EhMSqHIHP0rU2VDW2m3PPiM6dazOjKQ1Cse5aa2JhBE2TW+MfEcPnMqN405ndK/BsAo+2b2UleXbnC4rFl1mjPlB4Pc3gGe62lCH4cxauwvYFfi90hizFhgONO/kzgX+Y621wHxjTB9jzNDAY0VERHqM+q3I0zJorNy5u9X92hqJ6k4wa3lfsAGt+XE7s5CIExe/bqkng5sCWmSa3mccN48/g32zRpFTXcC9K55lTuEqp8uKWdbaR4BHQtFWp845M8aMBg4AFrS4aziwo9nt3MC2PTo5Y8zNwM0ASQOzOlmqiIhI54Sy30oZlBm2OiNROKat/eCYwxjSO5PfvPcpPrtniOluMAt2+mJnR9Ga19D
gitextract_wm5wtx1e/ ├── .gitignore ├── LICENSE.txt ├── README.md ├── carbs/ │ ├── __init__.py │ ├── carbs.py │ ├── model.py │ ├── serialization.py │ ├── test_carbs.py │ └── utils.py ├── excluded.txt ├── notebooks/ │ ├── analyze_carbs.sync.ipynb │ ├── carbs_demo.ipynb │ └── carbs_simple_2d.sync.ipynb ├── pyproject.toml └── setup.py
SYMBOL INDEX (142 symbols across 5 files)
FILE: carbs/carbs.py
class CARBS (line 62) | class CARBS:
method __init__ (line 75) | def __init__(self, config: CARBSParams, params: List[Param]) -> None:
method set_search_center (line 145) | def set_search_center(self, input_in_param: ParamDictType) -> None:
method suggest (line 150) | def suggest(
method observe (line 218) | def observe(self, new_observation_in_param: ObservationInParam) -> Obs...
method forget_suggestion (line 240) | def forget_suggestion(self, suggestion_to_forget: ParamDictType) -> None:
method initialize_from_observations (line 254) | def initialize_from_observations(
method __getstate__ (line 272) | def __getstate__(self) -> Dict[str, object]:
method __setstate__ (line 278) | def __setstate__(self, state: Dict[str, object]) -> None:
method _set_seed (line 282) | def _set_seed(self, seed: int) -> None:
method _get_mask_for_invalid_points_in_basic (line 287) | def _get_mask_for_invalid_points_in_basic(self, input_in_basic: Tensor...
method _round_integer_values_in_basic (line 298) | def _round_integer_values_in_basic(self, input_in_basic: Tensor) -> Te...
method _param_space_real_to_basic_space_real (line 305) | def _param_space_real_to_basic_space_real(
method _param_space_obs_to_basic_space_obs (line 315) | def _param_space_obs_to_basic_space_obs(
method _basic_space_to_param_space (line 328) | def _basic_space_to_param_space(self, real_number_input: Tensor) -> Pa...
method _basic_space_to_unrounded_param_space (line 336) | def _basic_space_to_unrounded_param_space(
method _remember_suggestion (line 346) | def _remember_suggestion(
method _add_observation (line 358) | def _add_observation(
method _search_distribution_in_basic (line 376) | def _search_distribution_in_basic(self) -> Distribution:
method _sample_around_origins_in_basic (line 379) | def _sample_around_origins_in_basic(
method _get_probability_in_search_space (line 401) | def _get_probability_in_search_space(
method _is_random_sampling (line 418) | def _is_random_sampling(self) -> bool:
method sample_search_space (line 422) | def sample_search_space(self, num_samples: int) -> List[SuggestionInBa...
method _generate_candidate (line 456) | def _generate_candidate(self) -> Optional[SuggestionInBasic]:
method get_surrogate_model (line 586) | def get_surrogate_model(self) -> SurrogateModel:
method _get_random_suggestion (line 596) | def _get_random_suggestion(
method _observation_group_output_pos_better (line 616) | def _observation_group_output_pos_better(self, group: Sequence[Observa...
method _crank_oversampling_up (line 619) | def _crank_oversampling_up(self) -> None:
method _get_pareto_groups (line 624) | def _get_pareto_groups(
method _get_pareto_set (line 649) | def _get_pareto_set(
method _get_resample_suggestion (line 657) | def _get_resample_suggestion(self) -> SuggestionInBasic:
method _init_wandb (line 688) | def _init_wandb(self) -> None:
method cumulative_cost (line 712) | def cumulative_cost(self) -> float:
method observation_count (line 717) | def observation_count(self) -> int:
method _get_observation_log (line 720) | def _get_observation_log(self, observation: ObservationInParam) -> Dic...
method _get_suggestion_log (line 805) | def _get_suggestion_log(
method _autosave (line 826) | def _autosave(self) -> None:
method load_from_file (line 833) | def load_from_file(
method load_from_string (line 847) | def load_from_string(
method get_state_dict (line 854) | def get_state_dict(self) -> Dict[str, Any]:
method load_state_dict (line 871) | def load_state_dict(cls, state: Dict[str, Any]) -> "CARBS":
method save_to_file (line 879) | def save_to_file(self, filename: str, upload_to_wandb: bool = False) -...
method serialize (line 888) | def serialize(self) -> str:
method warm_start_from_wandb (line 894) | def warm_start_from_wandb(
method warm_start (line 903) | def warm_start(
FILE: carbs/model.py
class SurrogateObservationOutputs (line 24) | class SurrogateObservationOutputs:
class SurrogateModel (line 35) | class SurrogateModel:
method __init__ (line 36) | def __init__(
method _get_kernel (line 52) | def _get_kernel(self) -> Kernel:
method _get_model (line 60) | def _get_model(self, inputs: Tensor, outputs: Tensor, kernel: Optional...
method fit_observations (line 80) | def fit_observations(self, success_observations: List[ObservationInBas...
method _fit_target_transformers (line 90) | def _fit_target_transformers(self, success_observations: List[Observat...
method _target_to_surrogate (line 106) | def _target_to_surrogate(self, x: Tensor) -> Tensor:
method _surrogate_to_target (line 116) | def _surrogate_to_target(self, x: Tensor) -> Tensor:
method _cost_to_logcost (line 128) | def _cost_to_logcost(self, x: Tensor) -> Tensor:
method _logcost_to_cost (line 136) | def _logcost_to_cost(self, x: Tensor) -> Tensor:
method fit_suggestions (line 145) | def fit_suggestions(self, outstanding_suggestions: List[SuggestionInBa...
method fit_pareto_set (line 175) | def fit_pareto_set(self, pareto_observations: List[ObservationInBasic]...
method get_pareto_surrogate_for_cost (line 185) | def get_pareto_surrogate_for_cost(self, cost: float) -> float:
method fit_failures (line 194) | def fit_failures(
method _get_success_prob (line 209) | def _get_success_prob(self, samples_in_natural: Tensor):
method observe_surrogate (line 219) | def observe_surrogate(self, samples_in_basic: Tensor) -> SurrogateObse...
FILE: carbs/serialization.py
class Serializable (line 33) | class Serializable:
method __attrs_post_init__ (line 35) | def __attrs_post_init__(self) -> None:
method __setattr__ (line 38) | def __setattr__(self, item: str, value: Any):
method mutable_clone (line 44) | def mutable_clone(self: TC) -> ContextManager[TC]:
method to_dict (line 76) | def to_dict(self) -> dict:
method from_dict (line 85) | def from_dict(cls: Type[TC], dump: dict, is_upgrade_allowed: bool = Fa...
function _to_dict (line 120) | def _to_dict(v: Any) -> Any:
function _from_value (line 144) | def _from_value(v: Dict[str, Any], is_upgrade_allowed: bool) -> Any:
class ParamTypeError (line 183) | class ParamTypeError(TypeError):
function _get_all_subclasses (line 187) | def _get_all_subclasses(cls):
function _compute_unserializable (line 195) | def _compute_unserializable(qualnames: List[str]) -> None:
function _dedupe_subclasses_by_qualname (line 206) | def _dedupe_subclasses_by_qualname(subclasses: List[type]) -> List[type]:
function _dedupe_subclasses_by_id (line 211) | def _dedupe_subclasses_by_id(subclasses: List[type]) -> List[type]:
function get_all_serializable_classes (line 216) | def get_all_serializable_classes() -> List[type]:
function get_serializable_type_from_qualname (line 224) | def get_serializable_type_from_qualname(qualname: str) -> Type[Serializa...
function get_qualname_from_serializable_type (line 239) | def get_qualname_from_serializable_type(serializable_type: type) -> str:
function flatten_dict (line 260) | def flatten_dict(d: DictNest, prefix: str = "") -> DictFlat:
function inflate_dict (line 270) | def inflate_dict(d: DictFlat) -> DictNest:
FILE: carbs/test_carbs.py
function carbs_config (line 23) | def carbs_config() -> CARBSParams:
function params (line 28) | def params() -> List[Param]:
function carbs_instance (line 37) | def carbs_instance(carbs_config: CARBSParams, params: List[Param]) -> CA...
function test_suggest_one (line 41) | def test_suggest_one(carbs_instance: CARBS) -> None:
function test_suggest_observe_ten (line 51) | def test_suggest_observe_ten(carbs_instance: CARBS) -> None:
function test_observe (line 63) | def test_observe(carbs_instance: CARBS) -> None:
function test_forget (line 76) | def test_forget(carbs_instance: CARBS) -> None:
FILE: carbs/utils.py
class ParamSpace (line 35) | class ParamSpace(Serializable):
method basic_from_param (line 36) | def basic_from_param(self, value: ParamType) -> Any:
method param_from_basic (line 39) | def param_from_basic(self, value: Any) -> ParamType:
method drop_type (line 42) | def drop_type(self) -> Any:
class Param (line 47) | class Param:
class RealNumberSpace (line 54) | class RealNumberSpace(ParamSpace):
method basic_from_param (line 61) | def basic_from_param(self, value: ParamType) -> float:
method param_from_basic (line 64) | def param_from_basic(self, value: float, is_rounded: bool = True) -> P...
method round_tensor_in_basic (line 67) | def round_tensor_in_basic(self, value: Tensor) -> Tensor:
method min_bound (line 73) | def min_bound(self):
method max_bound (line 80) | def max_bound(self):
method plot_scale (line 87) | def plot_scale(self) -> str:
class LinearSpace (line 92) | class LinearSpace(RealNumberSpace):
method __attrs_post_init__ (line 96) | def __attrs_post_init__(self) -> None:
method basic_from_param (line 102) | def basic_from_param(self, value: ParamType) -> float:
method param_from_basic (line 106) | def param_from_basic(self, value: float, is_rounded: bool = True) -> f...
method round_tensor_in_basic (line 112) | def round_tensor_in_basic(self, value: Tensor) -> Tensor:
method plot_scale (line 123) | def plot_scale(self) -> str:
class LogSpace (line 128) | class LogSpace(RealNumberSpace):
method basic_from_param (line 133) | def basic_from_param(self, value: ParamType) -> float:
method param_from_basic (line 139) | def param_from_basic(self, value: float, is_rounded: bool = True) -> f...
method round_tensor_in_basic (line 145) | def round_tensor_in_basic(self, value: Tensor) -> Tensor:
method plot_scale (line 159) | def plot_scale(self) -> str:
class LogitSpace (line 164) | class LogitSpace(RealNumberSpace):
method basic_from_param (line 168) | def basic_from_param(self, value: ParamType) -> float:
method param_from_basic (line 176) | def param_from_basic(self, value: float, is_rounded: bool = True) -> f...
method plot_scale (line 181) | def plot_scale(self) -> str:
function log_norm_cdf (line 189) | def log_norm_cdf(z: Tensor):
function expected_improvement (line 204) | def expected_improvement(
function probability_of_improvement (line 222) | def probability_of_improvement(
function aggregate_logical_and_across_dim (line 236) | def aggregate_logical_and_across_dim(x: Tensor, dim: int = -1) -> Tensor:
function add_dict_key_prefix (line 243) | def add_dict_key_prefix(input_dict: Dict[str, Any], prefix: str):
class ObservationInParam (line 248) | class ObservationInParam(Serializable):
class ObservationInBasic (line 256) | class ObservationInBasic(Serializable):
class SuggestionInBasic (line 264) | class SuggestionInBasic(Serializable):
class OutstandingSuggestionEstimatorEnum (line 269) | class OutstandingSuggestionEstimatorEnum(Enum):
class SuggestionRedistributionMethodEnum (line 275) | class SuggestionRedistributionMethodEnum(Enum):
class WandbLoggingParams (line 281) | class WandbLoggingParams(Serializable):
class CARBSParams (line 293) | class CARBSParams(Serializable):
class SurrogateModelParams (line 344) | class SurrogateModelParams(Serializable):
class SuggestOutput (line 356) | class SuggestOutput:
class ObserveOutput (line 362) | class ObserveOutput:
function load_observations_from_wandb_run (line 366) | def load_observations_from_wandb_run(
function get_checkpoint_obs_count (line 398) | def get_checkpoint_obs_count(checkpoint_name: str) -> int:
function load_latest_checkpoint_from_wandb_run (line 406) | def load_latest_checkpoint_from_wandb_run(
function load_checkpoint_from_wandb_run (line 424) | def load_checkpoint_from_wandb_run(
function assert_empty (line 438) | def assert_empty(x: Sized, message: str = "unexpected elements") -> None:
function ordered_dict_index (line 442) | def ordered_dict_index(od: OrderedDict, value: Any) -> int:
function group_observations (line 453) | def group_observations(
function observation_group_cost (line 475) | def observation_group_cost(group: Sequence[ObservationInBasic]) -> float:
function observation_group_output (line 479) | def observation_group_output(group: Sequence[ObservationInBasic]) -> float:
function pareto_area_from_groups (line 483) | def pareto_area_from_groups(obs_groups: Tuple[ObservationGroup, ...]) ->...
function get_pareto_groups (line 498) | def get_pareto_groups(
function get_pareto_groups_conservative (line 538) | def get_pareto_groups_conservative(
function get_pareto_curve_plot (line 611) | def get_pareto_curve_plot(
Condensed preview — 15 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,074K chars).
[
{
"path": ".gitignore",
"chars": 3137,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": "LICENSE.txt",
"chars": 1062,
"preview": "MIT License\n\nCopyright (c) 2024 Imbue\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof t"
},
{
"path": "README.md",
"chars": 7579,
"preview": "# Cost Aware pareto-Region Bayesian Search\n\nCARBS is a hyperparameter optimizer that can optimize both regular hyperpara"
},
{
"path": "carbs/__init__.py",
"chars": 56,
"preview": "from carbs.carbs import CARBS\nfrom carbs.utils import *\n"
},
{
"path": "carbs/carbs.py",
"chars": 41570,
"preview": "# %%\nimport base64\nimport io\nimport math\nimport os\nimport random\nimport threading\nimport traceback\nimport uuid\nfrom coll"
},
{
"path": "carbs/model.py",
"chars": 13036,
"preview": "import math\nfrom typing import List\nfrom typing import Optional\n\nimport attr\nimport numpy as np\nimport pyro\nimport torch"
},
{
"path": "carbs/serialization.py",
"chars": 10805,
"preview": "from __future__ import (\n annotations, # using this to get Postponed Evaluation of Annotations -- https://www.python"
},
{
"path": "carbs/test_carbs.py",
"chars": 2995,
"preview": "import os\nfrom typing import List\n\nimport pytest\nimport wandb\n\nfrom carbs import LogitSpace\nfrom carbs import Observatio"
},
{
"path": "carbs/utils.py",
"chars": 20781,
"preview": "import math\nimport os\nfrom collections import OrderedDict\nfrom enum import Enum\nfrom pathlib import Path\nfrom typing imp"
},
{
"path": "excluded.txt",
"chars": 9,
"preview": "setup.py\n"
},
{
"path": "notebooks/analyze_carbs.sync.ipynb",
"chars": 16519,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"name\": \"\"\n"
},
{
"path": "notebooks/carbs_demo.ipynb",
"chars": 6968,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": null,\n \"metadata\": {\n \"pycharm\": {\n \"is_executi"
},
{
"path": "notebooks/carbs_simple_2d.sync.ipynb",
"chars": 940219,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"code\",\n \"execution_count\": 1,\n \"metadata\": {},\n \"outputs\": [\n {\n \"name\":"
},
{
"path": "pyproject.toml",
"chars": 584,
"preview": "[build-system]\nrequires = [\"setuptools\", \"wheel\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"carbs\"\nvers"
},
{
"path": "setup.py",
"chars": 496,
"preview": "from distutils.core import setup\n\nsetup(\n name=\"carbs\",\n version=\"0.1.0\",\n author=\"Untitled AI\",\n author_ema"
}
]
About this extraction
This page contains the full source code of the imbue-ai/carbs GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 15 files (1.0 MB), approximately 664.7k tokens, and a symbol index with 142 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.