Showing preview only (242K chars total). Download the full file or copy to clipboard to get everything.
Repository: mistralai/mistral-inference
Branch: main
Commit: 2557e12d0e18
Files: 25
Total size: 232.0 KB
Directory structure:
gitextract_yufqghio/
├── .github/
│ └── ISSUE_TEMPLATE/
│ ├── bug_report.yml
│ └── config.yml
├── .gitignore
├── LICENSE
├── README.md
├── deploy/
│ ├── .dockerignore
│ ├── Dockerfile
│ └── entrypoint.sh
├── pyproject.toml
├── src/
│ └── mistral_inference/
│ ├── __init__.py
│ ├── args.py
│ ├── cache.py
│ ├── generate.py
│ ├── lora.py
│ ├── main.py
│ ├── mamba.py
│ ├── model.py
│ ├── moe.py
│ ├── rope.py
│ ├── transformer.py
│ ├── transformer_layers.py
│ └── vision_encoder.py
├── tests/
│ └── test_generate.py
└── tutorials/
├── classifier.ipynb
└── getting_started.ipynb
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/bug_report.yml
================================================
name: Bug report related to mistral-inference
description: Submit a bug report that's related to mistral-inference
title: '[BUG: '
labels: ['bug', 'triage']
body:
- type: markdown
attributes:
value: |
Thanks for taking the time to fill out this bug report!
- type: textarea
id: python-vv
attributes:
label: Python -VV
description: Run `python -VV` from your virtual environment
placeholder: Copy-paste the output (no need for backticks, will be formatted into code automatically)
render: shell
validations:
required: true
- type: textarea
id: pip-freeze
attributes:
label: Pip Freeze
description: Run `pip freeze` from your virtual environment
placeholder: Copy-paste the output (no need for backticks, will be formatted into code automatically)
render: shell
validations:
required: true
- type: textarea
id: reproduction-steps
attributes:
label: Reproduction Steps
description: Provide a clear and concise description of the steps that lead to your issue.
placeholder: |
1. First step...
2. Step 2...
...
validations:
required: true
- type: textarea
id: expected-behavior
attributes:
label: Expected Behavior
description: Explain briefly what you expected to happen.
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional Context
description: Add any context about your problem that you deem relevant.
- type: textarea
id: suggested-solutions
attributes:
label: Suggested Solutions
description: Please list any solutions you recommend we consider.
================================================
FILE: .github/ISSUE_TEMPLATE/config.yml
================================================
blank_issues_enabled: false
contact_links:
- name: Documentation
url: https://docs.mistral.ai
about: Developer documentation for the Mistral AI platform
- name: Discord
url: https://discord.com/invite/mistralai)
about: Chat with the Mistral community
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# C extensions
*.so
# Distribution / packaging
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
# Unit test / coverage reports
htmlcov/
.tox/
.nox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
*.py,cover
.hypothesis/
.pytest_cache/
cover/
# Translations
*.mo
*.pot
# Django stuff:
*.log
local_settings.py
db.sqlite3
db.sqlite3-journal
# Flask stuff:
instance/
.webassets-cache
# Scrapy stuff:
.scrapy
# Sphinx documentation
docs/_build/
# PyBuilder
.pybuilder/
target/
# Jupyter Notebook
.ipynb_checkpoints
# IPython
profile_default/
ipython_config.py
# pyenv
# For a library or package, you might want to ignore these files since the code is
# intended to run in multiple environments; otherwise, check them in:
# .python-version
# pipenv
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
# However, in case of collaboration, if having platform-specific dependencies or dependencies
# having no cross-platform support, pipenv may install dependencies that don't work, or not
# install all needed dependencies.
#Pipfile.lock
# poetry
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
# This is especially recommended for binary packages to ensure reproducibility, and is more
# commonly ignored for libraries.
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
#poetry.lock
# pdm
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
#pdm.lock
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
# in version control.
# https://pdm.fming.dev/#use-with-ide
.pdm.toml
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
__pypackages__/
# Celery stuff
celerybeat-schedule
celerybeat.pid
# SageMath parsed files
*.sage.py
# Environments
.env
.venv
env/
venv/
ENV/
env.bak/
venv.bak/
# Spyder project settings
.spyderproject
.spyproject
# Rope project settings
.ropeproject
# mkdocs documentation
/site
# mypy
.mypy_cache/
.dmypy.json
dmypy.json
# Pyre type checker
.pyre/
# pytype static type analyzer
.pytype/
# Cython debug symbols
cython_debug/
# PyCharm
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
# and can be added to the global gitignore or merged into this file. For a more nuclear
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
#.idea/
================================================
FILE: LICENSE
================================================
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
================================================
FILE: README.md
================================================
# Mistral Inference
<a target="_blank" href="https://colab.research.google.com/github/mistralai/mistral-inference/blob/main/tutorials/getting_started.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
This repository contains minimal code to run Mistral models.
Blog 7B: [https://mistral.ai/news/announcing-mistral-7b/](https://mistral.ai/news/announcing-mistral-7b/)\
Blog 8x7B: [https://mistral.ai/news/mixtral-of-experts/](https://mistral.ai/news/mixtral-of-experts/)\
Blog 8x22B: [https://mistral.ai/news/mixtral-8x22b/](https://mistral.ai/news/mixtral-8x22b/)\
Blog Codestral 22B: [https://mistral.ai/news/codestral](https://mistral.ai/news/codestral/) \
Blog Codestral Mamba 7B: [https://mistral.ai/news/codestral-mamba/](https://mistral.ai/news/codestral-mamba/) \
Blog Mathstral 7B: [https://mistral.ai/news/mathstral/](https://mistral.ai/news/mathstral/) \
Blog Nemo: [https://mistral.ai/news/mistral-nemo/](https://mistral.ai/news/mistral-nemo/) \
Blog Mistral Large 2: [https://mistral.ai/news/mistral-large-2407/](https://mistral.ai/news/mistral-large-2407/) \
Blog Pixtral 12B: [https://mistral.ai/news/pixtral-12b/](https://mistral.ai/news/pixtral-12b/)
Blog Mistral Small 3.1: [https://mistral.ai/news/mistral-small-3-1/](https://mistral.ai/news/mistral-small-3-1/)
Discord: [https://discord.com/invite/mistralai](https://discord.com/invite/mistralai)\
Documentation: [https://docs.mistral.ai/](https://docs.mistral.ai/)\
Guardrailing: [https://docs.mistral.ai/usage/guardrailing](https://docs.mistral.ai/usage/guardrailing)
## Installation
Note: You will use a GPU to install `mistral-inference`, as it currently requires `xformers` to be installed and `xformers` itself needs a GPU for installation.
### PyPI
```
pip install mistral-inference
```
### Local
```
cd $HOME && git clone https://github.com/mistralai/mistral-inference
cd $HOME/mistral-inference && poetry install .
```
## Model download
### Direct links
| Name | Download | md5sum |
|-------------|-------|-------|
| 7B Instruct | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-Instruct-v0.3.tar | `80b71fcb6416085bcb4efad86dfb4d52` |
| 8x7B Instruct | https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar (**Updated model coming soon!**) | `8e2d3930145dc43d3084396f49d38a3f` |
| 8x22 Instruct | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-Instruct-v0.3.tar | `471a02a6902706a2f1e44a693813855b` |
| 7B Base | https://models.mistralcdn.com/mistral-7b-v0-3/mistral-7B-v0.3.tar | `0663b293810d7571dad25dae2f2a5806` |
| 8x7B | **Updated model coming soon!** | - |
| 8x22B | https://models.mistralcdn.com/mixtral-8x22b-v0-3/mixtral-8x22B-v0.3.tar | `a2fa75117174f87d1197e3a4eb50371a` |
| Codestral 22B | https://models.mistralcdn.com/codestral-22b-v0-1/codestral-22B-v0.1.tar | `1ea95d474a1d374b1d1b20a8e0159de3` |
| Mathstral 7B | https://models.mistralcdn.com/mathstral-7b-v0-1/mathstral-7B-v0.1.tar | `5f05443e94489c261462794b1016f10b` |
| Codestral-Mamba 7B | https://models.mistralcdn.com/codestral-mamba-7b-v0-1/codestral-mamba-7B-v0.1.tar | `d3993e4024d1395910c55db0d11db163` |
| Nemo Base | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-base-2407.tar | `c5d079ac4b55fc1ae35f51f0a3c0eb83` |
| Nemo Instruct | https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar | `296fbdf911cb88e6f0be74cd04827fe7` |
| Mistral Large 2 | https://models.mistralcdn.com/mistral-large-2407/mistral-large-instruct-2407.tar | `fc602155f9e39151fba81fcaab2fa7c4` |
Note:
- **Important**:
- `mixtral-8x22B-Instruct-v0.3.tar` is exactly the same as [Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1), only stored in `.safetensors` format
- `mixtral-8x22B-v0.3.tar` is the same as [Mixtral-8x22B-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-v0.1), but has an extended vocabulary of 32768 tokens.
- `codestral-22B-v0.1.tar` has a custom non-commercial license, called [Mistral AI Non-Production (MNPL) License](https://mistral.ai/licenses/MNPL-0.1.md)
- `mistral-large-instruct-2407.tar` has a custom non-commercial license, called [Mistral AI Research (MRL) License](https://mistral.ai/licenses/MRL-0.1.md)
- All of the listed models above support function calling. For example, Mistral 7B Base/Instruct v3 is a minor update to Mistral 7B Base/Instruct v2, with the addition of function calling capabilities.
- The "coming soon" models will include function calling as well.
- You can download the previous versions of our models from our [docs](https://docs.mistral.ai/getting-started/open_weight_models/#downloading).
### From Hugging Face Hub
| Name | ID | URL |
|-------------|-------|-------|
| Pixtral Large Instruct | mistralai/Pixtral-Large-Instruct-2411 | https://huggingface.co/mistralai/Pixtral-Large-Instruct-2411 |
| Pixtral 12B Base | mistralai/Pixtral-12B-Base-2409 | https://huggingface.co/mistralai/Pixtral-12B-Base-2409 |
| Pixtral 12B | mistralai/Pixtral-12B-2409 | https://huggingface.co/mistralai/Pixtral-12B-2409 |
| Mistral Small 3.1 24B Base | mistralai/Mistral-Small-3.1-24B-Base-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503
| Mistral Small 3.1 24B Instruct | mistralai/Mistral-Small-3.1-24B-Instruct-2503 | https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503 |
### Usage
**News!!!**: Mistral Large 2 is out. Read more about its capabilities [here](https://mistral.ai/news/mistral-large-2407/).
Create a local folder to store models
```sh
export MISTRAL_MODEL=$HOME/mistral_models
mkdir -p $MISTRAL_MODEL
```
Download any of the above links and extract the content, *e.g.*:
```sh
export 12B_DIR=$MISTRAL_MODEL/12B_Nemo
wget https://models.mistralcdn.com/mistral-nemo-2407/mistral-nemo-instruct-2407.tar
mkdir -p $12B_DIR
tar -xf mistral-nemo-instruct-2407.tar -C $12B_DIR
```
or
```sh
export M8x7B_DIR=$MISTRAL_MODEL/8x7b_instruct
wget https://models.mistralcdn.com/mixtral-8x7b-v0-1/Mixtral-8x7B-v0.1-Instruct.tar
mkdir -p $M8x7B_DIR
tar -xf Mixtral-8x7B-v0.1-Instruct.tar -C $M8x7B_DIR
```
For Hugging Face models' weights, here is an example to download [Mistral Small 3.1 24B Instruct](https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503):
```python
from pathlib import Path
from huggingface_hub import snapshot_download
mistral_models_path = Path.home().joinpath("mistral_models")
model_path = mistral_models_path / "mistral-small-3.1-instruct"
model_path.mkdir(parents=True, exist_ok=True)
repo_id = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"
snapshot_download(
repo_id=repo_id,
allow_patterns=["params.json", "consolidated.safetensors", "tekken.json"],
local_dir=model_path,
)
```
## Usage
The following sections give an overview of how to run the model from the Command-line interface (CLI) or directly within Python.
### CLI
- **Demo**
To test that a model works in your setup, you can run the `mistral-demo` command.
*E.g.* the 12B Mistral-Nemo model can be tested on a single GPU as follows:
```sh
mistral-demo $12B_DIR
```
Large models, such **8x7B** and **8x22B** have to be run in a multi-GPU setup.
For these models, you can use the following command:
```sh
torchrun --nproc-per-node 2 --no-python mistral-demo $M8x7B_DIR
```
*Note*: Change `--nproc-per-node` to more GPUs if available.
- **Chat**
To interactively chat with the models, you can make use of the `mistral-chat` command.
```sh
mistral-chat $12B_DIR --instruct --max_tokens 1024 --temperature 0.35
```
For large models, you can make use of `torchrun`.
```sh
torchrun --nproc-per-node 2 --no-python mistral-chat $M8x7B_DIR --instruct
```
*Note*: Change `--nproc-per-node` to more GPUs if necessary (*e.g.* for 8x22B).
- **Chat with Codestral**
To use [Codestral](https://mistral.ai/news/codestral/) as a coding assistant you can run the following command using `mistral-chat`.
Make sure `$M22B_CODESTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/Codestral-22B-v0.1`
```sh
mistral-chat $M22B_CODESTRAL --instruct --max_tokens 256
```
If you prompt it with *"Write me a function that computes fibonacci in Rust"*, the model should generate something along the following lines:
```sh
Sure, here's a simple implementation of a function that computes the Fibonacci sequence in Rust. This function takes an integer `n` as an argument and returns the `n`th Fibonacci number.
fn fibonacci(n: u32) -> u32 {
match n {
0 => 0,
1 => 1,
_ => fibonacci(n - 1) + fibonacci(n - 2),
}
}
fn main() {
let n = 10;
println!("The {}th Fibonacci number is: {}", n, fibonacci(n));
}
This function uses recursion to calculate the Fibonacci number. However, it's not the most efficient solution because it performs a lot of redundant calculations. A more efficient solution would use a loop to iteratively calculate the Fibonacci numbers.
```
You can continue chatting afterwards, *e.g.* with *"Translate it to Python"*.
- **Chat with Codestral-Mamba**
To use [Codestral-Mamba](https://mistral.ai/news/codestral-mamba/) as a coding assistant you can run the following command using `mistral-chat`.
Make sure `$7B_CODESTRAL_MAMBA` is set to a valid path to the downloaded codestral-mamba folder, e.g. `$HOME/mistral_models/mamba-codestral-7B-v0.1`.
You then need to additionally install the following packages:
```
pip install packaging mamba-ssm causal-conv1d transformers
```
before you can start chatting:
```sh
mistral-chat $7B_CODESTRAL_MAMBA --instruct --max_tokens 256
```
- **Chat with Mathstral**
To use [Mathstral](https://mistral.ai/news/mathstral/) as an assistant you can run the following command using `mistral-chat`.
Make sure `$7B_MATHSTRAL` is set to a valid path to the downloaded codestral folder, e.g. `$HOME/mistral_models/mathstral-7B-v0.1`
```sh
mistral-chat $7B_MATHSTRAL --instruct --max_tokens 256
```
If you prompt it with *"Albert likes to surf every week. Each surfing session lasts for 4 hours and costs $20 per hour. How much would Albert spend in 5 weeks?"*, the model should answer with the correct calculation.
You can then continue chatting afterwards, *e.g.* with *"How much would he spend in a year?"*.
- **Chat with Mistral Small 3.1 24B Instruct**
To use [Mistral Small 3.1 24B Instruct](https://mistral.ai/news/mistral-small-3-1/) as an assistant you can run the following command using `mistral-chat`.
Make sure `$MISTRAL_SMALL_3_1_INSTRUCT` is set to a valid path to the downloaded mistral small folder, e.g. `$HOME/mistral_models/mistral-small-3.1-instruct`
```sh
mistral-chat $MISTRAL_SMALL_3_1_INSTRUCT --instruct --max_tokens 256
```
If you prompt it with *"The above image presents an image of which park ? Please give the hints to identify the park."* with the following image URL *https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png*, the model should answer with the Yosemite park and give hints to identify it.
You can then continue chatting afterwards, *e.g.* with *"What is the name of the lake in the image?"*. The model should respond that it is not a lake but a river.
### Python
- *Instruction Following*:
```py
from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.protocol.instruct.messages import UserMessage
from mistral_common.protocol.instruct.request import ChatCompletionRequest
tokenizer = MistralTokenizer.from_file("./mistral-nemo-instruct-v0.1/tekken.json") # change to extracted tokenizer file
model = Transformer.from_folder("./mistral-nemo-instruct-v0.1") # change to extracted model dir
prompt = "How expensive would it be to ask a window cleaner to clean all windows in Paris. Make a reasonable guess in US Dollar."
completion_request = ChatCompletionRequest(messages=[UserMessage(content=prompt)])
tokens = tokenizer.encode_chat_completion(completion_request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=1024, temperature=0.35, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
print(result)
```
- *Multimodal Instruction Following*:
```python
from pathlib import Path
from huggingface_hub import snapshot_download
from mistral_common.protocol.instruct.messages import ImageURLChunk, TextChunk
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_inference.generate import generate
from mistral_inference.transformer import Transformer
model_path = Path.home().joinpath("mistral_models") / "mistral-small-3.1-instruct" # change to extracted model
tokenizer = MistralTokenizer.from_file(model_path / "tekken.json")
model = Transformer.from_folder(model_path)
url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/yosemite.png"
prompt = "The above image presents an image of which park ? Please give the hints to identify the park."
user_content = [ImageURLChunk(image_url=url), TextChunk(text=prompt)]
tokens, images = tokenizer.instruct_tokenizer.encode_user_content(user_content, False)
out_tokens, _ = generate(
[tokens],
model,
images=[images],
max_tokens=256,
temperature=0.15,
eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id,
)
result = tokenizer.decode(out_tokens[0])
print("Prompt:", prompt)
print("Completion:", result)
```
- *Function Calling*:
```py
from mistral_common.protocol.instruct.tool_calls import Function, Tool
completion_request = ChatCompletionRequest(
tools=[
Tool(
function=Function(
name="get_current_weather",
description="Get the current weather",
parameters={
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA",
},
"format": {
"type": "string",
"enum": ["celsius", "fahrenheit"],
"description": "The temperature unit to use. Infer this from the users location.",
},
},
"required": ["location", "format"],
},
)
)
],
messages=[
UserMessage(content="What's the weather like today in Paris?"),
],
)
tokens = tokenizer.encode_chat_completion(completion_request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=64, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.instruct_tokenizer.tokenizer.decode(out_tokens[0])
print(result)
```
- *Fill-in-the-middle (FIM)*:
Make sure to have `mistral-common >= 1.2.0` installed:
```
pip install --upgrade mistral-common
```
You can simulate a code completion in-filling as follows.
```py
from mistral_inference.transformer import Transformer
from mistral_inference.generate import generate
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.tokens.instruct.request import FIMRequest
tokenizer = MistralTokenizer.from_model("codestral-22b")
model = Transformer.from_folder("./mistral_22b_codestral")
prefix = """def add("""
suffix = """ return sum"""
request = FIMRequest(prompt=prefix, suffix=suffix)
tokens = tokenizer.encode_fim(request).tokens
out_tokens, _ = generate([tokens], model, max_tokens=256, temperature=0.0, eos_id=tokenizer.instruct_tokenizer.tokenizer.eos_id)
result = tokenizer.decode(out_tokens[0])
middle = result.split(suffix)[0].strip()
print(middle)
```
### Test
To run logits equivalence:
```
python -m pytest tests
```
## Deployment
The `deploy` folder contains code to build a [vLLM](https://M7B_DIR.com/vllm-project/vllm) image with the required dependencies to serve the Mistral AI model. In the image, the [transformers](https://github.com/huggingface/transformers/) library is used instead of the reference implementation. To build it:
```bash
docker build deploy --build-arg MAX_JOBS=8
```
Instructions to run the image can be found in the [official documentation](https://docs.mistral.ai/quickstart).
## Model platforms
- Use Mistral models on [Mistral AI official API](https://console.mistral.ai/) (La Plateforme)
- Use Mistral models via [cloud providers](https://docs.mistral.ai/deployment/cloud/overview/)
## References
[1]: [LoRA](https://arxiv.org/abs/2106.09685): Low-Rank Adaptation of Large Language Models, Hu et al. 2021
## License
This library is licensed under the Apache 2.0 License. See the [LICENCE](./LICENCE) file for more information.
*You must not use this library or our models in a manner that infringes, misappropriates, or otherwise violates any third party’s rights, including intellectual property rights.*
================================================
FILE: deploy/.dockerignore
================================================
*
!entrypoint.sh
================================================
FILE: deploy/Dockerfile
================================================
FROM --platform=amd64 nvcr.io/nvidia/cuda:12.1.0-devel-ubuntu22.04 as base
WORKDIR /workspace
RUN apt update && \
apt install -y python3-pip python3-packaging \
git ninja-build && \
pip3 install -U pip
# Tweak this list to reduce build time
# https://developer.nvidia.com/cuda-gpus
ENV TORCH_CUDA_ARCH_LIST "7.0;7.2;7.5;8.0;8.6;8.9;9.0"
RUN pip3 install "torch==2.1.1"
# This build is slow but NVIDIA does not provide binaries. Increase MAX_JOBS as needed.
RUN pip3 install "git+https://github.com/stanford-futuredata/megablocks.git"
RUN pip3 install "git+https://github.com/vllm-project/vllm.git"
RUN pip3 install "xformers==0.0.23" "transformers==4.36.0" "fschat[model_worker]==0.2.34"
RUN git clone https://github.com/NVIDIA/apex && \
cd apex && git checkout 2386a912164b0c5cfcd8be7a2b890fbac5607c82 && \
sed -i '/check_cuda_torch_binary_vs_bare_metal(CUDA_HOME)/d' setup.py && \
python3 setup.py install --cpp_ext --cuda_ext
COPY entrypoint.sh .
RUN chmod +x /workspace/entrypoint.sh
ENTRYPOINT ["/workspace/entrypoint.sh"]
================================================
FILE: deploy/entrypoint.sh
================================================
#!/bin/bash
if [[ ! -z "${HF_TOKEN}" ]]; then
echo "The HF_TOKEN environment variable is set, logging to Hugging Face."
python3 -c "import huggingface_hub; huggingface_hub.login('${HF_TOKEN}')"
else
echo "The HF_TOKEN environment variable is not set or empty, not logging to Hugging Face."
fi
# Run the provided command
exec python3 -u -m vllm.entrypoints.openai.api_server "$@"
================================================
FILE: pyproject.toml
================================================
[tool.poetry]
name = "mistral_inference"
version = "1.6.0"
description = ""
authors = ["bam4d <bam4d@mistral.ai>"]
readme = "README.md"
packages = [{ include = "mistral_inference", from = "src" }]
[tool.ruff]
lint.select = ["E", "F", "W", "Q", "I"]
lint.ignore = ["E203"]
lint.fixable = ["ALL"]
lint.unfixable = []
line-length = 120
exclude = ["docs", "build", "tutorials"]
[tool.mypy]
disallow_untyped_defs = true
show_error_codes = true
no_implicit_optional = true
warn_return_any = true
warn_unused_ignores = true
exclude = ["docs", "tools", "build"]
[tool.poetry.dependencies]
python = "^3.9.10"
xformers = ">=0.0.24"
simple-parsing = ">=0.1.5"
fire = ">=0.6.0"
mistral_common = ">=1.5.4"
safetensors = ">=0.4.0"
pillow = ">=10.3.0"
[tool.poetry.group.dev.dependencies]
types-protobuf = "4.24.0.20240129"
mypy-protobuf = "^3.5.0"
pytest = "7.4.4"
ruff = "^0.2.2"
mypy = "^1.8.0"
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
[tool.pytest.ini_options]
testpaths = ["./tests"]
[tool.poetry.scripts]
mistral-chat = "mistral_inference.main:mistral_chat"
mistral-demo = "mistral_inference.main:mistral_demo"
================================================
FILE: src/mistral_inference/__init__.py
================================================
__version__ = "1.6.0"
================================================
FILE: src/mistral_inference/args.py
================================================
from dataclasses import dataclass
from typing import List, Optional
from simple_parsing.helpers import Serializable
from mistral_inference.lora import LoraArgs
from mistral_inference.moe import MoeArgs
PATCH_MERGE = "patch_merge"
@dataclass
class VisionEncoderArgs:
hidden_size: int
num_channels: int
image_size: int
patch_size: int
intermediate_size: int
num_hidden_layers: int
num_attention_heads: int
rope_theta: float = 1e4 # for rope-2D
image_token_id: int = 10
adapter_bias: bool = True
spatial_merge_size: int = 1
add_pre_mm_projector_layer_norm: bool = False
mm_projector_id: str = ""
@dataclass
class TransformerArgs(Serializable):
dim: int
n_layers: int
head_dim: int
hidden_dim: int
n_heads: int
n_kv_heads: int
norm_eps: float
vocab_size: int
max_batch_size: int = 0
# For rotary embeddings. If not set, will be inferred
rope_theta: Optional[float] = None
# If this is set, we will use MoE layers instead of dense layers.
moe: Optional[MoeArgs] = None
# If this is set, we will load LoRA linear layers instead of linear layers.
lora: Optional[LoraArgs] = None
sliding_window: Optional[int] | Optional[List[int]] = None
_sliding_window: Optional[int] | Optional[List[int]] = None
model_type: str = "transformer"
vision_encoder: Optional[VisionEncoderArgs] = None
def __post_init__(self) -> None:
assert self.model_type == "transformer", self.model_type
assert self.sliding_window is None or self._sliding_window is None
# hack for now so that vLLM is supported correctly
self.sliding_window = self.sliding_window if self.sliding_window is not None else self._sliding_window
@dataclass
class MambaArgs(Serializable):
dim: int
n_layers: int
vocab_size: int
n_groups: int
rms_norm: bool
residual_in_fp32: bool
fused_add_norm: bool
pad_vocab_size_multiple: int
tie_embeddings: bool
model_type: str = "mamba"
def __post_init__(self) -> None:
assert self.model_type == "mamba", self.model_type
================================================
FILE: src/mistral_inference/cache.py
================================================
from dataclasses import dataclass
from typing import List, Optional, Tuple
import torch
from xformers.ops.fmha.attn_bias import ( # type: ignore
AttentionBias,
BlockDiagonalCausalMask,
BlockDiagonalCausalWithOffsetPaddedKeysMask,
BlockDiagonalMask,
)
def get_cache_sizes(n_layers: int, max_seq_len: int, sliding_window: Optional[int] | Optional[List[int]]) -> List[int]:
if sliding_window is None:
return n_layers * [max_seq_len]
elif isinstance(sliding_window, int):
return n_layers * [sliding_window]
else:
assert isinstance(sliding_window, list), f"Expected list, got {type(sliding_window)}"
assert (
n_layers % len(sliding_window) == 0
), f"Expected n_layers % len(sliding_window) == 0, got {n_layers} % {len(sliding_window)}"
num_repeats = n_layers // len(sliding_window)
return num_repeats * [w if w is not None else max_seq_len for w in sliding_window]
@dataclass
class CacheInputMetadata:
# # rope absolute positions
# positions: torch.Tensor
# # where tokens should go in the cache
# cache_positions: torch.Tensor
# # if prefill, use block diagonal causal mask
# # else use causal with padded key mask
# prefill: bool
# mask: AttentionBias
# seqlens: List[int]
# rope absolute positions
positions: torch.Tensor
# which elements in the sequences need to be cached
to_cache_mask: torch.Tensor
# how many elements are cached per sequence
cached_elements: torch.Tensor
# where tokens should go in the cache
cache_positions: torch.Tensor
# if prefill, use block diagonal causal mask
# else use causal with padded key mask
prefill: bool
mask: AttentionBias
seqlens: List[int]
def interleave_list(l1: List[torch.Tensor], l2: List[torch.Tensor]) -> List[torch.Tensor]:
assert len(l1) == len(l2)
return [v for pair in zip(l1, l2) for v in pair]
def unrotate(cache: torch.Tensor, seqlen: int) -> torch.Tensor:
assert cache.ndim == 3 # (W, H, D)
position = seqlen % cache.shape[0]
if seqlen < cache.shape[0]:
return cache[:seqlen]
elif position == 0:
return cache
else:
return torch.cat([cache[position:], cache[:position]], dim=0)
class CacheView:
def __init__(
self,
cache_k: torch.Tensor,
cache_v: torch.Tensor,
metadata: CacheInputMetadata,
kv_seqlens: torch.Tensor,
):
self.cache_k = cache_k
self.cache_v = cache_v
self.kv_seqlens = kv_seqlens
self.metadata = metadata
def update(self, xk: torch.Tensor, xv: torch.Tensor) -> None:
"""
to_cache_mask masks the last [max_seq_len] tokens in each sequence
"""
n_kv_heads, head_dim = self.cache_k.shape[-2:]
flat_cache_k = self.cache_k.view(-1, n_kv_heads, head_dim)
flat_cache_v = self.cache_v.view(-1, n_kv_heads, head_dim)
flat_cache_k.index_copy_(0, self.metadata.cache_positions, xk[self.metadata.to_cache_mask])
flat_cache_v.index_copy_(0, self.metadata.cache_positions, xv[self.metadata.to_cache_mask])
def interleave_kv(self, xk: torch.Tensor, xv: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
"""
This is a naive implementation and not optimized for speed.
"""
assert xk.ndim == xv.ndim == 3 # (B * T, H, D)
assert xk.shape == xv.shape
if all([s == 0 for s in self.metadata.seqlens]):
# No cache to interleave
return xk, xv
# Make it a list of [(T, H, D)]
xk: Tuple[torch.Tensor] = torch.split(xk, self.metadata.seqlens) # type: ignore
xv: Tuple[torch.Tensor] = torch.split(xv, self.metadata.seqlens) # type: ignore
assert len(xk) == len(self.kv_seqlens), f"Batch size is {len(self.kv_seqlens)}, got {len(xk)}"
# Order elements in cache by position by unrotating
cache_k = [unrotate(t, s) for t, s in zip(self.cache_k, self.kv_seqlens)]
cache_v = [unrotate(t, s) for t, s in zip(self.cache_v, self.kv_seqlens)]
interleaved_k = interleave_list(cache_k, list(xk))
interleaved_v = interleave_list(cache_v, list(xv))
return torch.cat(interleaved_k, dim=0), torch.cat(interleaved_v, dim=0)
@property
def max_seq_len(self) -> int:
return self.cache_k.shape[1]
@property
def key(self) -> torch.Tensor:
return self.cache_k[: len(self.kv_seqlens)]
@property
def value(self) -> torch.Tensor:
return self.cache_v[: len(self.kv_seqlens)]
@property
def prefill(self) -> bool:
return self.metadata.prefill
@property
def mask(self) -> AttentionBias:
return self.metadata.mask
class BufferCache:
"""
This is an example that implements a buffer cache, allowing for variable length sequences.
Allocated cache is rectangular which is wasteful (see PagedAttention for better mechanisms)
"""
def __init__(
self,
n_layers: int,
max_batch_size: int,
max_seq_len: int,
n_kv_heads: int,
head_dim: int,
sliding_window: Optional[int] | Optional[List[int]] = None,
):
self.max_seq_len = max_seq_len
self.n_kv_heads = n_kv_heads
self.head_dim = head_dim
self.n_layers = n_layers
self.cache_sizes: List[int] = get_cache_sizes(n_layers, max_seq_len, sliding_window)
assert len(self.cache_sizes) == n_layers, f"Expected {n_layers} cache sizes, got {len(self.cache_sizes)}"
self.cache_k = {}
self.cache_v = {}
for i, cache_size in enumerate(self.cache_sizes):
self.cache_k[i] = torch.empty((max_batch_size, cache_size, n_kv_heads, head_dim))
self.cache_v[i] = torch.empty((max_batch_size, cache_size, n_kv_heads, head_dim))
# holds the valid length for each batch element in the cache
self.kv_seqlens: Optional[torch.Tensor] = None
def get_view(self, layer_id: int, metadata: CacheInputMetadata) -> CacheView:
assert self.kv_seqlens is not None
return CacheView(self.cache_k[layer_id], self.cache_v[layer_id], metadata, self.kv_seqlens)
def reset(self) -> None:
self.kv_seqlens = None
def init_kvseqlens(self, batch_size: int) -> None:
self.kv_seqlens = torch.zeros((batch_size,), device=self.device, dtype=torch.long)
@property
def device(self) -> torch.device:
return self.cache_k[0].device
def to(self, device: torch.device, dtype: torch.dtype) -> "BufferCache":
for i in range(self.n_layers):
self.cache_k[i] = self.cache_k[i].to(device=device, dtype=dtype)
self.cache_v[i] = self.cache_v[i].to(device=device, dtype=dtype)
return self
def update_seqlens(self, seqlens: List[int]) -> None:
assert self.kv_seqlens is not None
self.kv_seqlens += torch.tensor(seqlens, device=self.device, dtype=torch.long)
def get_input_metadata(self, seqlens: List[int]) -> List[CacheInputMetadata]:
"""
input = seqlens [5,7,2] // seqpos [0, 1, 3] // sliding_window 3
--> only cache last 3 tokens in each sequence
- to_cache_mask = [0 0 1 1 1 | 0 0 0 0 1 1 1 | 1 1]
- cached_elements = [3 | 3 | 2]
--> absolute positions are used for rope
- positions = [0 1 2 3 4 | 1 2 3 4 5 6 7 | 3 4]
--> cache positions are positions cache_masked, modulo sliding_window + batch_idx * sliding_window
- cache_positions = [2 0 1 | 5 3 4 | 6 7]
"""
metadata: List[CacheInputMetadata] = []
if self.kv_seqlens is None:
self.init_kvseqlens(len(seqlens))
assert self.kv_seqlens is not None
assert len(seqlens) == len(
self.kv_seqlens
), f"Batch size is {len(self.kv_seqlens)}, got {len(seqlens)}, did you forget to reset cache?"
seqpos = self.kv_seqlens.tolist()
assert len(seqlens) > 0, seqlens
for cache_size in self.cache_sizes:
metadata.append(self._get_input_metadata_layer(cache_size, seqlens, seqpos))
return metadata
def _get_input_metadata_layer(self, cache_size: int, seqlens: List[int], seqpos: List[int]) -> CacheInputMetadata:
masks = [[x >= seqlen - cache_size for x in range(seqlen)] for seqlen in seqlens]
to_cache_mask = torch.tensor(sum(masks, []), device=self.device, dtype=torch.bool)
cached_elements = torch.tensor([sum(mask) for mask in masks], device=self.device, dtype=torch.long)
positions = torch.cat([torch.arange(pos, pos + seqlen) for pos, seqlen in zip(seqpos, seqlens)]).to(
device=self.device, dtype=torch.long
)
batch_idx = torch.tensor(
sum([[i] * seqlen for i, seqlen in enumerate(seqlens)], []), device=self.device, dtype=torch.long
)
cache_positions = positions % cache_size + batch_idx * cache_size
first_prefill = seqpos[0] == 0
subsequent_prefill = any(seqlen > 1 for seqlen in seqlens)
if first_prefill:
assert all([pos == 0 for pos in seqpos]), seqpos
mask = BlockDiagonalCausalMask.from_seqlens(seqlens).make_local_attention(cache_size)
elif subsequent_prefill:
assert self.kv_seqlens is not None
mask = BlockDiagonalMask.from_seqlens(
q_seqlen=seqlens,
kv_seqlen=[
s + cached_s.clamp(max=cache_size).item() for (s, cached_s) in zip(seqlens, self.kv_seqlens)
],
).make_local_attention_from_bottomright(cache_size)
else:
mask = BlockDiagonalCausalWithOffsetPaddedKeysMask.from_seqlens(
q_seqlen=seqlens,
kv_padding=cache_size,
kv_seqlen=(self.kv_seqlens + cached_elements).clamp(max=cache_size).tolist(),
)
return CacheInputMetadata(
positions=positions,
to_cache_mask=to_cache_mask,
cached_elements=cached_elements,
cache_positions=cache_positions[to_cache_mask],
prefill=first_prefill or subsequent_prefill,
mask=mask,
seqlens=seqlens,
)
================================================
FILE: src/mistral_inference/generate.py
================================================
from typing import List, Optional, Tuple
import numpy as np
import torch
from mistral_inference.cache import BufferCache
from mistral_inference.mamba import Mamba
from mistral_inference.transformer import Transformer
@torch.inference_mode()
def generate_mamba(
encoded_prompts: List[List[int]],
model: Mamba,
*,
max_tokens: int,
temperature: float,
chunk_size: Optional[int] = None,
eos_id: Optional[int] = None,
) -> Tuple[List[List[int]], List[List[float]]]:
input_ids = torch.tensor(encoded_prompts, device=model.device)
output = model.model.generate(
input_ids=input_ids,
max_length=input_ids.shape[-1] + max_tokens,
cg=True,
return_dict_in_generate=True,
output_scores=True,
enable_timing=False,
eos_token_id=eos_id,
temperature=temperature,
top_p=0.8,
)
generated_tokens = output.sequences[:, input_ids.shape[-1] :].tolist()
_logprobs: List[List[float]] = [[] for _ in range(len(generated_tokens))]
for seq_idx, batch_score in enumerate(output.scores):
for batch_idx, score in enumerate(batch_score.tolist()):
_logprobs[batch_idx].append(score[generated_tokens[batch_idx][seq_idx]])
return generated_tokens, _logprobs
@torch.inference_mode()
def generate(
encoded_prompts: List[List[int]],
model: Transformer,
images: List[List[np.ndarray]] = [],
*,
max_tokens: int,
temperature: float,
chunk_size: Optional[int] = None,
eos_id: Optional[int] = None,
) -> Tuple[List[List[int]], List[List[float]]]:
images_torch: List[List[torch.Tensor]] = []
if images:
assert chunk_size is None
images_torch = [
[torch.tensor(im, device=model.device, dtype=model.dtype) for im in images_for_sample]
for images_for_sample in images
]
model = model.eval()
B, V = len(encoded_prompts), model.args.vocab_size
seqlens = [len(x) for x in encoded_prompts]
# Cache
cache_window = max(seqlens) + max_tokens
cache = BufferCache(
model.n_local_layers,
model.args.max_batch_size,
cache_window,
model.args.n_kv_heads,
model.args.head_dim,
model.args.sliding_window,
)
cache.to(device=model.device, dtype=model.dtype)
cache.reset()
# Bookkeeping
logprobs: List[List[float]] = [[] for _ in range(B)]
last_token_prelogits = None
# One chunk if size not specified
max_prompt_len = max(seqlens)
if chunk_size is None:
chunk_size = max_prompt_len
flattened_images: List[torch.Tensor] = sum(images_torch, [])
# Encode prompt by chunks
for s in range(0, max_prompt_len, chunk_size):
prompt_chunks = [p[s : s + chunk_size] for p in encoded_prompts]
assert all(len(p) > 0 for p in prompt_chunks)
prelogits = model.forward(
torch.tensor(sum(prompt_chunks, []), device=model.device, dtype=torch.long),
images=flattened_images,
seqlens=[len(p) for p in prompt_chunks],
cache=cache,
)
logits = torch.log_softmax(prelogits, dim=-1)
if last_token_prelogits is not None:
# Pass > 1
last_token_logits = torch.log_softmax(last_token_prelogits, dim=-1)
for i_seq in range(B):
logprobs[i_seq].append(last_token_logits[i_seq, prompt_chunks[i_seq][0]].item())
offset = 0
for i_seq, sequence in enumerate(prompt_chunks):
logprobs[i_seq].extend([logits[offset + i, sequence[i + 1]].item() for i in range(len(sequence) - 1)])
offset += len(sequence)
last_token_prelogits = prelogits.index_select(
0,
torch.tensor([len(p) for p in prompt_chunks], device=prelogits.device).cumsum(dim=0) - 1,
)
assert last_token_prelogits.shape == (B, V)
# decode
generated_tensors = []
is_finished = torch.tensor([False for _ in range(B)])
assert last_token_prelogits is not None
for _ in range(max_tokens):
next_token = sample(last_token_prelogits, temperature=temperature, top_p=0.8)
if eos_id is not None:
is_finished = is_finished | (next_token == eos_id).cpu()
if is_finished.all():
break
last_token_logits = torch.log_softmax(last_token_prelogits, dim=-1)
for i in range(B):
logprobs[i].append(last_token_logits[i, next_token[i]].item())
generated_tensors.append(next_token[:, None])
last_token_prelogits = model.forward(next_token, seqlens=[1] * B, cache=cache)
assert last_token_prelogits.shape == (B, V)
generated_tokens: List[List[int]]
if generated_tensors:
generated_tokens = torch.cat(generated_tensors, 1).tolist()
else:
generated_tokens = []
return generated_tokens, logprobs
def sample(logits: torch.Tensor, temperature: float, top_p: float) -> torch.Tensor:
if temperature > 0:
probs = torch.softmax(logits / temperature, dim=-1)
next_token = sample_top_p(probs, top_p)
else:
next_token = torch.argmax(logits, dim=-1).unsqueeze(0)
return next_token.reshape(-1)
def sample_top_p(probs: torch.Tensor, p: float) -> torch.Tensor:
assert 0 <= p <= 1
probs_sort, probs_idx = torch.sort(probs, dim=-1, descending=True)
probs_sum = torch.cumsum(probs_sort, dim=-1)
mask = probs_sum - probs_sort > p
probs_sort[mask] = 0.0
probs_sort.div_(probs_sort.sum(dim=-1, keepdim=True))
next_token = torch.multinomial(probs_sort, num_samples=1)
return torch.gather(probs_idx, -1, next_token)
================================================
FILE: src/mistral_inference/lora.py
================================================
import logging
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, NamedTuple, Union
import safetensors.torch
import torch
import torch.nn as nn
from simple_parsing.helpers import Serializable
@dataclass
class LoraArgs(Serializable):
rank: int
scaling: float
def __post_init__(self) -> None:
assert self.rank > 0
assert self.scaling > 0.0
class LoRALinear(nn.Module):
"""
Implementation of:
- LoRA: https://arxiv.org/abs/2106.09685
Notes:
- Freezing is handled at network level, not layer level.
- Scaling factor controls relative importance of LoRA skip
connection versus original frozen weight. General guidance is
to keep it to 2.0 and sweep over learning rate when changing
the rank.
"""
def __init__(
self,
in_features: int,
out_features: int,
rank: int,
scaling: float,
bias: bool = False,
):
super().__init__()
self.in_features = in_features
self.out_features = out_features
assert not bias
self.bias = bias
self.rank = rank
self.scaling = scaling
self.lora_A = nn.Linear(
self.in_features,
self.rank,
bias=self.bias,
)
self.lora_B = nn.Linear(
self.rank,
self.out_features,
bias=self.bias,
)
self.linear = nn.Linear(self.in_features, self.out_features, bias=self.bias)
# make sure no LoRA weights are marked as "missing" in load_state_dict
def ignore_missing_keys(m: nn.Module, incompatible_keys: NamedTuple) -> None:
incompatible_keys.missing_keys[:] = [] # type: ignore
self.register_load_state_dict_post_hook(ignore_missing_keys)
def forward(self, x: torch.Tensor) -> torch.Tensor:
lora = self.lora_B(self.lora_A(x))
result: torch.Tensor = self.linear(x) + lora * self.scaling
return result
def _load_from_state_dict(self, state_dict: Dict[str, Any], prefix: str, *args, **kwargs) -> None: # type: ignore[no-untyped-def]
key_name = prefix + "weight"
# full checkpoint
if key_name in state_dict:
w_ref = state_dict[key_name]
# load frozen weights
state_dict = {
"linear.weight": w_ref,
"lora_A.weight": torch.zeros_like(self.lora_A.weight, device=w_ref.device, dtype=w_ref.dtype),
"lora_B.weight": torch.zeros_like(self.lora_B.weight, device=w_ref.device, dtype=w_ref.dtype),
}
self.load_state_dict(state_dict, assign=True, strict=True)
class LoRALoaderMixin:
def load_lora(self, lora_path: Union[Path, str], scaling: float = 2.0) -> None:
"""Loads LoRA checkpoint"""
lora_path = Path(lora_path)
assert lora_path.is_file(), f"{lora_path} does not exist or is not a file"
state_dict = safetensors.torch.load_file(lora_path)
self._load_lora_state_dict(state_dict, scaling=scaling)
def _load_lora_state_dict(self, lora_state_dict: Dict[str, torch.Tensor], scaling: float = 2.0) -> None:
"""Loads LoRA state_dict"""
lora_dtypes = set([p.dtype for p in lora_state_dict.values()])
assert (
len(lora_dtypes) == 1
), f"LoRA weights have multiple different dtypes {lora_dtypes}. All weights need to have the same dtype"
lora_dtype = lora_dtypes.pop()
assert lora_dtype == self.dtype, f"LoRA weights dtype differs from model's dtype {lora_dtype} != {self.dtype}" # type: ignore[attr-defined]
assert all("lora" in key for key in lora_state_dict.keys())
# move tensors to device
lora_state_dict = {k: v.to(self.device) for k, v in lora_state_dict.items()} # type: ignore[attr-defined]
state_dict = self.state_dict() # type: ignore[attr-defined]
if self.args.lora is None: # type: ignore[attr-defined]
logging.info("Loading and merging LoRA weights...")
# replace every nn.Linear with a LoRALinear with 'meta' device except the output layer
named_modules = dict(self.named_modules()) # type: ignore[attr-defined]
for name, module in named_modules.items():
if isinstance(module, nn.Linear) and name != "output":
layer_id = name.split(".")[1]
if layer_id not in self.layers: # type: ignore[attr-defined]
logging.debug(
"Skipping parameter %s at pipeline rank %d",
name,
self.pipeline_rank, # type: ignore[attr-defined]
)
elif (name + ".lora_B.weight") in lora_state_dict:
weight = (
module.weight
+ (lora_state_dict[name + ".lora_B.weight"] @ lora_state_dict[name + ".lora_A.weight"])
* scaling
)
state_dict[name + ".weight"] = weight
else:
logging.info("Loading LoRA weights...")
for k, v in lora_state_dict.items():
state_dict.update(lora_state_dict)
layer_id = k.split(".")[1]
if layer_id in self.layers: # type: ignore[attr-defined]
state_dict[k] = v
else:
logging.debug(
"Skipping parameter %s at pipeline rank %d",
k,
self.pipeline_rank, # type: ignore[attr-defined]
)
self.load_state_dict(state_dict, strict=True) # type: ignore[attr-defined]
================================================
FILE: src/mistral_inference/main.py
================================================
import json
import logging
import os
import warnings
from pathlib import Path
from typing import List, Optional, Tuple, Type, Union
import fire # type: ignore
import torch
import torch.distributed as dist
from mistral_common.protocol.instruct.messages import (
AssistantMessage,
ContentChunk,
ImageChunk,
ImageURLChunk,
TextChunk,
UserMessage,
)
from mistral_common.protocol.instruct.request import ChatCompletionRequest
from mistral_common.tokens.tokenizers.base import Tokenizer
from mistral_common.tokens.tokenizers.mistral import MistralTokenizer
from mistral_common.tokens.tokenizers.sentencepiece import is_sentencepiece
from mistral_common.tokens.tokenizers.tekken import (
SpecialTokenPolicy,
Tekkenizer,
is_tekken,
)
from PIL import Image
from mistral_inference.args import TransformerArgs
from mistral_inference.generate import generate, generate_mamba
from mistral_inference.mamba import Mamba
from mistral_inference.transformer import Transformer
def is_torchrun() -> bool:
required_vars = ["MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE"]
return all(var in os.environ for var in required_vars)
def load_tokenizer(model_path: Path) -> MistralTokenizer:
tokenizer = [f for f in os.listdir(model_path) if is_tekken(model_path / f) or is_sentencepiece(model_path / f)]
assert len(tokenizer) > 0, (
f"No tokenizer in {model_path}, place a `tokenizer.model.[v1,v2,v3]` or `tekken.json` file in {model_path}."
)
assert len(tokenizer) == 1, (
f"Multiple tokenizers {', '.join(tokenizer)} found in `model_path`, make sure to only have one tokenizer"
)
mistral_tokenizer = MistralTokenizer.from_file(str(model_path / tokenizer[0]))
if isinstance(mistral_tokenizer.instruct_tokenizer.tokenizer, Tekkenizer):
mistral_tokenizer.instruct_tokenizer.tokenizer.special_token_policy = SpecialTokenPolicy.KEEP
logging.info(f"Loaded tokenizer of type {mistral_tokenizer.instruct_tokenizer.__class__}")
return mistral_tokenizer
def get_model_cls(model_path: str) -> Union[Type[Mamba], Type[Transformer]]:
with open(Path(model_path) / "params.json", "r") as f:
args_dict = json.load(f)
return {"mamba": Mamba, "transformer": Transformer}[args_dict.get("model_type", "transformer")] # type: ignore[return-value]
def pad_and_convert_to_tensor(list_of_lists: List[List[int]], pad_id: int) -> List[List[int]]:
# Determine the length of the longest list
max_len = max(len(lst) for lst in list_of_lists)
# Left pad each list to the maximum length
padded_lists = [[pad_id] * (max_len - len(lst)) + lst for lst in list_of_lists]
return padded_lists
def _get_multimodal_input() -> Tuple[UserMessage, bool]:
chunks: List[ContentChunk] = []
response = input("Text prompt: ")
if response:
chunks.append(TextChunk(text=response))
print("[You can input zero, one or more images now.]")
while True:
did_something = False
response = input("Image path or url [Leave empty and press enter to finish image input]: ")
if response:
if Path(response).is_file():
chunks.append(ImageChunk(image=Image.open(response)))
else:
assert response.startswith("http"), f"{response} does not seem to be a valid url."
chunks.append(ImageURLChunk(image_url=response))
did_something = True
if not did_something:
break
return UserMessage(content=chunks), not chunks
def interactive(
model_path: str,
max_tokens: int = 35,
temperature: float = 0.7,
num_pipeline_ranks: int = 1,
instruct: bool = False,
lora_path: Optional[str] = None,
) -> None:
if is_torchrun():
torch.distributed.init_process_group()
torch.cuda.set_device(torch.distributed.get_rank())
should_print = torch.distributed.get_rank() == 0
num_pipeline_ranks = torch.distributed.get_world_size()
else:
should_print = True
num_pipeline_ranks = 1
mistral_tokenizer: MistralTokenizer = load_tokenizer(Path(model_path))
tokenizer: Tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer
model_cls = get_model_cls(model_path)
model = model_cls.from_folder(Path(model_path), max_batch_size=3, num_pipeline_ranks=num_pipeline_ranks)
is_multimodal = isinstance(model.args, TransformerArgs) and model.args.vision_encoder is not None
if is_multimodal:
assert instruct, "Multimodal models should only be used in instruct mode"
# load LoRA
if lora_path is not None:
model.load_lora(Path(lora_path))
prompt: str = ""
messages: List[UserMessage | AssistantMessage] = []
while True:
if should_print:
if not is_multimodal:
user_input = input("Prompt: ")
if instruct:
if is_multimodal:
mm_input, finished = _get_multimodal_input()
if finished:
break
messages += [mm_input]
else:
messages += [UserMessage(content=user_input)]
chat_completion_request = ChatCompletionRequest(messages=messages)
tokenized = mistral_tokenizer.encode_chat_completion(chat_completion_request)
tokens = tokenized.tokens
images = tokenized.images
else:
prompt += user_input
tokens = tokenizer.encode(prompt, bos=True, eos=False)
images = []
length_tensor = torch.tensor([len(tokens)], dtype=torch.int)
else:
length_tensor = torch.tensor([0], dtype=torch.int)
images = []
if is_torchrun():
dist.broadcast(length_tensor, src=0)
if not should_print:
tokens = int(length_tensor.item()) * [0]
if isinstance(model, Transformer):
generated_tokens, _ = generate(
[tokens],
model,
[images],
max_tokens=max_tokens,
temperature=temperature,
eos_id=tokenizer.eos_id,
)
else:
# Mamba models don't support images
generated_tokens, _ = generate_mamba(
[tokens],
model,
max_tokens=max_tokens,
temperature=temperature,
eos_id=tokenizer.eos_id,
)
answer = tokenizer.decode(generated_tokens[0])
if should_print:
print(answer)
print("=====================")
if instruct:
messages += [AssistantMessage(content=answer)]
else:
prompt += answer
def demo(
model_path: str,
max_tokens: int = 35,
temperature: float = 0,
lora_path: Optional[str] = None,
) -> None:
if is_torchrun():
torch.distributed.init_process_group()
torch.cuda.set_device(torch.distributed.get_rank())
should_print = torch.distributed.get_rank() == 0
num_pipeline_ranks = torch.distributed.get_world_size()
else:
should_print = True
num_pipeline_ranks = 1
model_cls = get_model_cls(model_path)
model = model_cls.from_folder(Path(model_path), max_batch_size=3, num_pipeline_ranks=num_pipeline_ranks)
# load LoRA
if lora_path is not None:
model.load_lora(Path(lora_path))
mistral_tokenizer: MistralTokenizer = load_tokenizer(Path(model_path))
tokenizer: Tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer
prompts = [
"This is a test",
"This is another great test",
"This is a third test, mistral AI is very good at testing. ",
]
encoded_prompts = [tokenizer.encode(prompt, bos=True, eos=False) for prompt in prompts]
if isinstance(model, Transformer):
generate_fn = generate
else:
generate_fn = generate_mamba # type: ignore[assignment]
warnings.warn(
"Batched generation is not correctly supported at the moment and therefore might lead to worse results "
"as compared to non-batched generation. "
"See https://github.com/state-spaces/mamba/issues/66#issuecomment-1862349718 for more information."
)
encoded_prompts = pad_and_convert_to_tensor(encoded_prompts, mistral_tokenizer.instruct_tokenizer.BOS) # type: ignore[attr-defined]
generated_tokens, _logprobs = generate_fn(
encoded_prompts,
model, # type: ignore[arg-type]
max_tokens=max_tokens,
temperature=temperature,
eos_id=tokenizer.eos_id,
)
generated_words = []
for i, x in enumerate(generated_tokens):
generated_words.append(tokenizer.decode(encoded_prompts[i] + x))
res = generated_words
if should_print:
for w, logprob in zip(res, _logprobs):
print(w)
logging.debug("Logprobs: %s", logprob)
print("=====================")
def mistral_chat() -> None:
fire.Fire(interactive)
def mistral_demo() -> None:
fire.Fire(demo)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
fire.Fire(
{
"interactive": interactive,
"demo": demo,
}
)
================================================
FILE: src/mistral_inference/mamba.py
================================================
import json
from pathlib import Path
from typing import List, Optional, Union
import safetensors
import torch
import torch.nn as nn
from mistral_inference.args import MambaArgs
from mistral_inference.cache import BufferCache
from mistral_inference.model import ModelBase
_is_mamba_installed = False
try:
from mamba_ssm.models.config_mamba import MambaConfig
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel
_is_mamba_installed = True
except ImportError:
_is_mamba_installed = False
class Mamba(ModelBase, nn.Module):
def __init__(self, args: MambaArgs):
super().__init__()
self.args = args
assert _is_mamba_installed, "Mamba is not installed. Please install it using `pip install mamba-ssm`."
# make sure naming is consistent with `mamba_ssm`
config = MambaConfig(
d_model=args.dim,
n_layer=args.n_layers,
vocab_size=args.vocab_size,
ssm_cfg={"ngroups": args.n_groups, "layer": "Mamba2"},
attn_layer_idx=[],
attn_cfg={},
rms_norm=args.rms_norm,
residual_in_fp32=args.residual_in_fp32,
fused_add_norm=args.fused_add_norm,
pad_vocab_size_multiple=args.pad_vocab_size_multiple,
tie_embeddings=args.tie_embeddings,
)
self.model = MambaLMHeadModel(config)
@property
def dtype(self) -> torch.dtype:
return next(self.parameters()).dtype
@property
def device(self) -> torch.device:
return next(self.parameters()).device
def forward(
self,
input_ids: torch.Tensor,
seqlens: List[int], # not supported for now
cache: Optional[BufferCache] = None, # not supported for now
) -> torch.Tensor:
lm_output = self.model(input_ids)
result: torch.Tensor = lm_output.logits
return result
@staticmethod
def from_folder(
folder: Union[Path, str],
max_batch_size: int = 1,
num_pipeline_ranks: int = 1,
device: Union[torch.device, str] = "cuda",
dtype: Optional[torch.dtype] = None,
) -> "Mamba":
with open(Path(folder) / "params.json", "r") as f:
model_args = MambaArgs.from_dict(json.load(f))
with torch.device("meta"):
model = Mamba(model_args)
model_file = Path(folder) / "consolidated.safetensors"
assert model_file.exists(), f"Make sure {model_file} exists."
loaded = safetensors.torch.load_file(str(model_file))
model.load_state_dict(loaded, assign=True, strict=True)
return model.to(device=device, dtype=dtype)
================================================
FILE: src/mistral_inference/model.py
================================================
from abc import ABC, abstractmethod
from pathlib import Path
from typing import List, Optional, Union
import torch
import torch.nn as nn
from mistral_inference.cache import BufferCache
class ModelBase(nn.Module, ABC):
def __init__(self) -> None:
super().__init__()
@property
@abstractmethod
def dtype(self) -> torch.dtype:
pass
@property
@abstractmethod
def device(self) -> torch.device:
pass
@abstractmethod
def forward(
self,
input_ids: torch.Tensor,
seqlens: List[int], # not supported for now
cache: Optional[BufferCache] = None, # not supported for now
) -> torch.Tensor:
pass
@staticmethod
@abstractmethod
def from_folder(
folder: Union[Path, str],
max_batch_size: int = 1,
num_pipeline_ranks: int = 1,
device: Union[torch.device, str] = "cuda",
dtype: Optional[torch.dtype] = None,
) -> "ModelBase":
pass
================================================
FILE: src/mistral_inference/moe.py
================================================
import dataclasses
from typing import List
import torch
import torch.nn.functional as F
from simple_parsing.helpers import Serializable
from torch import nn
@dataclasses.dataclass
class MoeArgs(Serializable):
num_experts: int
num_experts_per_tok: int
class MoeLayer(nn.Module):
def __init__(self, experts: List[nn.Module], gate: nn.Module, moe_args: MoeArgs):
super().__init__()
assert len(experts) > 0
self.experts = nn.ModuleList(experts)
self.gate = gate
self.args = moe_args
def forward(self, inputs: torch.Tensor) -> torch.Tensor:
gate_logits = self.gate(inputs)
weights, selected_experts = torch.topk(gate_logits, self.args.num_experts_per_tok)
weights = F.softmax(weights, dim=1, dtype=torch.float).to(inputs.dtype)
results = torch.zeros_like(inputs)
for i, expert in enumerate(self.experts):
batch_idx, nth_expert = torch.where(selected_experts == i)
results[batch_idx] += weights[batch_idx, nth_expert, None] * expert(inputs[batch_idx])
return results
================================================
FILE: src/mistral_inference/rope.py
================================================
from typing import Tuple
import torch
def precompute_freqs_cis(dim: int, end: int, theta: float) -> torch.Tensor:
freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
t = torch.arange(end, device=freqs.device)
freqs = torch.outer(t, freqs).float()
return torch.polar(torch.ones_like(freqs), freqs) # complex64
def apply_rotary_emb(
xq: torch.Tensor,
xk: torch.Tensor,
freqs_cis: torch.Tensor,
) -> Tuple[torch.Tensor, torch.Tensor]:
xq_ = torch.view_as_complex(xq.float().reshape(*xq.shape[:-1], -1, 2))
xk_ = torch.view_as_complex(xk.float().reshape(*xk.shape[:-1], -1, 2))
freqs_cis = freqs_cis[:, None, :]
xq_out = torch.view_as_real(xq_ * freqs_cis).flatten(-2)
xk_out = torch.view_as_real(xk_ * freqs_cis).flatten(-2)
return xq_out.type_as(xq), xk_out.type_as(xk)
def precompute_freqs_cis_2d(
dim: int,
height: int,
width: int,
theta: float,
) -> torch.Tensor:
"""
freqs_cis: 2D complex tensor of shape (height, width, dim // 2) to be indexed by
(height, width) position tuples
"""
# (dim / 2) frequency bases
freqs = 1.0 / (theta ** (torch.arange(0, dim, 2).float() / dim))
h = torch.arange(height, device=freqs.device)
w = torch.arange(width, device=freqs.device)
freqs_h = torch.outer(h, freqs[::2]).float()
freqs_w = torch.outer(w, freqs[1::2]).float()
freqs_2d = torch.cat(
[
freqs_h[:, None, :].repeat(1, width, 1),
freqs_w[None, :, :].repeat(height, 1, 1),
],
dim=-1,
)
return torch.polar(torch.ones_like(freqs_2d), freqs_2d)
================================================
FILE: src/mistral_inference/transformer.py
================================================
import json
import logging
import math
from dataclasses import dataclass
from pathlib import Path
from typing import Any, List, Mapping, Optional, Union
import safetensors.torch
import torch
from torch import nn
from mistral_inference.args import PATCH_MERGE, TransformerArgs
from mistral_inference.cache import BufferCache, CacheInputMetadata
from mistral_inference.lora import LoRALoaderMixin
from mistral_inference.model import ModelBase
from mistral_inference.rope import precompute_freqs_cis
from mistral_inference.transformer_layers import RMSNorm, TransformerBlock
from mistral_inference.vision_encoder import PatchMerger, VisionLanguageAdapter, VisionTransformer
@dataclass
class SimpleInputMetadata:
# rope absolute positions
positions: torch.Tensor
@staticmethod
def from_seqlens(seqlens: List[int], device: torch.device) -> "SimpleInputMetadata":
return SimpleInputMetadata(
positions=torch.cat([torch.arange(0, seqlen) for seqlen in seqlens]).to(device=device, dtype=torch.long)
)
class Transformer(ModelBase, LoRALoaderMixin):
def __init__(
self,
args: TransformerArgs,
pipeline_rank: int = 0,
num_pipeline_ranks: int = 1,
softmax_fp32: bool = True,
):
super().__init__()
self.args = args
self.vocab_size = args.vocab_size
self.n_layers = args.n_layers
self._precomputed_freqs_cis: Optional[torch.Tensor] = None
assert self.vocab_size > 0
assert pipeline_rank < num_pipeline_ranks, (pipeline_rank, num_pipeline_ranks)
self.pipeline_rank = pipeline_rank
self.num_pipeline_ranks = num_pipeline_ranks
self.softmax_fp32 = softmax_fp32
# Modules specific to some ranks:
self.tok_embeddings: Optional[nn.Embedding] = None
self.norm: Optional[RMSNorm] = None
self.output: Optional[nn.Linear] = None
if pipeline_rank == 0:
self.tok_embeddings = nn.Embedding(args.vocab_size, args.dim)
self.vision_encoder: Optional[VisionTransformer] = None
self.vision_language_adapter: Optional[VisionLanguageAdapter] = None
if args.vision_encoder is not None:
self.vision_encoder = VisionTransformer(args.vision_encoder)
self.vision_language_adapter = VisionLanguageAdapter(
args.vision_encoder.hidden_size, args.dim, args.vision_encoder.adapter_bias
)
if args.vision_encoder.add_pre_mm_projector_layer_norm:
self.pre_mm_projector_norm = RMSNorm(args.vision_encoder.hidden_size, eps=1e-5)
if args.vision_encoder.mm_projector_id == PATCH_MERGE:
self.patch_merger = PatchMerger(
vision_encoder_dim=args.vision_encoder.hidden_size,
spatial_merge_size=args.vision_encoder.spatial_merge_size,
)
if pipeline_rank == num_pipeline_ranks - 1:
self.norm = RMSNorm(args.dim, eps=args.norm_eps)
self.output = nn.Linear(args.dim, args.vocab_size, bias=False)
# Initialize all layers but slice off those not of this rank.
layers = [
TransformerBlock(
dim=args.dim,
hidden_dim=args.hidden_dim,
n_heads=args.n_heads,
n_kv_heads=args.n_kv_heads,
head_dim=args.head_dim,
norm_eps=args.norm_eps,
lora=args.lora,
moe=args.moe,
)
for _ in range(args.n_layers)
]
num_layers_per_rank = math.ceil(self.n_layers / self.num_pipeline_ranks)
offset = self.pipeline_rank * num_layers_per_rank
end = min(self.n_layers, offset + num_layers_per_rank)
self.layers = nn.ModuleDict({str(i): layers[i] for i in range(offset, end)})
self.n_local_layers = len(self.layers)
@property
def dtype(self) -> torch.dtype:
return next(self.parameters()).dtype
@property
def device(self) -> torch.device:
return next(self.parameters()).device
@property
def freqs_cis(self) -> torch.Tensor:
# We cache freqs_cis but need to take care that it is on the right device
# and has the right dtype (complex64). The fact that the dtype is different
# from the module's dtype means we cannot register it as a buffer
if self._precomputed_freqs_cis is None:
# default to 10**6
theta = self.args.rope_theta or 1000000.0
self._precomputed_freqs_cis = precompute_freqs_cis(self.args.head_dim, 128_000, theta)
if self._precomputed_freqs_cis.device != self.device:
self._precomputed_freqs_cis = self._precomputed_freqs_cis.to(device=self.device)
return self._precomputed_freqs_cis
def embed_vision_language_features(self, input_ids: torch.Tensor, images: List[torch.Tensor]) -> torch.Tensor:
assert self.tok_embeddings is not None
assert self.vision_encoder is not None
assert self.vision_language_adapter is not None
assert self.args.vision_encoder is not None
text_locations = input_ids != self.args.vision_encoder.image_token_id
image_locations = input_ids == self.args.vision_encoder.image_token_id
text_features = self.tok_embeddings(input_ids[text_locations])
image_features = self.vision_encoder(images)
if self.args.vision_encoder.add_pre_mm_projector_layer_norm:
image_features = self.pre_mm_projector_norm(image_features)
if self.args.vision_encoder.mm_projector_id == PATCH_MERGE:
patch_size = self.args.vision_encoder.patch_size
img_patch_dims = [(img.shape[1] // patch_size, img.shape[2] // patch_size) for img in images]
image_features = self.patch_merger(image_features, image_sizes=img_patch_dims)
image_features = self.vision_language_adapter(image_features)
N_txt, D_txt = text_features.shape
N_img, D_img = image_features.shape
seq_len = input_ids.shape[0]
assert D_txt == D_img, f"Text features dim {D_txt} should be equal to image features dim {D_img}"
assert seq_len == N_txt + N_img, (
f"seq_len {seq_len} should be equal to N_txt + N_img {(N_txt, N_img, image_locations.sum().item())}"
)
combined_features = torch.empty(
(seq_len, D_txt),
dtype=text_features.dtype,
device=text_features.device,
)
combined_features[text_locations, :] = text_features
combined_features[image_locations, :] = image_features
return combined_features
def forward_partial(
self,
input_ids: torch.Tensor,
seqlens: List[int],
cache: Optional[BufferCache] = None,
images: Optional[List[torch.Tensor]] = None,
) -> torch.Tensor:
"""Local forward pass.
If doing pipeline parallelism, this will return the activations of the last layer of this stage.
For the last stage, this will return the normalized final embeddings.
"""
assert len(seqlens) <= self.args.max_batch_size, (
f"Max batch size is {self.args.max_batch_size}, got batch size of {len(seqlens)}"
)
(num_toks,) = input_ids.shape
assert sum(seqlens) == num_toks, (sum(seqlens), num_toks)
input_metadata: List[CacheInputMetadata] | List[SimpleInputMetadata]
if cache is not None:
input_metadata = cache.get_input_metadata(seqlens)
else:
input_metadata = [SimpleInputMetadata.from_seqlens(seqlens, self.device) for _ in range(len(self.layers))]
if self.pipeline_rank == 0:
assert self.tok_embeddings is not None
if self.vision_encoder is not None and images:
h = self.embed_vision_language_features(input_ids, images)
else:
h = self.tok_embeddings(input_ids)
else:
h = torch.empty(num_toks, self.args.dim, device=self.device, dtype=self.dtype)
torch.distributed.recv(h, src=self.pipeline_rank - 1)
# freqs_cis is always the same for every layer
freqs_cis = self.freqs_cis[input_metadata[0].positions]
for local_layer_id, layer in enumerate(self.layers.values()):
if cache is not None:
assert input_metadata is not None
cache_metadata = input_metadata[local_layer_id]
assert isinstance(cache_metadata, CacheInputMetadata)
cache_view = cache.get_view(local_layer_id, cache_metadata)
else:
cache_view = None
h = layer(h, freqs_cis, cache_view)
if cache is not None:
cache.update_seqlens(seqlens)
if self.pipeline_rank < self.num_pipeline_ranks - 1:
torch.distributed.send(h, dst=self.pipeline_rank + 1)
return h
else:
# Last rank has a final normalization step.
assert self.norm is not None
return self.norm(h) # type: ignore
def forward(
self,
input_ids: torch.Tensor,
seqlens: List[int],
cache: Optional[BufferCache] = None,
images: Optional[List[torch.Tensor]] = None,
) -> torch.Tensor:
h = self.forward_partial(input_ids, seqlens, cache=cache, images=images)
if self.pipeline_rank < self.num_pipeline_ranks - 1:
# ignore the intermediate activations as we'll get the final output from
# the last stage
outs = torch.empty(h.shape[0], self.vocab_size, device=h.device, dtype=h.dtype)
else:
assert self.output is not None
outs = self.output(h)
if self.num_pipeline_ranks > 1:
torch.distributed.broadcast(outs, src=self.num_pipeline_ranks - 1)
if self.softmax_fp32:
return outs.float()
else:
return outs
def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool = True, assign: bool = False) -> None:
state_to_load = {}
skipped = set([])
for k, v in state_dict.items():
if k.startswith("tok_embeddings"):
if self.pipeline_rank == 0:
state_to_load[k] = v
else:
logging.debug(
"Skipping parameter %s at pipeline rank %d",
k,
self.pipeline_rank,
)
skipped.add(k)
elif k.startswith("norm") or k.startswith("output"):
if self.pipeline_rank == self.num_pipeline_ranks - 1:
state_to_load[k] = v
else:
logging.debug(
"Skipping parameter %s at pipeline rank %d",
k,
self.pipeline_rank,
)
skipped.add(k)
elif k.startswith("layers"):
layer_id = k.split(".")[1]
if layer_id in self.layers:
state_to_load[k] = v
else:
logging.debug(
"Skipping parameter %s at pipeline rank %d",
k,
self.pipeline_rank,
)
skipped.add(k)
elif any(
k.startswith(key)
for key in ["vision_encoder", "vision_language_adapter", "patch_merger", "pre_mm_projector_norm"]
):
if self.pipeline_rank == 0:
state_to_load[k] = v
else:
logging.debug(
"Skipping parameter %s at pipeline rank %d",
k,
self.pipeline_rank,
)
skipped.add(k)
else:
raise ValueError(f"Unexpected key {k}")
assert set(state_dict.keys()) == skipped.union(set(state_to_load.keys()))
super().load_state_dict(state_to_load, strict=strict, assign=assign)
@staticmethod
def from_folder(
folder: Union[Path, str],
max_batch_size: int = 1,
num_pipeline_ranks: int = 1,
device: Union[torch.device, str] = "cuda",
dtype: Optional[torch.dtype] = None,
softmax_fp32: bool = True,
) -> "Transformer":
with open(Path(folder) / "params.json", "r") as f:
model_args = TransformerArgs.from_dict(json.load(f))
model_args.max_batch_size = max_batch_size
if num_pipeline_ranks > 1:
pipeline_rank = torch.distributed.get_rank()
else:
pipeline_rank = 0
with torch.device("meta"):
model = Transformer(
model_args,
pipeline_rank=pipeline_rank,
num_pipeline_ranks=num_pipeline_ranks,
softmax_fp32=softmax_fp32,
)
pt_model_file = Path(folder) / "consolidated.00.pth"
safetensors_model_file = Path(folder) / "consolidated.safetensors"
assert pt_model_file.exists() or safetensors_model_file.exists(), (
f"Make sure either {pt_model_file} or {safetensors_model_file} exists"
)
assert not (pt_model_file.exists() and safetensors_model_file.exists()), (
f"Both {pt_model_file} and {safetensors_model_file} cannot exist"
)
if pt_model_file.exists():
loaded = torch.load(str(pt_model_file), mmap=True)
else:
loaded = safetensors.torch.load_file(str(safetensors_model_file))
model.load_state_dict(loaded, assign=True, strict=True)
return model.to(device=device, dtype=dtype)
================================================
FILE: src/mistral_inference/transformer_layers.py
================================================
from functools import partial
from typing import Optional, Tuple, Type, Union
import torch
from torch import nn
from xformers.ops.fmha import memory_efficient_attention # type: ignore
from xformers.ops.fmha.attn_bias import BlockDiagonalMask
from mistral_inference.args import LoraArgs
from mistral_inference.cache import CacheView
from mistral_inference.lora import LoRALinear
from mistral_inference.moe import MoeArgs, MoeLayer
from mistral_inference.rope import apply_rotary_emb
def repeat_kv(keys: torch.Tensor, values: torch.Tensor, repeats: int, dim: int) -> Tuple[torch.Tensor, torch.Tensor]:
keys = torch.repeat_interleave(keys, repeats=repeats, dim=dim)
values = torch.repeat_interleave(values, repeats=repeats, dim=dim)
return keys, values
def maybe_lora(
lora_args: Optional[LoraArgs],
) -> Union[Type[nn.Linear], partial[LoRALinear]]:
if lora_args is None:
return nn.Linear
else:
return partial(LoRALinear, rank=lora_args.rank, scaling=lora_args.scaling)
class Attention(nn.Module):
def __init__(
self,
dim: int,
n_heads: int,
head_dim: int,
n_kv_heads: int,
lora: Optional[LoraArgs] = None,
):
super().__init__()
self.n_heads: int = n_heads
self.head_dim: int = head_dim
self.n_kv_heads: int = n_kv_heads
self.repeats = self.n_heads // self.n_kv_heads
self.scale = self.head_dim**-0.5
MaybeLora = maybe_lora(lora)
self.wq = MaybeLora(dim, n_heads * head_dim, bias=False)
self.wk = MaybeLora(dim, n_kv_heads * head_dim, bias=False)
self.wv = MaybeLora(dim, n_kv_heads * head_dim, bias=False)
self.wo = MaybeLora(n_heads * head_dim, dim, bias=False)
def forward(
self,
x: torch.Tensor,
freqs_cis: torch.Tensor,
cache: Optional[CacheView] = None,
mask: Optional[BlockDiagonalMask] = None,
) -> torch.Tensor:
assert mask is None or cache is None
seqlen_sum, _ = x.shape
xq, xk, xv = self.wq(x), self.wk(x), self.wv(x)
xq = xq.view(seqlen_sum, self.n_heads, self.head_dim)
xk = xk.view(seqlen_sum, self.n_kv_heads, self.head_dim)
xv = xv.view(seqlen_sum, self.n_kv_heads, self.head_dim)
xq, xk = apply_rotary_emb(xq, xk, freqs_cis=freqs_cis)
if cache is None:
key, val = xk, xv
elif cache.prefill:
key, val = cache.interleave_kv(xk, xv)
cache.update(xk, xv)
else:
cache.update(xk, xv)
key, val = cache.key, cache.value
key = key.view(seqlen_sum * cache.max_seq_len, self.n_kv_heads, self.head_dim)
val = val.view(seqlen_sum * cache.max_seq_len, self.n_kv_heads, self.head_dim)
# Repeat keys and values to match number of query heads
key, val = repeat_kv(key, val, self.repeats, dim=1)
# xformers requires (B=1, S, H, D)
xq, key, val = xq[None, ...], key[None, ...], val[None, ...]
output = memory_efficient_attention(xq, key, val, mask if cache is None else cache.mask)
output = output.view(seqlen_sum, self.n_heads * self.head_dim)
assert isinstance(output, torch.Tensor)
return self.wo(output) # type: ignore
class FeedForward(nn.Module):
def __init__(self, dim: int, hidden_dim: int, lora: Optional[LoraArgs] = None):
super().__init__()
MaybeLora = maybe_lora(lora)
self.w1 = MaybeLora(dim, hidden_dim, bias=False)
self.w2 = MaybeLora(hidden_dim, dim, bias=False)
self.w3 = MaybeLora(dim, hidden_dim, bias=False)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.w2(nn.functional.silu(self.w1(x)) * self.w3(x)) # type: ignore
class RMSNorm(torch.nn.Module):
def __init__(self, dim: int, eps: float = 1e-6):
super().__init__()
self.eps = eps
self.weight = nn.Parameter(torch.ones(dim))
def _norm(self, x: torch.Tensor) -> torch.Tensor:
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)
def forward(self, x: torch.Tensor) -> torch.Tensor:
output = self._norm(x.float()).type_as(x)
return output * self.weight
class TransformerBlock(nn.Module):
def __init__(
self,
dim: int,
hidden_dim: int,
n_heads: int,
n_kv_heads: int,
head_dim: int,
norm_eps: float,
lora: Optional[LoraArgs] = None,
moe: Optional[MoeArgs] = None,
):
super().__init__()
self.n_heads = n_heads
self.dim = dim
self.attention = Attention(
dim=dim,
n_heads=n_heads,
head_dim=head_dim,
n_kv_heads=n_kv_heads,
lora=lora,
)
self.attention_norm = RMSNorm(dim, eps=norm_eps)
self.ffn_norm = RMSNorm(dim, eps=norm_eps)
self.feed_forward: nn.Module
if moe is not None:
self.feed_forward = MoeLayer(
experts=[FeedForward(dim=dim, hidden_dim=hidden_dim, lora=lora) for _ in range(moe.num_experts)],
gate=nn.Linear(dim, moe.num_experts, bias=False),
moe_args=moe,
)
else:
self.feed_forward = FeedForward(dim=dim, hidden_dim=hidden_dim, lora=lora)
def forward(
self,
x: torch.Tensor,
freqs_cis: torch.Tensor,
cache: Optional[CacheView] = None,
mask: Optional[BlockDiagonalMask] = None,
) -> torch.Tensor:
r = self.attention.forward(self.attention_norm(x), freqs_cis, cache)
h = x + r
r = self.feed_forward.forward(self.ffn_norm(h))
out = h + r
return out
================================================
FILE: src/mistral_inference/vision_encoder.py
================================================
from typing import List, Optional
import torch
import torch.nn as nn
from xformers.ops.fmha.attn_bias import BlockDiagonalMask
from mistral_inference.args import VisionEncoderArgs
from mistral_inference.rope import precompute_freqs_cis_2d
from mistral_inference.transformer_layers import RMSNorm, TransformerBlock
def position_meshgrid(
patch_embeds_list: list[torch.Tensor],
) -> torch.Tensor:
positions = torch.cat(
[
torch.stack(
torch.meshgrid(
torch.arange(p.shape[-2]),
torch.arange(p.shape[-1]),
indexing="ij",
),
dim=-1,
).reshape(-1, 2)
for p in patch_embeds_list
]
)
return positions
class VisionTransformer(nn.Module):
def __init__(self, args: VisionEncoderArgs):
super().__init__()
self.args = args
self.patch_conv = nn.Conv2d(
in_channels=args.num_channels,
out_channels=args.hidden_size,
kernel_size=args.patch_size,
stride=args.patch_size,
bias=False,
)
self.ln_pre = RMSNorm(args.hidden_size, eps=1e-5)
self.transformer = VisionTransformerBlocks(args)
head_dim = self.args.hidden_size // self.args.num_attention_heads
assert head_dim % 2 == 0, "ROPE requires even head_dim"
self._freqs_cis: Optional[torch.Tensor] = None
@property
def max_patches_per_side(self) -> int:
return self.args.image_size // self.args.patch_size
@property
def device(self) -> torch.device:
return next(self.parameters()).device
@property
def freqs_cis(self) -> torch.Tensor:
if self._freqs_cis is None:
self._freqs_cis = precompute_freqs_cis_2d(
dim=self.args.hidden_size // self.args.num_attention_heads,
height=self.max_patches_per_side,
width=self.max_patches_per_side,
theta=self.args.rope_theta,
)
if self._freqs_cis.device != self.device:
self._freqs_cis = self._freqs_cis.to(device=self.device)
return self._freqs_cis
def forward(
self,
images: List[torch.Tensor],
) -> torch.Tensor:
"""
Args:
images: list of N_img images of variable sizes, each of shape (C, H, W)
Returns:
image_features: tensor of token features for all tokens of all images of
shape (N_toks, D)
"""
# pass images through initial convolution independently
patch_embeds_list = [self.patch_conv(img.unsqueeze(0)).squeeze(0) for img in images]
# flatten to a single sequence
patch_embeds = torch.cat([p.flatten(1).permute(1, 0) for p in patch_embeds_list], dim=0)
patch_embeds = self.ln_pre(patch_embeds)
# positional embeddings
positions = position_meshgrid(patch_embeds_list).to(self.device)
freqs_cis = self.freqs_cis[positions[:, 0], positions[:, 1]]
# pass through Transformer with a block diagonal mask delimiting images
mask = BlockDiagonalMask.from_seqlens(
[p.shape[-2] * p.shape[-1] for p in patch_embeds_list],
)
out = self.transformer(patch_embeds, mask=mask, freqs_cis=freqs_cis)
# remove batch dimension of the single sequence
return out # type: ignore[no-any-return]
class VisionLanguageAdapter(nn.Module):
def __init__(self, in_dim: int, out_dim: int, bias: bool = True):
super().__init__()
self.w_in = nn.Linear(
in_dim,
out_dim,
bias=bias,
)
self.gelu = nn.GELU()
self.w_out = nn.Linear(out_dim, out_dim, bias=bias)
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.w_out(self.gelu(self.w_in(x))) # type: ignore[no-any-return]
class VisionTransformerBlocks(nn.Module):
def __init__(self, args: VisionEncoderArgs):
super().__init__()
self.layers = torch.nn.ModuleList()
for _ in range(args.num_hidden_layers):
self.layers.append(
TransformerBlock(
dim=args.hidden_size,
hidden_dim=args.intermediate_size,
n_heads=args.num_attention_heads,
n_kv_heads=args.num_attention_heads,
head_dim=args.hidden_size // args.num_attention_heads,
norm_eps=1e-5,
)
)
def forward(
self,
x: torch.Tensor,
mask: BlockDiagonalMask,
freqs_cis: Optional[torch.Tensor],
) -> torch.Tensor:
for layer in self.layers:
x = layer(x, mask=mask, freqs_cis=freqs_cis)
return x
class PatchMerger(nn.Module):
"""
Learned merging of spatial_merge_size ** 2 patches
"""
def __init__(
self,
vision_encoder_dim: int,
spatial_merge_size: int,
) -> None:
super().__init__()
mlp_input_dim = vision_encoder_dim * (spatial_merge_size**2)
self.spatial_merge_size = spatial_merge_size
self.mlp_input_dim = mlp_input_dim
self.merging_layer = nn.Linear(mlp_input_dim, vision_encoder_dim, bias=False)
def forward(self, x: torch.Tensor, image_sizes: list[tuple[int, int]]) -> torch.Tensor:
# image_sizes specified in tokens
assert sum([h * w for h, w in image_sizes]) == len(x), f"{sum([h * w for h, w in image_sizes])} != {len(x)}"
# x is (N, vision_encoder_dim)
x = self.permute(x, image_sizes)
# x is (N / spatial_merge_size ** 2,
# vision_encoder_dim * spatial_merge_size ** 2)
x = self.merging_layer(x)
# x is (N / spatial_merge_size ** 2, vision_encoder_dim)
return x
def permute(
self,
x: torch.Tensor,
image_sizes: list[tuple[int, int]],
) -> torch.Tensor:
"""
Args:
x: (N, D) where N is flattened and concatenated patch tokens
for all images
image_sizes: list of tuple of (height, width) in tokens for
each image
Returns:
image_features: reorders patch tokens so each grid of
(spatial_merge_size, spatial_merge_size) is contiguous.
now (N / spatial_merge_size ** 2, D * spatial_merge_size ** 2)
"""
sub_grids = get_sub_grids(
x=x, image_sizes=image_sizes, spatial_merge_size=self.spatial_merge_size
) # list of [d x sub_grid_size x sub_grid_size x n_patches]
permuted_tensor = [
grid.view(-1, grid.shape[-1]).t() for grid in sub_grids
] # n_patches x d * sub_grid_size * sub_grid_size
return torch.cat(permuted_tensor, dim=0) # (N / spatial_merge_size ** 2, d * spatial_merge_size ** 2)
def get_sub_grids(
x: torch.Tensor,
image_sizes: list[tuple[int, int]],
spatial_merge_size: int,
) -> list[torch.Tensor]:
# image_sizes specified in tokens
tokens_per_image = [h * w for h, w in image_sizes]
d = x.shape[-1]
all_img_sub_grids: list[torch.Tensor] = []
sub_grid_size = spatial_merge_size
for image_index, image_tokens in enumerate(x.split(tokens_per_image)):
# Reshape image_tokens into a 2D grid
h, w = image_sizes[image_index]
image_grid = image_tokens.view(h, w, d).permute(2, 0, 1)[None, :, :, :] # 1 x d x h x w
sub_grids = torch.nn.functional.unfold(image_grid, kernel_size=sub_grid_size, stride=sub_grid_size)
sub_grids = sub_grids.view(
1, d, sub_grid_size, sub_grid_size, -1
) # 1 x d x sub_grid_size x sub_grid_size x n_patches
all_img_sub_grids.append(sub_grids[0])
return all_img_sub_grids
================================================
FILE: tests/test_generate.py
================================================
from typing import List
import numpy as np
import torch
from mistral_inference.args import VisionEncoderArgs
from mistral_inference.generate import generate_mamba
from mistral_inference.main import generate
from mistral_inference.mamba import Mamba, MambaArgs
from mistral_inference.transformer import Transformer, TransformerArgs
class DebugTokenizer:
@property
def bos_id(self) -> int:
return 0
@property
def eos_id(self) -> int:
return 1
@property
def pad_id(self) -> int:
return -1
def encode(self, s: str, bos: bool = True) -> List[int]:
assert isinstance(s, str)
t = [int(x) for x in s.split()]
if bos:
t = [self.bos_id, *t]
return t
def decode(self, t: List[int]) -> str:
return " ".join([str(x) for x in t])
def test_generation_transformer() -> None:
torch.manual_seed(42)
sequences = ["1 2 3 4 5 6 7", "0 1 2", "12 13 14", "2 4 34"]
args = TransformerArgs(
dim=512,
n_layers=1,
head_dim=128,
hidden_dim=2048,
n_heads=4,
n_kv_heads=2,
norm_eps=1e-5,
vocab_size=32_000,
max_batch_size=len(sequences),
)
model = Transformer(args).to("cuda", dtype=torch.float32)
tokenizer = DebugTokenizer()
encoded = [tokenizer.encode(s, bos=True) for s in sequences]
toks, all_logprobs_old = generate(encoded, model, temperature=0.0, max_tokens=7)
# concat generated and prompt
encoded = [e + t for e, t in zip(encoded, toks)]
generated, all_logprobs_new = generate(encoded, model, temperature=0.0, max_tokens=0)
assert generated == []
# Verify that logprobs are the same
assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new)
for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):
assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}"
print("All tests passed.")
def test_generation_pixtral() -> None:
torch.manual_seed(42)
gen = np.random.default_rng(seed=42)
sequences = ["1 2 2 2 2 4 5 6 7", "12 13 14", "2 2 2 2 7 8 9"]
images = [[gen.normal(size=(3, 4, 4))], [], [gen.normal(size=(3, 4, 4))]]
args = TransformerArgs(
dim=512,
n_layers=1,
head_dim=128,
hidden_dim=2048,
n_heads=4,
n_kv_heads=2,
norm_eps=1e-5,
vocab_size=32_000,
max_batch_size=len(sequences),
vision_encoder=VisionEncoderArgs(
hidden_size=128,
num_channels=3,
image_size=4,
patch_size=2,
intermediate_size=256,
num_hidden_layers=1,
num_attention_heads=2,
rope_theta=10000,
image_token_id=2,
),
)
model = Transformer(args).to("cuda", dtype=torch.float32)
tokenizer = DebugTokenizer()
encoded = [tokenizer.encode(s, bos=True) for s in sequences]
toks, all_logprobs_old = generate(encoded, model, images=images, temperature=0.0, max_tokens=7)
# concat generated and prompt
encoded = [e + t for e, t in zip(encoded, toks)]
generated, all_logprobs_new = generate(encoded, model, images=images, temperature=0.0, max_tokens=0)
assert generated == []
# Verify that logprobs are the same
assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new)
for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):
assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}"
print("All tests passed.")
def test_generation_pixtral_patch_merger() -> None:
torch.manual_seed(42)
gen = np.random.default_rng(seed=42)
sequences = ["1 2 2 2 2 4 5 6 7", "12 13 14", "2 2 2 2 7 8 9"]
images = [[gen.normal(size=(3, 8, 8))], [], [gen.normal(size=(3, 8, 8))]]
args = TransformerArgs(
dim=512,
n_layers=1,
head_dim=128,
hidden_dim=2048,
n_heads=4,
n_kv_heads=2,
norm_eps=1e-5,
vocab_size=32_000,
max_batch_size=len(sequences),
vision_encoder=VisionEncoderArgs(
hidden_size=128,
num_channels=3,
image_size=8,
patch_size=2,
intermediate_size=256,
num_hidden_layers=1,
num_attention_heads=2,
rope_theta=10000,
image_token_id=2,
adapter_bias=False,
spatial_merge_size=2,
add_pre_mm_projector_layer_norm=True,
mm_projector_id="patch_merge",
),
)
model = Transformer(args).to("cuda", dtype=torch.float32)
tokenizer = DebugTokenizer()
encoded = [tokenizer.encode(s, bos=True) for s in sequences]
toks, all_logprobs_old = generate(encoded, model, images=images, temperature=0.0, max_tokens=7)
# concat generated and prompt
encoded = [e + t for e, t in zip(encoded, toks)]
generated, all_logprobs_new = generate(encoded, model, images=images, temperature=0.0, max_tokens=0)
assert generated == []
# Verify that logprobs are the same
assert len(sequences) == len(all_logprobs_old) == len(all_logprobs_new)
for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):
assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}"
print("All tests passed.")
def test_generation_mamba() -> None:
torch.manual_seed(42)
sequences = ["1 2 3 4 5 6 7"]
args = MambaArgs(
dim=512,
n_layers=1,
n_groups=1,
rms_norm=True,
residual_in_fp32=True,
fused_add_norm=True,
pad_vocab_size_multiple=1,
tie_embeddings=False,
vocab_size=32768,
)
model = Mamba(args).to("cuda", dtype=torch.float32)
tokenizer = DebugTokenizer()
encoded = [tokenizer.encode(s, bos=True) for s in sequences]
toks, all_logprobs_old = generate_mamba(encoded, model, temperature=0.0, max_tokens=7)
assert len(toks[0]) == 7
assert toks == [[25574, 14821, 11843, 23698, 12735, 23522, 27542]]
def test_chunks_transformer() -> None:
torch.manual_seed(42)
sequences = [
" ".join([str(i) for i in range(7)]),
" ".join([str(i) for i in range(9, 0, -1)]),
]
args = TransformerArgs(
dim=512,
n_layers=1,
head_dim=128,
hidden_dim=2048,
n_heads=4,
n_kv_heads=2,
norm_eps=1e-5,
vocab_size=32_000,
max_batch_size=3,
)
model = Transformer(args).to("cuda", dtype=torch.float32)
tokenizer = DebugTokenizer()
encoded = [tokenizer.encode(s, bos=True) for s in sequences]
toks, all_logprobs_old = generate(encoded, model, temperature=0.0, max_tokens=8)
# concat generated and prompt
encoded = [e + t for e, t in zip(encoded, toks)]
generated, all_logprobs_new = generate(encoded, model, temperature=0.0, max_tokens=0, chunk_size=5)
assert len(generated) == 0
for lp_old, lp_new in zip(all_logprobs_old, all_logprobs_new):
assert all([abs(x - y) < 5e-4 for x, y in zip(lp_old, lp_new)]), f"\n{lp_old}\n{lp_new}"
================================================
FILE: tutorials/classifier.ipynb
================================================
{
"cells": [
{
"cell_type": "markdown",
"id": "0f47fc4b-7cdc-48ce-b42c-e0b73d902ece",
"metadata": {},
"source": [
"# Train a classifier with Mistral 7B"
]
},
{
"cell_type": "markdown",
"id": "bfd6bdfb-c8cc-4fb6-8969-153a97dfb576",
"metadata": {},
"source": [
"In this tutorial, we will see how we can leverage Mistral 7B to train a classifier. We need:\n",
"- The mistral codebase: `https://github.com/mistralai/mistral-inference`\n",
"- The pretrained model and its tokenizer: `https://docs.mistral.ai/llm/mistral-v0.1`\n",
"- A dataset to train our classifier\n",
"\n",
"We will train and evaluate the classifier on `Symptom2Disease`, a public classification dataset from Kaggle provided in a CSV format: https://www.kaggle.com/datasets/niyarrbarman/symptom2disease/"
]
},
{
"cell_type": "markdown",
"id": "225f88da-96ab-4262-88f8-8f1d6d5224bd",
"metadata": {},
"source": [
"## Set paths"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ebe4bc9b-d0f7-42a6-9e41-bb3d042e0e07",
"metadata": {},
"outputs": [],
"source": [
"from pathlib import Path\n",
"\n",
"code_path = \"/codebase_path/mistral-inference\" # codebase\n",
"data_path = Path(\"/dataset_path/Symptom2Disease.csv\") # dataset downloaded from Kaggle\n",
"model_path = Path(\"/model_path/\") # model and tokenizer location"
]
},
{
"cell_type": "markdown",
"id": "3fcc420b-2ac7-42e1-a3b1-8a80f87ccc10",
"metadata": {},
"source": [
"## Imports"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "5e8227fe-d2fb-4a2f-a136-96ee2d32287f",
"metadata": {},
"outputs": [],
"source": [
"import csv\n",
"import tqdm\n",
"import torch\n",
"import numpy as np\n",
"\n",
"import sys\n",
"sys.path.append(code_path) # append the path where mistral-inference was cloned\n",
"\n",
"from mistral.model import Transformer\n",
"from mistral_common.tokens.tokenizers.mistral import MistralTokenizer"
]
},
{
"cell_type": "markdown",
"id": "3b9c8408-1bc1-4e79-8b6d-391e07d78eb4",
"metadata": {},
"source": [
"## Load the model and tokenizer"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "806ff308-5909-46f7-b10e-8b1f1c7b7edc",
"metadata": {},
"outputs": [],
"source": [
"model = Transformer.from_folder(model_path, dtype=torch.bfloat16)\n",
"mistral_tokenizer = MistralTokenizer.from_file(str(model_path / \"tokenizer.model\"))\n",
"tokenizer = mistral_tokenizer.instruct_tokenizer.tokenizer"
]
},
{
"cell_type": "markdown",
"id": "4a9aa34b-cbe3-49a6-948b-17f6c84d93c6",
"metadata": {},
"source": [
"The tokenizer is a critical component of the model that segments input text into word-pieces.\n",
"\n",
"For instance, the sentence `\"Roasted barramundi fish\"` is encoded as `['▁Ro', 'asted', '▁barr', 'am', 'und', 'i', '▁fish']`.\n",
"\n",
"The number of subwords in the model is set to `32000`, and each word is decomposed into known word pieces to avoid unknown words."
]
},
{
"cell_type": "markdown",
"id": "ce0e03ef-439f-4078-a848-d69bf4e339ec",
"metadata": {},
"source": [
"## Load the dataset"
]
},
{
"cell_type": "markdown",
"id": "65d1f696-138d-467d-bb53-032e08d47bf1",
"metadata": {},
"source": [
"We reload the Kaggle dataset from the disk. Each sample is composed of a sentence and a label. The dataset contains 24 different labels."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "cc2bf18b-d99c-4c31-9988-1675d019e04b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Reloaded 1200 samples with 24 labels.\n"
]
}
],
"source": [
"data = [] # list of (text, label)\n",
"with open(data_path, newline='') as csvfile:\n",
" reader = csv.reader(csvfile)\n",
" for i, row in enumerate(reader):\n",
" if i == 0: # skip csv header\n",
" continue\n",
" data.append((row[2], row[1]))\n",
"\n",
"# label text to label ID\n",
"labels = sorted({x[1] for x in data})\n",
"txt_to_label = {x: i for i, x in enumerate(labels)}\n",
"print(f\"Reloaded {len(data)} samples with {len(labels)} labels.\")\n",
"\n",
"# integer class for each datapoint\n",
"data_class = [txt_to_label[x[1]] for x in data]"
]
},
{
"cell_type": "markdown",
"id": "c5434265-fad8-4daa-b54b-438d420bcb4d",
"metadata": {},
"source": [
"The task is to classify a symptom, for instance `\"I have a dry cough that never stops.\"` to one of the 24 disease labels (`Acne, Arthritis, Bronchial Asthma, Cervical spondylosis, Chicken pox, Common Cold, etc.`)."
]
},
{
"cell_type": "markdown",
"id": "1540b537-c3cc-4851-80e3-91a9b9e8cc7c",
"metadata": {},
"source": [
"## Embed data points in the dataset"
]
},
{
"cell_type": "markdown",
"id": "c73d5d75-55eb-4210-8ffe-d0db3e9f8735",
"metadata": {},
"source": [
"We will now learn a linear classifier on frozen features provided by Mistral 7B.\n",
"In particular, each sentence in the dataset will be tokenized, and provided to the model.\n",
"\n",
"If the input sentence is composed of `N` tokens, the model output will be a list of `N` vectors of dimension `d=4096`, where `d` is the dimensionality of the model.\n",
"These vectors are then averaged along the dimension 0 to get a vector of size `d`.\n",
"\n",
"Finally, the vectors of all sentences in the dataset are concatenated into a matrix of shape `(D, d)` where `D` is the number of samples in the dataset (in particular, D=1200)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "78019021-01d7-499f-94ea-a3c88bbcb42c",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"1200it [00:25, 46.65it/s]\n"
]
}
],
"source": [
"with torch.no_grad():\n",
" featurized_x = []\n",
" # compute an embedding for each sentence\n",
" for i, (x, y) in tqdm.tqdm(enumerate(data)):\n",
" tokens = tokenizer.encode(x, bos=True)\n",
" tensor = torch.tensor(tokens).to(model.device)\n",
" features = model.forward_partial(tensor, [len(tokens)]) # (n_tokens, model_dim)\n",
" featurized_x.append(features.float().mean(0).cpu().detach().numpy())\n",
"\n",
"# concatenate sentence embeddings\n",
"X = np.concatenate([x[None] for x in featurized_x], axis=0) # (n_points, model_dim)"
]
},
{
"cell_type": "markdown",
"id": "7a997101-7844-4928-bdf4-d7a3051eca75",
"metadata": {},
"source": [
"## Plot t-SNE embeddings"
]
},
{
"cell_type": "markdown",
"id": "61bbc6b4-dfc5-42e0-b99b-1ed281d929cb",
"metadata": {},
"source": [
"Now that we have one vector/embedding for each sample in our dataset, we can visualize them using t-SNE.\n",
"t-SNE is a powerful tool for reducing the dimensionality of high-dimensional data, while preserving the underlying structure and relationships between the data points, which can help uncover patterns and insights that would be difficult to discern in the high-dimensional space.\n",
"\n",
"In the graph below, we assign a different color to each class in the dataset. It is apparent that several distinct clusters are visible in the data."
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "8e3cd894-ac30-4f34-a4c8-ef34293b587d",
"metadata": {},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAi8AAAGdCAYAAADaPpOnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/OQEPoAAAACXBIWXMAAA9hAAAPYQGoP6dpAAEAAElEQVR4nOyddXhcVfrHP+fekbg2TdI2dTcqlFKglBaKS6HI4ostC8V3F1tBFn4sK8guLgss7lasUKe01N09tUjjNnbP74+bNEkzcmcysXI+zzNP03vPPedMMnPP977nFSGllCgUCoVCoVB0ELS2noBCoVAoFApFOCjxolAoFAqFokOhxItCoVAoFIoOhRIvCoVCoVAoOhRKvCgUCoVCoehQKPGiUCgUCoWiQ6HEi0KhUCgUig6FEi8KhUKhUCg6FLa2nkBzMQyDffv2kZiYiBCiraejUCgUCoXCAlJKysvL6dKlC5oWni2lw4uXffv2kZOT09bTUCgUCoVCEQG5ubl069YtrGs6vHhJTEwEzDeflJTUxrNRKBQKhUJhhbKyMnJycg6t4+HQ4cVL3VZRUlKSEi8KhUKhUHQwInH5UA67CoVCoVAoOhRKvCgUCoVCoehQKPGiUCgUCoWiQ6HEi0KhUCgUig6FEi8KhUKhUCg6FEq8KBQKhUKh6FAo8aJQKBQKhaJDocSLQqFQKBSKDkWHT1KnUASiQlaQL/MB6Cw6kyAS2nhGCoVCoYgGSrwojjhqZA0LfQvJlbmNjncX3Rmnj8MpnG00M4VCoVBEA7VtpDii8EovM7wz2CP3NDmXK3OZ4Z2BT/raYGYKhUKhiBbK8qI4otgut1NCid9zEkkxxeyQO+hJT3bKnRQYBQghyBJZ5IgcdKG37oQVCoVCETZKvCiOKLb6toZss9q3mqUsxY0bgQAJm9lMHHGcYjuFFJHS8hNVKBQKRcQo8aI4oqimOmSbCioO/SyRja79xvsNXUQXSmUpNmz01/rTW/RG09QOq0KhULQXlHhRHFHEEUcllRFdK5F48LBL7jp0rNAoZAlLOIdzSNCCRytJKSmUheySu/DgIZFEuoquJIgE7MIe0ZwUCoVC0RQlXhRHFP30fhT4CqLapwcPX/q+5BIuCWiB8UgPc3xz2C/3IxCHLDrLWQ5AjshhuD6cdJEe1bkpFArFLxFlC1ccUfQSvUgn3fRliSIePGyRWwKen+ebxwF5AGi8FVVHrszlG+83HDAORHVeCoVC8UtEWV4URxS60Jlsm8xi32K2y+1R7XudsY7+Wn9qqGGrsZUiWYSGRqpIZa/cG/J6A4M5vjkk+ZIopxw7dnppvRigDSBOxEV1rgqFQnEkI6SUTR8TOxBlZWUkJydTWlpKUlJSW09H0Y7Y6dvJPGNeVPtMIYVSSg9ZVxpuEUWCQGDHzqm2U0kTadGapkKhULR7mrN+K8uL4ojFX6K65nJ4DpnmCJe66924me6djh07/bR+jBKjVHSTQqFQBEHdIRVHLFVUtfUUwsKDh/XGej7xfYLX8Lb1dBQKhaLdosSL4ogljrioO+62BlVURX27S6FQKI4klHhRHLH01fo2e1unrdgr92IYRltPQ6FQKNolSrwojlgyRSY5IqetpxERdXWYFAqFQtEUJV4URyxCCE7UT2SAGNAm20c55NBf64+OKvaoUCgU0USJF8URjS50xtrGcpF+EbHEturYueTSmc4M1gaHfa1AkEpqC8xKoVAoOj5KvCh+EcRoMYzWR7f6uIuNxQwUA8MWTj1FTxUurVAoFAFQd0fFL4beWm8yRWaz+xkihpBBhqW2btxUUMFwbbjl/lNI4Xjt+Einp1AoFEc8Kkmd4hfDZt9m8mRes/roIXow2jYaj/TwifcTXLhCXlMhK/DitZSNd4AYwBhtjLK6KBQKRRCUeFH8YlhhrGjW9XXi4xvPNxzkIAbWQpmTRBIGhqWw7QH6ADShhItCoVAEQ4kXxS+CCllhyUoSDIlkt9wd9nWa0MghBw0tqOBJIYVkkpszRYUiKFJKDC/o9o6XvFGhaIgSL4pfBDWyps3G/tr7NafbTme4NpyVxsqA7UbpoxBCLSqK6FO8XzLzFVj4MdSUQ1yK5PiL4eTrILGT+swpOh5KvCh+EcSJuDYb24ePOd45TNGngAarjdUYGIe2oZw4OVY/lm5atxYZ3zAMlhpL2Sq34sWsmZRAAqP10fTQerTImIr2Q94OyRMXQ1UpGD7zWFUJzHwVlnwBv/9QktpFCRhFx0KJF8UvAh0dG7ZDi3drU0klP/h+YLJtMgO1geTKXFzSRYJIoKvoii5aJpGdYRh85vuMCioaHa+ggrm+uQyRQ9okhFzRerzxu8bCpQ7DB6V58Oa9cNv/2mZuCkWkKM9AxRGPIQ1memdGLFwyaX54NUAeeWyVW3EIB320PgzWB9Nd695iwgVgkbGoiXBpyDpjHaVGaYuNr2hb9myQ7FrVVLjUISVsWgDvPygxjI5ZB0zxy0SJF8URzz65j0IKI74+jzycOKMylw2+DVHpxyrb5faQbZYaS1thJoq2IHedtXbz3oSvn27ZuSgU0USJF8URz05jZ7NrGzU3UqmOUkqRsnWecF0+l6Vw7mKpCkAeqdQENro1YcaLUFmirC+KjoESL4ojHjduSzlWQhEN64uO3moRRXvkHkvt2qJopaJ12L3WelufB5Z/1XJzUSiiiRIviiOeRJEYlQXahYuJ2kR6iB7EE08ssWSKTAYwAM3CV0kgyBE5zZ6HVaxsGQF0EV1aeCaKtsDwSVZ8Hd41H/4Vln6prC+K9o+KNlIc8fTT+rHBiI6vSaaW6VeAHCWPYrOxmVXGqqBWniH6kKjMIxRSSst+PqM1FW10JOKuBk+Yu50+D7x2J8QmSYZMUBY5RftFWV4URzwpIoUhWvNFQzLJOITD77kYEcNwfThTbFNIJBEwLS11Fh8dnQn6BNJFerPnYYVCWYgHT8h28cTj0Py/J0XHxhEHzgjSGwlg+pNRn45CEVWU5UXxi2CUNooEElhjrKGKKqDe18OqP4wVq0miSOQ823nskXvYY+zBwCBdpNNH6xNQ+LQEW42tltoN1Ya28EwUbYWmCcZdJJnzRnjXSQm718DBPZL0bsr6omifKPGi+EUghGCAPoD+Wn9KKcWHjySSANOxtUbW4MPHamM1PnyHBE1dFtwBYgB9RB9LY2lCo7voTnete4u9n1DUCbRQ9NGsvSdFx2TyjTDvrcB5XoJRXR79+SgU0UKJF8UvCiEEKaQ0OtZL9Dr0c2+tN5uNzewyduHFS5pIY4A2gGyR3aHqDsUQc0h4BcKBA5tQt4AjmZRMQUYPSZ413+1DaDqkZrfMnBSKaKDuXApFA+JEHCP0EYzQR7T1VJpFH60P23zbAp4XCPpp/VpxRoq2YvTZ8M0zIEOn/AFM4TLyDIhP6ThiXfHLQznsKhRHIJkik66iq98QcYEghhgGa4PbYGaK1uaES8ERC1ayBWg6xKfAlHtaelYKRfNQ4kWhiAKGNFjrW8vX3q/52vs1K30r8RptUwQSzO2xk/ST6Cv6NhEwnejE6bbTiRWxbTQ7RWuS3Flwy+sQlxi8XZ3F5e7PIE1VmVa0c4RsrVzlLURZWRnJycmUlpaSlJTU1tNR/ALZb+xnpm9mk1T8AsEEfUKbOu4CVMtq9sv9hyKfUkVqm85H0Ta4qiRLv4DNP4O7CpI6w6gzoFN30zk3NVttFSlal+as30q8KBTNoNKo5GPfx0HbnKOfQ6qmBINCoVA0pDnrd6ttG/3tb39DCMEdd9xx6FhNTQ3Tpk0jPT2dhIQEpk6dSl5eXmtNSaFoNkuMJVFpo1C0BLnrJG/8TvLARMnvRkjuHyd56x5J7roO/cyqULROtNGSJUt48cUXGT58eKPjd955J1999RUffvghycnJ3HLLLVxwwQUsWLCgNaalUDSbfXJfyDb5Mr8VZqJoL1SVSZZ8Dge2mRluR54OPYa37naMzyt5535YdJhRsKYcFn5kvi55WHLi5WqbSNExaXHxUlFRweWXX87LL7/MI488cuh4aWkpr776Ku+88w6TJk0C4LXXXmPQoEEsWrSIY489tqWnplA0m8P9XCJtozgyWPy55O37wOsGXTez1X7/Igw8QXLdf2DfJvjpAyjcDQlpMOY8GH4y6PboioivnoJFnwRv8/5foLJEUrgLfD7oNQKOmQKxiUrQKNo/LS5epk2bxllnncUpp5zSSLwsW7YMj8fDKaeccujYwIED6d69OwsXLgwoXlwuFy5XfbWxsrKylpu8QhGCGGJCZrO1Y2+l2Sjako0LJG/8DuryAvoaBJtt+gn+ehqU5ZtRPYYPhAarZkD3YXDL65L4FEFNhWThR7DoIyg/CGld4bhLYMy5YHcGFhVSSrb8bGbT3b0GDu6tn0cwpj9hzgdgyefw2d/hhmclg09UAkbRvmlR8fLee++xfPlylixpuud/4MABHA4HKSkpjY5nZmZy4MCBgH0+9thjPPTQQ9GeqkIREQO1gSw3lgdt01f0baXZKNqSr/8DQpjWlsORhilcoD5Vf13SuD3r4aWb4Lw/SP73ByjYVXcRlBXCjhWw4D249X+SmPimokJKySePwaxX64VRODRs766C52+A+76QdBmgBIyi/dJiDru5ubncfvvtvP3228TExESt3/vuu4/S0tJDr9zc3Kj1rVCEy2Ax+FAVaX/EEstIbWQrzkjRFlSWSLYtsZ7FtiGGD7Yuhn9dBAU7MS0mtQKorr9dq+Gjv4LP01QZLfnCFC51fTUXwwtPXQ4VRcqpV9F+aTHxsmzZMvLz8xk1ahQ2mw2bzcbcuXP597//jc1mIzMzE7fbTUlJSaPr8vLyyMrKCtiv0+kkKSmp0UuhaCs0TeNc/VxyRE6TZHDZZDNFn4JNU1U4jnTc1upgRow0YOGHcNtAuH+c5O37JDtXSaSUzHzFtPhEk8pieO766PapUESTFsvzUl5ezq5duxodu+aaaxg4cCD33HMPOTk5ZGRk8O677zJ16lQANm3axMCBA4P6vByOyvOiaC94DS8FFCCRZJCBXevYvi4bdxqs2CgRAiaNEXROUwm5A+F1S+4ZAzUVrTtuVl84sLXl+u81Cqb9VznxKlqG5qzfLfZImJiYyNChQxsdi4+PJz09/dDx6667jrvuuou0tDSSkpK49dZbGTdunIo0UnRIbJqNbNpfKd6qGklRGSTFQ1K8QErJ7gNQUiHpnCrI7tR4Ydq1X3L3v70cOFh/7PmPoU83H0/dpZOUoETM4dgcgmMvlMx5vXXHbUnhArBzBbxwA9zxruxQVdUVRz5tas9+8skn0TSNqVOn4nK5OO2003juuefackoKxRHD/kLJq5/7mLVE4qv1nejZBaprIK+ovl33LLjnap2hfTTyiyQ3POLF5Wna37Y9cNmffHz0OMQ4lYBpiJSSU66j1cVLSyMlbF1ivvod09azUSjqUeUBFIojkD35kpse81JZzSHhEooHb9T5aZXBjEXBbwkXTBLc/ivlxyNLd2DsnIEsWAWGDxnflT/ffjflxQ4slXDuIGg2szL1JQ8eOe9J0T7oEOUBFApF6/H0uz4qwhAuAI+84mPm4lDPMpJPF3j5U+U2fvSU4OvYzz4RYxxYgu/nvyHzV5jhOUhE5V5OmvCVpetFR7rzSrNwo0LRnuhIXyGFQmGBvIOSxeskRphhu16fFbEjkC6dFZ4K/q96F3+p2o47kvjgDox0lWGseQ2Qh8VGS06aNJOcHrsIliHu6n/BQ7MhJqGlZxodpITMXm09C4WiMUq8KBRHGHvyW9gaoklk7Z1jpa+CN1yBk0oeich9P4H0n1DFZvdy++/+wUmnLcLmaHwutQv85gU4ZoogvZvgtjfB5myFCUeBcRe29QwUisaojWuFIoq4pIEEYsLcFyirlKzaLPH4YEB3QdfO4fkXuDySr380+Hyuwb6CsC5tRGIclIfIWaINqt9DkMDX7kKucGYSK/TIB+5AyLJdQc/b7V7On/I6U/8zlvwdGqX5kJRhhjU3jNjpMVww4jTJ0i9aesaRUZct+Px7ISVL+bso2hdKvCgUzURKyfvufD5x5VNRW4QxFo0zHelc68wOGmLq9kie+9DHl/Ml3gYP88cMEdx9lU5GqsAwJNv3QmW1JDEOtu6BGpekRxfB8L6Cahfc9aSXDTua/14mj4Uv5tFoLg3eKQhwntG4SrYLyVZfNcNsHWQfpLlotsB1AA4hEJogq68gK0h1iHEX0W7FS1ZfOONWGH2WEi6K9ocSLwpFM3mwagdLfI09Gqsx+NhdwBpvBU/G9/MrYKSUPPiSj59Wyybr4LINkml/93LlmRpvf2Owv9D/2DmZ0CmVqAgXAVxyqo2zT5Dc9i8fFYdbYOwGzmty0bLczR+sAyM6DUPu/zlICw06DUFYsL4NPE6QkiUpaQc7b0KD7kPhsv+D2CRI64LK7aJotyjxolA0g3me4ibCpSGbjWo+cOdziTOzybkVmyQLVvl/evcZkH8Q/vlmcGfY3DzzFQ3+cJVOVrqAdMH0JwU/rjL44mcvyzzl6IPKsY0sR9iazteBoI8eG51JdABE5ijYkg6u4gDFjAz0nqda7u/mV+Bv50WnLpHQIquvJATEJUP/cZDVx0y6p1C0Z5TDrkLRDN6uCa0cPnf5d0L5dqGBHuQbGE2327qlSDtsTRICjhkKT/9O56wTNLxeicttjjx+hM4/bnQy+poSnGPK/AqXFFHNOc4CZnu/5Vvvt6zzraNG1kRx5u0PodnQR98BzpS6Iw3+1dCGXIVIG2C5v66DBHe8E9lcYhIguVYXO2Jh2CSwOayFYg+dVP+zlGY9o+9fhLuGwZaQIfMKRduiLC8KRTPIk6G3UErx/0hdUBxeHpbmcP0UjcR4yC+C1CSYeLRGSqIpZoQQLFlncOcTXlZslEigWyZceLLGuSdq3BPbg7srt5In3YcElQb01QsY5diHQHAQCRLyZT5rjDVMtk0mXaS3zptrA0R8Jvrxf0XmLUMWrAbDA4k5aN3GI2JSw+6vz9GCiddIZr8Wxhw06DEcbntTYPgkQoOHJ4PPG9z6ktwZRp4ZOBuwzwv/vgIeWyxJSFEWGEX7RIkXhaIZaAhC2UgC3f4zUkHTJIbR8gvE8SM0enXxP86nc3w89Y6BptW/kz158NQ7ZmHGB35j45mE/sz0FPO9u4hS6aWPXkVP+z4A5GHv34OHH7w/cIHtAuyiYxenDIbQ7Ygux0KX6NRiO/9ecFfDgvdA0wFJ0Fw90jAz3wJoumDrEkm+Bd+nO96BF38bvI3hg88fh8sfszx9haJVUdtGCkUzGKTHhWzTXcQ0OeaVkrKRha0iXAb0IKBw2V8oefpdc4X0t1DOXS759idJnNA5VaQzZUNfTpk/AMeiWKrLmr4vMMWMCxc7ZBS8iH9B6DbBZY8KHpwFZ94KJ1wOXQbgV/0KAUMnwojT6o9ZLdK4coa1tmtmWetPoWgLlOVFoWgGv3F24aaqzUHbXBOT1eTYSzV7Wd3rIPogB76NCSAPX6Ek0aqPc8elgZ9RvpxnBI36FQI+me0jNgb+9ZYZgaRr4JP9EB/0ZchJmxhz/io0rWkH+4399Nf6R+U9/JLI6CE441bzZ59XMuMFmP266ZMCEJcCE66EM6aZFpc6nPHW+v/879baeX/ZQWWKdo4SLwpFM+hhi+XWmG78p2aP3/OXOTIZY09udKzE8PC156CZM+WqPbi/yMS7OBV8DcVK9Cwy//1C8s87/J/bkhu8jICUsH0vPPSS79CMTD8dgZSCtbMGAjB26sqm10bV5fiXiW4TnHELnHqj5MB2QEJmb//RQJ1yojt2Wtfo9qdQRBO1baRQNJMzHOm8ET+IE2zJpAobyULnaD2B5+P7c4Ufq8tSbzl1ekHYJc6pB4j7y2ZEVg2I6C/4S9ZLzrvLw8Mve1m7rbFSiXGY1pVgSFmbk83vWcG6OQP8biFliIyI56xojG4XdB0g6DpQ+BUuBbskz/w6umOefUd0+1MooomyvCgUUSBDd3B/XE9LbWtoauoQ8T5ksd3P9lF0KKmA2UslM5f4uOlCya9ONVP5H3+UxrwVwROMhCocLaVgx4ocBk/YcuiYjk5fLUhqWUVU+eppcIUo6xAOg8bD8FNUpJGi/aIsLwpFK9ND8+/o2tLfRqNWhDz/kcGqzaaAmni0oHNq0/wv4SCExFVlViEUCDQ0TtJPwik6SNXBDo6rSrLsq8iS0x2OIxZOvh5ueV0JF0X7RokXhaKVGarHk0XTEGJ9UDn4cXyNNroGH80yVzqnQ/Dk72zozaipKA2N1PQaYmQs/bX+nGM7h65avcPEAcPFNl815dLb3Kkr/FBRDEYUfrVT7oEn1wouuE8JF0X7R20bKRStjAuJXdM4fPfIPqEI34pkohlp5A+fASs3mSJpxz7Ju9/68DRj8dN1mP/2Mcx6w0xud8FEjXNOlKyQ5fzPtZ/thplxVwPG21K4NiabDM0RhXeiAIhPjrwsQEM+exwqSyTn/UHVNFK0f5TlRaFoZb5yF7LHcDU5rnetwXn5XtBAawHH3YYIYP4Kg+se9vJ9hKng65Y3wwC315Rce/Lg3+8Z3PB0DQ+U72SHUV8qwADme0u4vXILBYaKw40WMQmCoybXJrZrJt+/CD++2/x+FIqWRokXhaKV+cp9MGAQsW1EGfH3b2HIaVUcPVhw7DDBKcdE9ylY12BYP3joZR8+I3gW16DUTquhQ6+sfe3YrOOZm97kfRpAmfTyes3+CAdV+OOs282aRtHgu+fBMFSYu6J9o7aNFEckbmkwy1PMDHcRLgx6a7Fc5swkW28dJ1KflJRJL14MDhge7LWVl20IDoSqh5TipdNpRTwYnwKA2yPZV+Bl4856p9tmzc2AtCSBz9e8zoJGIUnw/piKfWJhkyKBBjDPW8LNshvxIgrmgtbE54EN82H1DKgoMqsijjgD+o2NjukjQroMENzxjuSpy8wSA82heB/kbYPsftGZm0LREijxojji2Oer4fbKLVQ2cCrZYdQw01vMVEcG18V0abGxy6WX9135fOMupPowu0MiOhc4M3ACTTeN6tGAeK3+q+mwC/51p41n3vfx3SKJN3hkc0B0zRQuv7tCZ8Eqo9lCSNOCWW0EstyOrNARSU0n7AMKDQ/xzfEUbm2qy+Gde+BAfUg4BTtg6yLI7g9XPRk980cE9BguuOKJEv57U3LQdsEyKtfh9URxYrX4PJJVP8DWxeYc+o2FYSebifgUinBR4kVxROGTsolwacjH7gIyhYOznZ2iPu48TwnP1ewJOHY5Pt5wHQj5pTOAE20pjY7FxQjuvtrGjVMlG3dKhICNOw1e/TzwKjRiAAzsIVi6XlJWCfFx0LuLICEOBNJCScnmI2yBR4g/3CTT3pn+r8BFgfZvhrfvhqufatUp1VHp2ktuyQwYtJ9hl57EmnfHITQDaZi/Y003Bcuka2HmK8H7csRCRo/ozm/PBsmz10BZAWg2c8dxzhtmhetfPyXpP1YJGEV4KPGiOKKY4TkYUDzU8ZbrQFTFy37DxZ8rt7Mv1HZQLaECe5LROdqW6P9cgmDsUPNGP2awoNpl8M63/t/vxh1w2liNHxb7KCwBrRR2HzAT1cXFNE+4JMVDWWWQBkKidatGxPlJyAcM0OPo1JEijkr2w+YFwdvsWWeaFfoe0zpzqqXStZfNBW8hMetUHfPbOWSPyGXdh0eTv74rus3GsIk6k66BboNh+VdQkuc/OknTYdyFEBMfPTGx+WfJM1eBr/aD3zCsuzQfnr4MugyQnHU7jDhNiRiFNZR4URxRfOcuCtmmDB9Fhoc0rWmulXCpkQb3Vm7joIyend2NRLcQqupym+n9A87NDY//r37LpuEWT3XtvpUmIvOjGTlAsCdfsnNfXa2jw5ACx8kHA15/pbNp2YR2za7V1trNernVxUtuyfdIDBrK0Zxjt5Fz7DYAdC2WYdm3odX6F13/rOTpK8DrAqPBjp7QzCrW5/wuenNbP0/y3HWhw7j3bYKXb4YL/yyZ+GslYBSh6WB2W4UiONUhrC51lERJbMzzFFMgPRZHtYbLQm9fzjc4/w9e/vtFZCPX+TwIEVlGmX0FksdvtdGltnyRptX/K4DfXig4c6Qdgfn/Os+WODTuje3ByACWpXZLKCeROgp2Qu7aFp1KQ2o8B6ny7COYHc1nVFNWU7/d1fMowX1fwnGX1FeiTusC5/4O7nofYhOjIx7WzZU8e214+Wc++isU5qpIJ0VolOVFcUTRVTjJDeoOa7LJW8VBw8sIWwL2Zvhe/OgpjbrvSJII/rX8bqHBP9+M0Gv3MPxaTSyws8bNnbbNdPtDDGPXZ7B2pc5+lxct00W/cdX06JLCxbZu/MqZyQJPCVXSoIvm5Hh7Ms6O5usC0G2Q9bazXoWrn2y5uTTA5S211M7tLWv0/849BZf+FS79K0gpo56U7uBeyYs3EtEX4/U74fcfRXU6iiMQJV4URxSXOTNZVFUWst1/XHsBMwLo1zHZnOFIj2i8GoyoO70O0eMDnvMZkpc+iY5wiRhNYmRXUyS9FFPB6sEVMNi0sHiBDcC66hKOsSXyx9ieXODs3LbzjQadephh0aV5odvuWWu2S85s0SlJKSmoXGaprU2PC3iuJbLp/vhO4y2pcNi5EiqKJAlpavtIEZgO+AikUDRFSsnnrgLurtoW1nXl+PhPzR4eq9rJqzX7+NCVT14Y2V97ajFEO9j3XHsnpJQcMFzs8FVTKetXgbXbJIXWHrZbDkNgP64YaPxgXfdznTFnibect10WFvuOwtS/WG9bWdJi06gjv+Jnymq2hGynCTvJMa2btGXtrMjLFUgJa2ZFdz6KIw9leVEcEXzhLuRF176Ir5/vLUXHXIBfc+3nTHs6N8V0Dek4e4YjnemewI6p4dJDOCmXXqZVbmZnbWp9G4IJthSuicmmrKIN86IICVJgO/EgWp+qkM0lMN1dyKXOzI65VXQ42f1h1Dmw/MvQbRMis+RZRUofeeWLLLXNTjoRvZUjuyK1ugAgwBX646X4hXME3FEUv3RqpMH/XAea3Y+PeqvBN56DvFITWgz10mO5wmluDzTHyC2AZKFzoj2FR2p2satBTSAvktneYu6s3EJcWtttGWlZLhyX7sVxTh5WdxqqMNjha2bK1/bEpOtADyIEhAa9RkFSdPMIHU6N9yBeI/QKn+TsQ+eE1o1+AjPgKuKEwxKy+kR1OoojECVeFB2epd4yy1FGVpHAl55Cio3QUUmXObO4N7YHaSEcbQ8nUejYEaQJGxc5OvP3uL68584/NH5DDOCg9LAg4wB9u2FZPDSHuiHuvkrjsscLSfjdduyjS1tl7JAYPti/FvauBE9NyOZRwxkPJ9/g/5zQQLfBpADno4i0FP0kiHd2bZMK0SdeEXnNrLSu0H9cdOejOPJQ20aKDk+pDJX2LTIMYJG3zJIz7xhbIk/K8Kwi59o7cXlMfb6TL92FeIO4/xrALG8xf76sC3f/y8AwolPrKBDZGXDLRTrHj9BY5o3n8whM+TFo9NRjojcpw4D5/4FdCxs7VXQeCCffC47AjqlRY8wUcMTAnNfM+kZ1ZPWF028z/21hYuxpaMKBETQxoiTB0a3F5+KPrgMFFz8g+eDB8K7T7XD1v0DT2oNCVrRnlHhRdHg6iZbZz9eAKouCZLG3DFeYcUcjbAmN/r/PcKFhbl8Fwo0ks5eH//zBwbMf+lgbnn+yZTQB6clw/AjTODtST6Sb5mSfYSULjYkAznKkExPN4otf3QtFO5sez98IH0+DC58HexTFUiCOOh2GTTaz6tZUQEo2dO7V8uPWogk7GfGjyKv4Gf/xyAKnLZUEZ5Tz/IfBhCsFOYMlHzwEuetCt+82CC7/G3QfqoSLIjRq20jR4RltSyS5BaoTG0BXzVoV6jLpC8vnJVXYGHxYSLQmgwuXOv5SuZ19XUs46vZ8hp4TOiw8EgwJa7bCpl3mwqgJwYNxvUgVtkbv098NpO7YUXpCdDPpbpntX7jU4a6ERSEK90QTTYfuw6H/ca0qXOrITh5PgiPHzxmBrsXQO31qm2wZNaT3aME9n8N5fzB/XUJr7Atjj4GjToM73oX7pgslXBSWUZYXRYfHJgRXO7P5d82eqPVpOtDaGGNLstS+s7BbtrvYEDwa27vJwlJj0aZxAA9PuHIB0I4TMDsWKmw0z2W4KZoGi9cZDOhhrjZdNCcvJgzkB08RczwlVEkf3bUYJtlT2W+4mOkppkz6yNYcnOFIZ7wtxVKZA8us/Sx0m12LYPwt0RuzHaMJO30zLqWoag2FFStweYvRNSdpcUPJSBiNXU8I3UkrIITg1N/CuAslS76Aon2QkAZjzoX0bkqsKCJDiRfFEcE6b0XU+qqzHNwVm3No8a2WPn72lFEhfQzU4+hra+xbMdqWRLLQKQ2xzXScLZkbnNlk6k0tOku95WHP1bBLYm7YTc2LPaCq7pE2OguCAHyHvZ04oXOuI4NzHRlN2p/f0snoqopDt/FZz9FzJKAJnU7xI+gUP6KtpxKSxE6CSde29SwURwpKvCg6NFJK3nIdYKa3JGp9Dq3d7hhii8eQkieqdzPbW9LIspIqbPwlticDbObWj00IpsV04/+qdzUpF6BhCoFz7J3I0ByU48Nf7tWKMB1+69C7uoi7ZxuuzzLxrUiJqA9/+AwY2LMdPRnrDvCGiixqR/NVKBQthhIvig7NdM9B3q0NL24OnYWdv8T0IE13kNKg2vSfqraz0tfUqlMsvfyuaiv/ju9H79rU6yfYU3gQjf+69rHbqK+v5ESjGoPPPYUASBcM1OO4L7YHGQ2Sh2VpDnYaNRGVGxDxPpwX76dqQyLUNN//R9MkGSmCMUPakRjofgxs+SF4m/TW9z1RKBStj3LYVXRYvFLyThSS0wHkSw87pKuRcNnsrfQrXOowgH9W7W507Bh7Es/HD+DZ+P48ENuTDOyHqkRL6i0ym31V3F25rVE005mO9ObVSRJgG1dEc8tECs3A7vDxyE029PYUsjr6cgjlmH2M2pdQKH4JKPGi6LBs8lWG9DEJh48Os+BYydq7U7qoPiwXuhCCXnosBdJDAR6/brgGkC/dfO+uzxNyqj2NQXpcRBsfngWpVD3UH+/sDMLbOjFI7FROQno5us1HTEINQ07azHn3f0Vm95aJZIoYZzyc+Yj/DLdCgxNugc79W39eCoWi1VHbRooOSzhZdTsLO4XSv5CoY5fh4oDPRVatM22+hey6AHmGm55abKNjXil5qya4+JHAD55iznOazq92ofFoXG/ecB3gG3chVl1PPfPTcH8eWUiy0KDvMTsZddbaJucKZAHJIjmifluMTn3gsv/B5u9hxwIzUV3WEBg+FWytW79HoVC0HUq8KDos3SzmYBmtJ5Am7Mzyho5WublyE4/F9WGALZ54oVnagUnVmn6N/lu9lzILWVvKDssOHCN0bozpylXOLHb6aljpLedjdwFVAWSXrNFwfx15lI80BP3G7vB7TjTD+bXIKGKP3IOGRm/RmzgtiplvNQ0Gnma+FArFLxIlXhQdlizNyUg9gVW+Cr9LuwAyhJ2H43ozy1PM9xbESw2Su6q28khcb85zZPD3mt1B26dhI7mBnwxAmeHhc2/oStMCyA5Q7TdW6AyyxTPIFs/Zzk68UrOP2Z6SQ+UDbIAX8K5LBE/kImPoyRtJ7FTp91ymyKRK+qg2PNRQhCF8JItkkkTg3DclRgk/+H6givpaAstZTidfJybrk7Ef9rtSKBSKSFDiRdGhmRbbjbsqt1ApfY3sHBpmMrh7Y3sghGC8PYVna/ZaSgQngUerdvJBwhBeRqc4iAXlupguTY595C6w5DIrwVLdpERh487Y7lwf04Vcnwu7EPTSYvFi8K7L4H9a+EXwhM3HqDPWctSp65ueQ5BIFo9U7ccldjLYlo9T1P8OskQWx+rHNhExlUYl033TMfz8jgsp5DPfZ1woLmzzrK8KhaLjoxx2FR2aLpqTp+P7M8meSl3iegGMsyXzZHw/BtbmYXEIjbMsCIU6qjBY5qvguYQBZAj/1oJrndlMdKQ2Ob7MYrK5XloMJ9hSLM8pUdgYbIunnx6HTQhihE63VD2i6r2xt24na/JuhFZf3LGuULFNJvFadTpCbGOkfX8j4QKQJ/P42vs1FbJxJNZC30K/wqWOaqrZaGwMf7IKhUJxGMryoujwZGoO7oztzs0x3SiTXhKETqyfkNqzHOl87C6w3O8Kbzlj7Em8njCIld5ypnsOUiMN+utxTHV2JiFA2O7hfiyBuDu2e7PT558wQhDjgBrLiWUlWu8qRFcPc9296aKV0dt2kHjhIYZYjtcGcX91BQ7hYaDNf/4cicSDh9W+1RxnO+7Q8f3sDzn6BmMDg/RBVierUCgUflHiRXHE4BQaGUEqTGdpTk6wJfOjt9RSf7HCNEwKIRhpT2KkvX6bxJCBN4aShY2DIQRMLBo99NigbSzN0Sm46UKNJ98JZX4x5yuSvMRcuaf2iGCvkcxedzI6cJo9nY048FJJf70ISeCga4lku9zOWDkWXehIKZEWNstcuEK2USgUilAo8aL4RXFHbA6FFW42yuqQbU+wpzT6f7Hh5pWa/Sz1llOODyeCifZULnB0pluDWkUn2VPZ4dofdCk/O4wtrFBMOUnHbhO89KmPkgY7VkLUbwXZYyVibBG20woQ9qYz8wET7Sl85ylCA2JF6DBxAwMXLuKIs+zHYkc57CoUiuajxIviF0Wc0PlXQj9urNjEHhnYCtBbi6FXrWWkrn7Se+78RoLEhWSGp4jZnmL+L64Pg2r9a05zpPGpu4BS6W3iAeLbHIfv20w+LIjlE83DiP6Cmy/SyO7UPPezs07QOO1YweqtktIKyO4E/XKgqkbgsMNBm5tbKwpxIZvMSQNG6AkM1uOZ6ykBoEaGFhkCgYN6S1cGGRQQfFuun9YvzHemUCgUTRFSBrF/dwDKyspITk6mtLSUpKTAIZwKRUPchsEtlZvYI5s6i2QIO0/H9ztUKuCNmv28H6R+ksAs1PhGwmA8GKz0VrDXcPGFu5AC6UHHtIBUvd0V38pk8LMhc8/VGmceb70mUbHh4R1XHlt8VTjRGG1L5DRHOsl+cs7UscVXxeNVu9gn3WjUlyuYYEvh9thuxAidld5y7q/aTqKo4ayYTUHes6CH6MGJthMbzKmYL31fBrzGjp2L9YvRtebXXurQHDwA370Ba340w8T6HAVTboLOOW09M4WiVWnO+q3Ei+IXi5SSHz2lfOjOp1h6SBY2znN04iR7KvZaf5ciw8NVFest5fI9yZbCIm9Zo3DsLOFguB7PntmJLPk8Iej1b/1VJyfTvwWmWvpY7i2n0vDxs7eMhb6mqft14O7YHow/bLurIVJKVvkq2O6rwSEEY2xJZDbINSOl5M7KLWw1qhllz6WPXoS/HSEbNs6yndUkA+8B4wCzfLPw0tjnJ554ztDPiG6yuo7I0h/gpfvwm/3wglvh9KtbfUoKRVuhxIsSL4oW4nNXAS+59kVc6lAD7FLD9cAAKquC+4VMGCV4+LeNLSduw+CJmlx+9JZYElACeCK+HwP0yEVCqeHlweodbPZVcpRtP/1sheii/jeQQgrH244nXQT229lp7GSPYWbY7af1I0PLiHg+RwwFe+CP5xM0bfPt/4Eh41ptSgpFW9Kc9Vv5vCgUQSiRXjSwkOjfPwbgKtVCCheAZRsaL2r7DRe3VWymMowaThL42JXP/XE9w5toA5I1G0/E9WWtr5KF3gwqfC6y9QoG6k4yRRqdRKeQDro9tZ701CKfwxHJZ88Tst7EB0/CQ0q8KBShUOJFoQhCumYPQzr4xzCsReIYDdY1tzS4p3JbWMKljoXeUqSUzcpkK4RgmC2BYbbgW12KMFi/MHSb/dtNBymVhVihCIoSLwpFEE60pfAS+w7VFIoEkewBmwHe4BFFfbrV/zzfU0KhtFbV+nB8+HMJDk5hvmTGl5KffwS3G3r3g9PPFQwdiUrnHy181pIX4nWD3VrRUYXil4oSLwpFEJI0G1c7s3jVFTh7bAo2SoPIG6GDfUwJnoVpQcf6zfn14maRt6lDrlW6CQdaGIJj41rJI/dJPO76GklFB2HxAsnZU+GqG48sAWNIyax9Pmbu9eIxJCPTdc7vZSfO1sLv0eaABgUrg7dTKBTBUOJFoQjBVGdnYoXOm64DlDbInJuEzuXOTMbakrimMnDNHg2YOLWaDdshN89/myknCYb3qw8hdsnIN6umODtbbltdLXnsTxK3qz6hHYBR6+Qz/WPoOwCOnxjxdNoVeyoMLvqhik2lBnVa5b/Swx+X1PDGxDgmZLfgLdFqbIS7BpzNz76sUBzJKPGiUFjgTEc6p9rT2OCrpFL66KI56a7HHDp/saOz31wwGuBE46r4LLIfsPHyZz6+nCeprDHPZ3eCa8/VOPXYxrlPeumxLPeVh+3xMkKP5zR7cAtPQ36cCVWVwdt88aHk+Ikdw/KSX23w4gY328oMusQJLu3rYFia+bt1+STnzahkd4UpIrwNtESFBy75oYq558QzIKWF8tAYFt2+O3YAqELRKijxolBYxFbrxOqPq5xZxAmd9115VDWQHH20WO6IzSGnVujcdKGNmy4MPdYZjjQ+CpIY73CcCM53ZHC5MyusYo8b1ko0jaCVqbdvAY/bwO5oP0Xo1xT5mL3PiyHh6Ayd4zN17lpUwxubG/sJvbDBw8BkQaxNsKPcoCRAAUsD02H6+fVunjquhaweiWlQXRG8jc0BMb/wXDgKhQWUeFEoooAQgoscGZyXl8fBTfMxPDU4O/eh07DTIIICjFmak9/GdOX5mr0ImgbYJqDxW2dXUjQbnTUH3RpYgcLFyoP+1s0waGjEQ0SN/GqDa+dW81OeD02YTsk+CSkOAgqTjaV1uYSD45Xw6U5Py4mXUy6Fdx4P3mbESS0ztkJxhKHEi0IRDarK4MO/4NizjmxNBwQY38GsV+Hce2Dg+LC7PMfRiWzNwUeufFb7zL2dVGFjsj2NK8O0sARiyFGC+TNDL+zffdH24sXlk5z3XRVby0wzUcPQ8kDCJVyqvfDqRjez9nnxGpKjM2xc0c9OdlwUrE7HnQPfvQkH9/k/74yDK+5r/jgKxS8AlWFX0f4pOwjzPoWfv4HyYkhMhZET4aQLYfF3sHsj2Oww7ATzuK2VKxdLCa/fCvs3+zFjCDNnx1VPQLchEQ/hkQZeJDFoUY38qak2uPLc0O0SEuG1T9p22+iDbW5++2NNi46hi3pRJAFNmMdeGh/LeT2j8LkqL4bn/wBbVzY+nt0Lbn8G0jKbP4ZC0UFQGXYVRy65m+Bfv4Wq8vpjVWXw7evmC0Bo5v7Boq8hPRvufK51i9ytnwP7AhUxrM24suBduOSRiIewC42WkGQxsRoxMQY1ITRBRTns3yPJ7tZ2jrsfbvegQbOTBgbDkI03mAxp6tHr51XTO0k75PwbMYmpcPcrsH8HbFoGSOg7ArqpatsKRTi0Hw88heJwvB749+1QFcLJURr1HqfF+fDETeBxtfz8wFzZvns2RBsDti42k4+1QwYOs9Zu984WnUZIilyyRYUL+PeMqTv2wvoofqaye5mWw5MuUsJFoYgAJV4U7ZcVs6G0ECvOlocwfFB0AJbNbLFpNSJ3LVSXWmgoW09Qhckxx1tr52jjpK99kjT0FjD8ZMcJeicG79gn4dtcixlyFQpFi6PEyy8Frwe2rIWNq6A6RGKP9sKWFeaWkFVi7ZDghHgnrJzdcvOqQ0qY+ZK1tjGJEBPfsvOJkOMmiJBuQjGxMNiihaaluLq/A59FHRtTIcjeZiN7q4240sDCJFaH78+MIzM29OfM09JmH4VCYZkWFS+PPfYYY8aMITExkc6dOzNlyhQ2bWrsG1BTU8O0adNIT08nISGBqVOnkpcXIA2pInwMA957AX51HNx8Htx6AVx0DDz7UAcRMRZWqzg7dEuGzgmQFgud4uDgalj8Scsm/Fo9A/YFzqzbiMEnhSfEWpH4BMGZU4LXAjz3IoEzpm0T1R2XqXNJb5vfmk2agDQn2NwwbG4Mx30Wx+CFMQxeFMO4L+MZ+X0MDj+VvV0GvL3Vy+gMPahVRxcwslMLJa9TKBRh06J307lz5zJt2jQWLVrE999/j8fj4dRTT6Wysn7RvPPOO/nyyy/58MMPmTt3Lvv27eOCCy5oyWn9cpASnrgPXv0nlBbVH3fVwBdvwd1Xgrt9bmUA0H90aPERa4dO8ebqBQ1WYAO+fx6WfNoyc5MSFn1ora3Q4KRrgve1ZwusWwR7t7ZJhtXLrhOcdKr5s66DVvsCOO1cmHp5q0+pCUIInjk+lj+PctLJWa804m1w4yAHy6ckcPmyBDrv1RGHSZyUfJ3RM2LRa92OkhzVnNl3Hb8+aiE1toVc2r8Eh+4hNaYKu9Z0e8gnzTEUCkX7oFVDpQsKCujcuTNz587lxBNPpLS0lIyMDN555x0uvNBMO7px40YGDRrEwoULOfbYY0P2qUKlg7B2Kdx5SeDzQsDNf4EpV7XenMLB64X7zq71ewlAlySwaYHNBvZYuON9cEQ58VhNJfxrirW24y6BSdc3Pe5xwawPYPYHUNSg8GO3/nDxXTDw6LCnJaWkwrWLgsrlVHvy0YWT1LhBpMcfhU0L/TvYvUMy9wdJSRGkdYKTJgu6dm/70gCF3lx2u9bgw0uqLZscfQSbyzS8BgxI0YizCRbMljz1f4FvZxLJ1lFuBpy1hiuGL0ETEkMKBBJdMzWjEOAzBOVuB3bNoMZr5+e9PUkTQ3lwZEbbFKisKoK9K8HnhrSekDEguJlMoeggdJhQ6dJS07ExLc2svbJs2TI8Hg+nnHLKoTYDBw6ke/fuAcWLy+XC5aq3FpSVRV5994jnmw/Mx2hfkJoqX73bfsWLzQa3/wcevRJ8fpwl7br5Coan2oz0GTwhypOzqPkTM2DC1Y2PGQZ88gzMfMf/+9q7BZ6aBrf9GwaPtT4jKckt+ZbCyhXQIKi4qnQ/eeWL6JdxObH2jKB9dO8luPKG9rMw1hiV/FT5AS5Zb60t9O1mC4sZGn8SOc763DlzZkiEZgZ3BaL3LsmvRyw+9H9N1P8d6/SArkmSnS6EgHiHhzP6bsAmNlPiO49UW3az35OUEi9uNDR0EcTZyOeGRf+FbXMav6nkbnDibaaQUSh+obSaeDEMgzvuuIPjjz+eoUPNVJ0HDhzA4XCQkpLSqG1mZiYHDhzw289jjz3GQw891NLTPTLYuzO4cJESDuS22nQiols/uOt5+McNTc9ZDT2pLI7unACc8ZDeHQ7mElTITLwW9AYLlM8LD18G+7cHvkZKwIB3H4eHP7b8lF1YuaJWuMDh2VC8RjXbCj9gSNZNiHbqe3M4hmEwv+IdvPjb2pSsdc3GqSXQ2d4DgOLi4MJFIIhxmblbtBC/0oa/ciEkPrwsqvqYDL0HPZ0jsAkHue61lBtF2HCQbe9LF3v/oGLEJ33sdK9kl3v1ITFmJ4ZkvTO9naNI0NKpNIoRCJK0DPS5T0PuUpp8vsr2wbcPwNmPQ1JW8DeiUByhtJp4mTZtGmvXruXHH39sVj/33Xcfd91116H/l5WVkZPTignJOgquGkhMImTVvYTk1ptTpPQbCUOOg/ULI/MHsbdAjK8QcOyF8NUTAc5rEJcEg05sfPzJm4MLlzqkhLzdsGMt9A4d5iOlJL/852AtcPtKKa3ZSkps/9DjtwCGIVm1DDatkwgBQ0cIBg8n4FbMbs+aAMKlng018+hsvxKAjM6QuyPwx10iSexUHlK4BKPAt4uCql2AKYZkrbA46Mtlm2spx8SfT5zW1PxtSB/Lqr7koG9Po+Meaij07aawanej4+lF1RyTuyTAGzHA64J1X8C430T+ZhSKDkyrPILdcsstTJ8+ndmzZ9OtW7dDx7OysnC73ZSUlDRqn5eXR1aW/ycKp9NJUlJSo1eHxTBMv5R538C6ZcFFhhWkhO8/gRvPgrOHwKLZwfvUNDjl/OaN2Vrc8Gj9Iu6wQWYCZCZau3bpFy3jBHvU6TDqbPNnrcFXSWimj83Fj5hVguv44R3YvDy8MQ7uD90G8BqVuHz1FqatBZks292L4qr6CsUCjXLXrvDGjxI7txnc9mvJ/90v+ew9+PRdePD3kt/fKMk/4P9vs9uzLmS/VbIUo/YzfvIZIujHXQADJq+PZPp+kYdZRGpkBcuqvsSfG+Eu95omwiUYnXP3YgSzuEkDts0LbmpSKI5gWtTyIqXk1ltv5dNPP2XOnDn06tWr0fnRo0djt9uZOXMmU6dOBWDTpk3s3r2bcePGteTU2p4FM+C5v0J+gyJt2Tkw7QEYOzH8/qSE/zwAX75tbZtB003LTHv1dzmcuET4wyuw7Dv4/t8gw0gYlrcV9qyDnChXFhQCTr8NBhwPy76EvG1gjzH9a0aeBQlp9W3LS+Cjp8IfIzEtdJsGfLjiWL5edzRuX932hSQrsZg7Jk6ne2oxYSX8iwCXS7JlA3g80L2XZNM6wWfvS3ZsqW/TcCdzzy5TxPzrZYiNbfy59UlrGYm9uHEQw6ixMOJoWLWsqVbVNOjeGwZM3OK/kyggkVQYxez1bMQr3fjwkqR1opMth13u1WH15XB5Qgtunxt8HrC1cfZAhaINaFHxMm3aNN555x0+//xzEhMTD/mxJCcnExsbS3JyMtdddx133XUXaWlpJCUlceuttzJu3DhLkUYdlrlfwyO3Nj2+Pxf+fAM8/BIcOym8PpfMNYULBL7pCWFaBQwfdOsFf3kW0juHN05bomlQsAmEDG8NFhpsXxZ98QLm77T30eYrGAu/DN+yltwJ+o2w1NSmxfPqwjOZtXkAjX85ggPlqdz/5RU8ds7/6JnWPbw5WMTnk3zyDnz5kaS6quGZ4H8ow4CCPHj4D5JhIyWVVVBWDEkpkHpcDnH9NoTU4jZM65amCf7wELz1suSHr8FTq310XXLcRMl1t+hs1/qy17OhidUkmqypMbM7120rOYnHRXg5lWrinKapKNg0HfGgq/BtxS+TFhUvzz//PAAnnXRSo+OvvfYav/71rwF48skn0TSNqVOn4nK5OO2003juuedaclpti8cDf7sz8Hkp4dmH4ZiTGm9FhOLzt0xrihHEQbdnfzj5PBg4AoYf0/HCLX1eWP198PfoF9n25vVt4T15A3DBraBb+4ruK5XM2tyfQ4UgGyEwJPxn7nmcM7Rlqha/9JRk1reRX791k/mqQ9fB9+VEcsb0ZOJdM7A5/P/NU7VstAbfE4dDcO00wZSrKpi/Zgv57j2k98kjNrmGjfSkp20ERd69VMuyFhUwUL+tFK5wAdjbPZPeW4JsMwkN+p3c8b7DCkWUaPFto1DExMTw7LPP8uyzIYrbHSncf62ZvyQYB3JhwwoYMjp4ux2b4NPXTavLwYLQC/S+3XDJjWFNt13hqjRN5eEiJXQbHP35WGXZTFg5x3r7mHgzz8u4syxf8sRsD01FS0MEe0pTKK4WpMUFaRYB2zY3T7j4o25rac/Snix8eTzjp81p0kagMTxucpPj1UY5y+R7OIa66NbgeL5vJ/nVO8mxD0HDxh7Penx4ojvxKFGRFM/OPl3puW1v05NCg7g0GHpu609MoWgndIyYySOFvTth5U/W2haGKJEw+0v47dkw42OzrRXLgqsadrXcnn+L44yzbIlohGYLva3TUuzdCi/fb91h+Owb4F8z4ITzwhpmx0Er/Qu25kfX2uDzST74n2wxA4CUgq1zBlFV3FhxxYlkxsdf6jeyZ0XVN0GjlHI966jwFTEp4RrS9C5NsvG2FzYM683GIb1w2xt+5gWlmaN5NPsvXPGTjdt+qmbefq+lB0WF4kiiVZPU/eL59iPrbdOCJBPbnwuP/y6y6KSFM6FHv/Cvaw/odhgyCdbODG/rKCW7Ptd9eTHM/Qjmf2o60TpioOdgmHo75LTA72Xme9bbXvdXGHtGRMMkWPTZTI+P3kK95CfJS09JSlogjU5DpCFI23gVAyfuwCvdpNu7Eaf5D/Gv8VVQauSH7POgkctuz1qKfPtpaSfmhmjoGFj87ArBjv457OzblZSiCuK8PXhzz0m8mpuITYBXerEJeGuLh16JgjNy7AxM0ZjS006CvX0KMoUiWijLS2tyYHfoNmBu+AfbMpr+TuT323efh+Ig6fbbOydcbkb0hJNorXNP89+83fDQJfDFi1CcD143VJXB+kXw10vhvX9GP6R69XxrQuvXD0YsXACuPib0c0iiE/pkROcrv3KJ5B8PtrxwqWPWtxr5a/rS1T44oHAB2O+1blnc6lpCawoXG05GxZ4Z9nVS0ziYnsRjeX15NddMD+CtnXbdvzvKJc+vd3PbTzUM/KCcj3e0z+0whSJaKPHSqlh8Ghp/pn9nXSnN1+rFETit1lJdWR+V1BFJ7QJXPQmde1u/5qjTzN/bC3eboSyBmPUe/BDl342Vv1O3/nDc2c0a5uQBtpBpb6aNj46hVUrJmy+37jbF5vXwyL2SP94uyc8z2LdHUlrcdA5erIfQt66/iyDHMYTdnjURXW1IjQV7egVtU/fbqPLCb+ZVM2tvGOkEFIoOhhIvrUVZMSydH7qdboNpf2587OfZcPvFcFo/87V7a+TzkNL0k+nIdO4F1z8P1z4Hp98O8Wn+LTFCg54joc8Y2LrK9D8J9aT99euhHarDofew+i0rf2g69B8VlaE+u95JVgABc82xNq4eG6SOThjs3Q27d7Ru8eu6HdKtm2DalXD7NZLrL5b85S6DdavqJ5Kqtb90+QJBnEikp+Mo8r07w7q27nf87tpRVLhjwhgT/rayHVeMVyiaifJ5aS2+/gAqSkO3e/AFOJgPr/zdFDtlJWb14YZUVTRvLmUlzbu+vZDdz3wNOB6++hdsbZAeX9Nh+Glw6s2miNm2ygwrDbXiVpbA7g2WUvJbYtIl5tZRIKQBEy6MylApcRqzb4tl/jYvb/zso8It6Z+hccdEG2lx0XtOKbPwMfbHoGGmBUXKZiSTPuzPt3EdPPQHye8fgGOOF6TbumLDgZcIotJaiCStM0fFTuaAJ/yHjsKqOD7aMJIfc/uEdZ0BLC30sb/KIDuKf3uFor2gxEtrMfuL0G269jSdSP/yG7NyXLCiihEjILNrC/TbhiSkwiWPQMkB2LfJ3HLLGQbxKfVtglk/DudwsdgcBh8LZ1wD37zWuM5UXU6ey++D7J7RGw8Y38fG+D4t99VOD16YugmaBseOhzv/pLFjq+Td/0pWBCjbEy51QXbP/kMyYgw4HBrDYiexojrKsdvNoNTI48fKd6076gIgeGnZsczb3RfZjGioco+k+XWwFYr2hxIvrUWlBWuJ2wX/vNu8I0eiWzTdvDaUdeGsX0XQeS011TB7OpQVQb+hMGJceMn0WpKULPPlj0FjrO1zCAHZwX0Lwub8adDnKJj5jpmsTmgw5Fg45TLoOyK6Y0WZ8jLJ7O9gxWKJzwf9B8HkswWDhsGmdaEtKJoGyalw1Y3mAtyrr+D+/xOUlki++0Ly4ZvRmWdVJSz+EU6YBFn2vozkdNZVz8VNdXQGaCbhCBeBzoH8c5m722LtrgA4NJTVRXHEosRLa9GjLxTsD+zAqem19vQwHQmcsWbfPp+ZNffsy+CtZ2DOl/4LvPQdAmdcEv78fV742+9g7leN+41PhN89DuNPC7/P1iRnAHTtW+v3EoTRp0BSevTHH36C+epAbFpvFlKsrqr/k29aB198JLnwcti2yXQPCiRg7A4YPwmGj4KtG6GqUpLT0xQxySmCCZOjJ150HfbtMbMLl5dJtiztg1Hdm/Tu+cT12USlLEKTOom2TmTaevNz1cctnmE3MgTHxJ3Lb3emENkTjIku4KLedhIPD5mWEgwPCFv7eehQKCJAiZfW4uzLYPGcwOcNHxQeCL9fZww8+1njY/f8A7r1hE9eg8py85jdAZMvgN/ca14TDlLCXb+C9Suanqssh4dvhgeehxNODX/+rYWUZmh0KC68reXn0gGoKK8VLtWNtWqdUPnoLbjp9/D9dNiyof58ZjZceCUMHg4L58Kn71GbfdfspO9AyW/vEvToJcjM1sjqYnCgQW3SSDEMcDrhtecMvvvS1Nomncnp0Zlb7xX06lu/kPd1jmWLa1HzB446knU1c6jyNC97bpId/jiyQfKfigJY8DwcWMchx6GkrjD2GugyvFljKRRtgZLercXYiTCxeeGwfuk7pOkx3QZX3Q4f/AxPfwhPvGf+fOejpqUkHHxemP6uf+HSkKf+2AwvzFZgzxYzz0sodm8K3eYXwJwZmBaXAH9SocGqpfB//9Z45GmYeBp0yTEf5lcsho/ehLdertfOdWzfDH+6XbJ3t7mA3v9/kSVNPhwpzUikbz5rKFxM9ubCX+6StZYZkz6O0XS1DwxrDIFAIwzfqQipMIrJTt7drLy/VT6ItdX2ULoPPrkNDqylkcdz2V74/hHYMqcZIykUbYOyvLQWmgb3PGGKjY9fg6LaLKA2O3ibkW9i2gOBzzmcMDjCMFzDgOcfgW8/MP1cQlFaZK0eU1tRUmCxXQdO4BdFViyRQV2EDJ/pY7LsZ4NnHjdduura5+0PrGMNw/w43Xm9xOGUjDga7vgjfP0JbDhsbQ2HMcfBogBBXYYBbjd8/Lbk1nvMBV0IQbqew17PRstjSCRd7P3Z69kQunEz8BmCzORdSHIi7sPtg/e2ublxkBO+exBkkC2ohS9Ar3Fgs5imWaFoByjx0hLs3AIv/h9srk1INWAY/PaP0L0vXPwbmHod5O2B3B3wp+siH2fYGOgeRrI2q1SWw3WnwcEQ9ZUO52DotOxtRkona+2SLbZr7+zbBj9/C5Wl0KkLHHu29d8BTa0X/vB44G9/anrcigFOSnDVwOIF8POP8Ns7BeddAn/7U/jq5ejjILtr8KLqhg8WzIHf3imxO2r9bvTOlscQCJL1TIY4J1BjVHDQl2vpqkjUmCYkDr15CfR0AeuKDdj+I1SXBG8sDVj/NQw/v1ljKhStido2ijZvPws3nA5L55mJ6cqKYck8Uwy8+7zZRtehS4/wxUFDBo+Cf74TnTkfzr1XRza3YPWY2ppu/SG7N4cqCAogKQYyE6BzAsTYIC4Rhh7XptNsNh43vPInePAS+O5/sOAL+PQ5uPdMM1zbIgOGtI4/Z5215sWnJIlJEj2CXZnrbxWUl4fOX+3zQlVV/f8T9FTS9W6WCjMmaZ0ZHXsWumbj6LizGRoziTjRuEyBQCdJyyBR60Sy1pnejlGMiTsPG46AYzhcbuIqqtG99arLkIK95Skh5xSKGA1YbvEekbe+2eMpFK2JsrxEk1U/w+tPBD7/33/C0NEw7Bjz/+HkHgHI6WPmaLnmLugfpSRqh7NtA2xcFf51CUmRb1G1BkLAxXfBv2+DBDukxdGoFHKs3aw+XXEQUjtwZoz3/g5LvjN/bmiGkMCnz0JiKpwwJWQ3p5wl+Oz91ovGkRJ+mmOGOs/7wXpUe7fukJYO6Z1CX+NwQHxC42PDY09hYeXH1MgKDreS6NjJsvWlq2MAaXpXRO3nRRM61UY5VbJxtj6JjzKjgFiRxLj4C3FqZiXscfEXssm1kHzvjkNtu+48QP/1O4hxeWqvhaJOyaw8eiA1MU5m72xekVCvhAvSC2GTxW1QPTrZlxWK1kJZXqLJ038O3eaFx+p/Hjmu8QIajAuvh//OgMdeaznhArBgRmTXTXug/YdeDjkWLroJ0uL9/94NLzx/DbhrWn9u0aA4H378Ivgq/uXLlvZ1MjoLbrlbIER4NTCbw/KfTd8Uq2UHpIQzLzD9VyZMFkHflqbBSaeBzdb47x6jJXB8wiX0cx5DrEhEw0acSKa/cxwTE3/NoNjjKfcdZF3NHDbUzKfIu49drjVscwfOslcty9hU89Oh/yfoaYyOO4tJCddxfPwlnLwlgWErNuN01W8NCSC9sJQJ3y/h6+UDKKxK8NOzNXQBw1I1jkkosX5R/8kRj6dQtAXK8hItfvgMcreFbretgXk2syuccJopGPzdeX0GVLghqxfkFsCs72DCKTSxrXu9Zg6Z+ERISmnOu4CaqtBtDmfaA3DKlOaN21qs+y74/oL0wfR/wgV+nDnaO6vnBQ4PqqM4D/Zshu6hI23GTxJ0zYEP/idZ1gpRxQcLYP/e8K55/w0YPkqS1UVw7sWSLz5o2kbTICERLrg0wNaNiKGvcwx9nWMaHd/r3sTamlmNEsztdFuzSu7zbiazYDybVzswDOg3CLp0i8VZVQNrv8bMSNMU3Wdw1e6ZvMtYS+M0RMMsCzAwReODyXForlRrF9pjoeuIsMdTKNoSJV6iwf7d8I+7rbU9/LHyd38z87tsWGneZb0+U7CU1YC7diEq2QSbNsO7r0Pf/vDaR9ClGxQXwiO3wpol9f0mp5lh0udeEdl76R6Bufrk8yIbq7XxecxtoVBs/il0m/aIq8Y0k4QSMC7rWWd79xPceg9cd6FsmWoVDXBHUI6opAj+8aDkHy8IrrhekJQEn74vG4VoDx0Jv7ldkJ5hPfi40JvL6prvw58Q4Km2s+DFE/nvjzZkg+/7sJGSeya/iZPA+lkAx7CdRKopJ9bymH2TBMdl2jiru42Tu9rQhICYrpDaA4p3E9Rx+KzHrFuAFYp2ghIv0eDLd7AcVdAps/H/4xPhyfdh4Ux471X4fk7TqsYNV40d2+Cq8+HdL+CayU0XotIi+M8DsHcX3PTHcN8JTDgTnn3IugVG08NPetdWeCxuB/maF+nRZmT3Ci1chAaZ3cPqNj5BcNxJkgWz22cqn53bYOsmSd8BZsTSmefDpvXgcpk+MZnZtQuzlOCpNhdqe3BhsM0VWfElwyeY8chZ5G/OQsrGgmDdKijsvosuicGNfwLoTR6r6BlyvBgNxmfrdIvXOD7LxoTsWuEC5vsccxV8/2jt7cnPPWrM1ZDcxeK7UyjaD0q8NJf9u+GzN6zf1S+9qekx3QaDRsMPVzcVLofj85kC5pZLgz9Bf/JfmHoNdA7zxhQbB/f8Ex6eFtr5QNfhuMlmPpmOgNOiH0G4jtTthSHHQmpnM1eNPxGj6TBiQtjlD3Ztl/QfZPqkVFVa90lpTTashr4DzJ/tDsHQEQ1OluXBolcgb53p1wSQ1AWOuhB6Hd/E6uCRLop8kaX93b20J3kb/X/nDANq3NZuuaXEWWpXY8D3e33o+Hhts4esWMGHp8QxJK32M5w9DE6+Fxa+BJUNnHcd8TDyUhjYjrNiKxRBUOKlOZSVwG/OMsNTrTDyODjrUv/nrr/Eej+6Dhs3Quf44O1eegz+9B9rfTbkhNPMrLxv/huWL/DfRmjm67Kbw++/rRAC0rpB0Z7g7fqNa/5YNZWwvzZbb3Z/iIncAROAyoOw6GXYt9pcgDU75BwNY6+D2CSzjabDdY/AU9NM54eGglrTzUiji++yPGTuTsmz/5Bs29ygG619ipcmpgxPDWyZBeunN1606yjbB/P/DSW7YdRljU75ZOMHCHeVnW3z+nNwewaazSBn9C66jtiNpjf9RWydPQChGUjDv5fz3O3j6ZO+w++5OqQths/P680H2z08usJahfM622xBjeS8GVX8PCWe9JjaOXQdAVOfgbwNZpkAZ6JZEkBFGCk6MEq8NIdnHrS+vdJnEDz+P/97y7t2wOoQ6fcb4vNBgJtjI376wbQMZYe3TQDA0KPN+R7Mh3/dC0vmmiuX0MyEGUkpcP9T/ssTtGcuehhevDbweaHBZD/WsVBUl0NZvmlFW/I5rPym/ilfaDD0ZDj9VnBY92M4RNEOmH5fY2uK4YFdC2HPUjjvSUisTbjWfxTc+xp89SqsnGteY3fCcWfDWddDirVcPAf2Sf50h2ySXLlODw0cBh4XjYRNW9LI0lJTBt8+AKUWvH/XfGaKwIz+hw45RCw2nHhxsXtpD+Y8cSpelw2hm29+43dDSe5WxGl/mk5CRuNq8VXF8QGFC8D3W07l8pHv4bQFflARQ84mJ0GjZ0L4YV4+CSUuydtbPdw2tIFFVGiQ1cG+qwpFEISU7fI5yjJlZWUkJydTWlpKUlJS6w5+1mBwW3gyEho880ngEOcXnoJ/PGx9XCEg2QlpFhbC9Ex46StIshh5EIjc7fDT9+b77dkfxp1sljboiOxaDe/e0zSNrM0JV/4Lugyw1o/XDcu/gmVfQNFeQvo9pXWF37wSfjGfd68Bd2Xg8/EZcOGzTY+7qqG6AuKTzcKcYfDM3w3mzwy+G5qYZJYFaGs/mMxseOZ/DRb6WX+HPctD+//U0edEOOGWRoc21SxkycZdfHHvVFOMHOa/IjSDxMwyzn/yPXS7OY6GjcX/upS1ixKD/k7GDNzJ3cf82RSgh9PzOJhwBwDlHsnA98upjsBRemS6xsyzm2ntUyhamOas38ryEilVFdaEC8Af/h48N0t5mbUokTqkhNgYMyrJqYM9iI9GUQF8/T6MmQRvvmKGW/t8MHosXHUDHH2stTFzesMlN1pr297pMRzu/QbWzYZ1s8zf/ZBJMGi89aQmu9fAh3+BmorQbeso2gvz3oCJYZSE2LsyuHABqCwwI0pSD7OwOWPNV5i43ZIfLTjnlpeF3XVEpHeCgwFyrdns8Ke/NThQUQC5S8MboGhnk0N9nKN5/fMMU7TIptZSaWiU7U/Bs/xUBp5YTLyWQmdbbzLO1Fj9U2ARKwQMmdgLznkV1k+nZvM8ql0uNsss/q5PZWtBL7K/ruCUrnau7G/n90c5+etyi/eZBlRaKO+gUHRklHiJFKsGq7QMmByiZkivPtaFSx37S+p/jrNBRjzofhZeacA7r8IDfzHnXBe59O0X8NWn8Ie/wG/vCG/sI4UhE81XuBTtgXfvA2/4iwrLp4cnXnZZTLCyc2FT8RIhVZXWahu1Fnc+XM2imbF8+0XjGqb9B8Pv/wKp6Q0+9wVbwh/A1jRaziYc7Pq5F9IIHBckNMmuJb05f3L9+CkDDYxePsQOrUlJAE0zK29POgOwx/Af/RweqDoVTYAhMf2UPJLcSsmSAhf/WuPijQkx3D/CyeOrXPgs3nJ0AUNT23nCSIWimSjxEilxCeaWTKgaQBPOCt3XmVPgoXvNVSMSqrywrxy6JoF22M3Wa8D6bYBoLLjqRMw/Hobho+C4EyMb+5fI4k8iD6euqTC3m2wWt3GsWoKiGCEVHw82W+jAtzri4s22Vg2RlhEGnXoXkpv5JRfccAEXX5XG1o3mR7d3f0hK9iMsIkkH3KOp9VFKiccbPPeJNESj97y/yuDMbyspHifp5XDQdYsdrVb8SCSOQQYPP2wjNlYwd7+XB5aZFxt+RIkEXD64cnYNi89PYGxnnfNmWPOv80m4dkB424QKRUdDyfNIEQKuuCV4G5sdrvld6L7i4uHRp5qXKMpjQJmf1aPcZd4JA1mKdB0evQ9uPAsuOwHuuQrmfUOLZyTryKyfG76lrCHeMIRPn5OstesdPfFpdwhOmGRdB1RVwl+fhN/9RXDjndGahYGmScZe+yMe3Cyv/oaYWBg2SjBijPAvXAAyB4YnYOxx0PekJoeFEHTpRtCELJoGOT3r///0WjcHXRKPBpvHuJk/tZKVE6tZNaGaHy+o4quR1eT6DKSUPLq8xkI5SPBIeGqNi/HZNvolWbs/nN5VY1xmBw33VygsosRLczjzV3DqVP/nbHb4+//MvCmBkBK2bISli+CYcfDfD+Co0f7bOmNCi5tyP+Kl2kLemM0bYftGs8TAyoXw11vgoZvCW2R/SVhNducPTYfvn4eti60JoBirTmzR9bu/8ApBnLVUIwgBq5cJjh0vOOVMjZ59mp+wNblrCac/9DmZAw8AkiqjhCKfheih2BTobdF3yR4HZzwcMP/PmVOC15uWEk4+w2xhSMnbW9yNtna8TjjY1Udhjg93nMQm4J+rXBz9aQVLCw3Lf7F3t3mQUjKlpzUH+RsGOQ4VkVQojlTUtlFz0DT4/eNmXpTP/gdb14HTCSedA1Ouapogbusm+PBt2JcLlZXm//fmmueEgPGT4B/PmpaYinLI2wvvvww7N8H2faH9bPxtisfFQ01p0+OBqPPSXDQL3nnOLDWgaExaN8jfQUSCwfDB2h9g9XeQ1Q9+9X8QnxK4fdF2a/0e3AaJmaHbWSQzW/DXp+DhuyUlRcHbahpUV9dX6zn3IsG//xaZmBLCIKN/Hmc9+mkTAZTv2Um6rVvoTsZeB+X5kL/BjyO8gITO0O8kGHJe0Mivk8+EpQth1bLGXz1NM78m19ws6JxlTvJgjQzpJGtI+HxX+M5EbgPm7veR7LAmSBz+fN8UiiMMJV6aixBm2PC4kwO3kRIe+zO8+py5TWMYTYWIlLBgDlwwGT75HlYtgBf/r/68lfuRftjNTQjQDfPfYMIn1s/HQEozc/CvfttxMui2FkefB18/Gfn1Ru2WXN42+PABuPqpwKYKYdH8L6L/Vc7pIfjni5LrLyaoTvP5oPggrFgiGT4KTpgE27fA9I/DG09oBvYYD+Onzfb76yj07bbUT77Xwcc595DsWMaxZfPpJopxJHaCvhMhZ3RI/yC3T1Ltg0Q73PNXwdefwtefSg4WmOf7D4bzfyUYNdac5O4Kg3O/C+2vFulGowBm7/NyQlboz4IAeiUq8aI48lHipTX473OmcIHgviQ+H1RXwR9ugvKdjc8lOqAqxDZO4mEiQ0qI02sfiIMImASHee7wFaO8FPbsgN6hKxD/ohh+KmycDzuWBf6dxqdBZQiThTRg73rz1S1AArHMweZiawT53Gg2yBpsbe5hkpyiMeEUg3k/BNe/s7+D2d9JUlLhhtsFV90oyNsvWWK5xqWk+5gdHH3FIpK7+LcUVhhFuI1qHJr/8G8pJU+ucfPYSheGBE0chSGPAuCmrg4e7u7EY8BHW9y8ucXD3iqD7FiNy/rZOa+HjQ+3e/hgm4eVRQaGhGQ7XD3Awe3nOjnnQkFFubkbHBsrGo156cwq9la2XLosTYBXwqQuNrJiBXnV0q+W1AWc3EWnS3yUxIvXA5uXm7mCsnpA177R6VehiAJKvLQ0Hg88/5T19j4frFoB3ZPB1uAmFGeHGBvUBDA72zVI8mMh0YVZRqCwJnD4SF6leYdMdkJyTOOIJbV33hTdBhc/DAs/gKWfQ2WxeTwuGXqNMkOhkzOhtACeuSx4X5oOWxYFFi8xidB3Emz+Af/mDwH9Jlmv2xQMaZj9HfY3v+IGwbpVkqLC0LlfSool/3xI8ruHPZxyppMlQXKe1CEEDD99O6Ov/y5kW5eswuOJ4aMdHhbn+9A1mJBl4+weNt7c4uGRBun0G+6iPrfejSElP+f7WHHQQMO0hOyr9LG00MfvFza1jJR64Nl1bqbv8vDtmfF0SmoqCuYd8LGhpGWz9PkkjMnQ0TXB8+NjufiHKgzZ+P3pAlIcgr+NjSCD8+FICbPfhy9fhsoGQrLHYLjyfuiuHmYUbY8SLy3N2pWmTT1cvEZj8SIEZCXAwSooPyy1eJwdMuKahkk3PH90L9i0FSrd4DOa+scYEoproNQFqTGmFSclDbr1Cn/uvwR0O5xwORz3K6goMq0f8SmHLfxWFjVhhk67K81Ec0KD9F6gNwh1PeZqsz7P3hX1Phx1/3YdYVYOjhQpzVwy66ZD4VbTSpc5BIacA91GApCSKnjsGfjgf5I5M0KV4BJIJC+/WME1T31FdvezydtjDyh6hDB92s+40EW+hemuKrBz5exySt3mgi2At7Z4yFxCyEy0L2zwHPqK1E2n7lsQ6C/lk7CrQvLg0hqeOaGpB/PcfV5stZaRlkAT0MkpOKu7eauekG3j69Pj+dvKGmbt8yExn1vO72njjyNjyImgpEATvn0dPvWTsXn3Rvj79XDf642tMF4v7FwLrhrI6gnpWc2fg0IRAiVeWhpXhMkvDvdfAfNOlhFvlgWo8Zp33hhbY5ETCOmrFSUO2B3EgdeQcLAaiqrhpClhp5VvFVbPhy9egML95vxGTIDzb4G4xNafi6ZDUoB6QQnppuWkpjzw9YYXynbD+zfU10Kyx8HgM2H4VLN/3WFWBj6wFrbOgaoiiEszQ3yzhkZuHZMSlv4P1n9V24c0P1N5682xRl0Gw6YApoD5ze2Cq38rmfWN5L9+1rb6fgWle9LYsdPD+Pvf5fuHplK8P97sH9HgX0jrBHc/JOjarRezKuYS2LlGEE9XLvkBXLVKo6H+LqixJhX95VQJhU/Ce9u9jM9yc0FvO/YGDwlWE8fVvetw2ukCYnV4a1JsozFHZ+h8ODmeohqDEjdkxAoS7VGykJYXw+cv+D8nDXDXwOsPwX1vmJ+Z2e+bdbTKi+vfwdDj4LK7oVPX6MxJofCDEi8tzZowCi6CeUOwa8FT/usaxIcpKvoMhqJ8U5RYQQLvvgdX3wbde4Y3VkshJTx9K6w/LOvs3I9h/mfw+5eh7/A2mZpfdBuMOhsWvh8gLFqYoSvl20E0WNo8VbDqIyg7AONvNT8TQkD2MPMVLfatMoULNHZoqZvr8negy1GmJagWp9NcJIWQIYPfakpjSe9VyLlPvsmun3uzc1FvqoriQUi69XZz8rE5jD7Ghq4LIJ6etqPY6V3pty+B4Kddo3Ab/gVIS5dXMiTctKCGB5a5eHVCLMdnmbfOMRk6/7GgSqxqpkEpGrsrDRJsggt62blxkCOgNSUtRiOtaXLg5rFkRugQ/l0b4I6J0HMIbFx82Elpfj8fuwb+9CakRi8CTqFoiBIvLcmLT8PfHwrvGikDF1w0pLntU1679WPTzO2deHvwp+/EZLjwWlg0EzxhJJ/zeuCtV+D+R8J7Dy3Fh082FS51GD544kb497z2VTDy+Eth5wrYv6mxQKjLQ5IW21i4NGTHj9D/5JarBrzx2+A1tYQGm2bAcY1rWqVnWKuOEd/JrPuk2w16n7CV3idsbXS+FAcVnE8yGayvns9u7xq//cSKJIbFnswfNibha+M6soUuyYXfVzHz7HgGp+qcnmOjS5xgX1XoecXo4Pb5F1oCODFb59NT46M+57AoKTCtfaHqQ9RU+hEutRg+01fmq1fhivujP0eFApWkruWYPSN84aLVOtfG+Vl8vQbsLYOCKnPLyGOYCejyK81tIHeQm81v7oPhY+Hk88LbYvD54Nsvw3sPLYWUpoUlGF4PfPe/1pmPVRyxcMU/4KRr67eXhIBYu7kFGBNEaAkNtsxqubkVbg/+lC0NKGxaK2jkGEgIskMnhEGnPnmkdCsO3Ajw4mZJ5ecsrZzOLs8qZIBl/ei4c0i3daWmHSR9NqTp3/L0WnM72KYJ3p4UhzNEFLMu4OLedvoma00S3wkgJ0Hw3AlRcLZtLrHxwSPbrGL4YOFXoRykFIqIUeKlpfjPP8Jrn5wMvVLNsGV/7C83BYs/fBL2lpu5xA9Re4s853I47UIzTGTqtTAhSD4af7iakU02muTvAY8F/6FlM1t+LuFijzEde294EQYMhK6pkBYHjhCGT2lAeYjaWc0hSIK2+jZNP492h+DaW+qW4MMsDsJAaJKx1y6wNAUPNRT4dgZpIVlVPQOAoWmaX1ewQ0NbGrH5+CR8ttOL12d+H49K15l5VjzOAHdTAdgE3DrUwfdnxfPQ0U76Jmkk2KBXouBPo5zMOTuB7Lg2vB0bBkx/2YwwipZ1y+OCqlYqPa74xaG2jVqCg4VmWk6rCAHDhkDe5qZ5YKSEvIrAwuVQO0BLhvQYcLuh/1A47yoYOxG+eR/eesZM/y8lOHSzv1A3KV2HYSOtv4+WxGoFZ287ftJb/TFUFWDZA0IIiEluufl0P8bcOgpofRHQfYzfM+MnCex2+N+LUNBAX6X1OMi4G+bXpvWPDmVGAV7Dw/UDHczdH9hnSwKZsZBn0a2rOXgMSPrhIKdmOPhdz1jGpzn47sx4LvqhioIaiV6bVkliFn1/c2IcfZJM88wtQ5zcMqSdJX787Fn49o3o9qnpEBuFEH6Fwg9KvLQENWHePQcOgZGj4NumJnpKXWbVaCvs2gXvroPM7Ppjb/4H/vdU/f/rQq4PVJgb8MHw+eCK66yN3dJk9sBSzIbTYkGe1sbnNbeADB8UV8OBMqjxmI7ZnROgU4LpiN0QKaHPeDMKKXcpFO0yq1F3Gw2p3Zs/p4Gnw+bva0NmDregaGCLMXPMBODY8YK+Y/czY81SyksFCZ0qSOsZQVoAC1QbZZyZk8Zlfey8s83T6JOgCXM75+6jHFw/0MG9P9fw2U7voU2oZAeM62zj2z3hp+YPhKFJPMB3B918XejmhcEJXNstlpVTE/hkh4d5B3z4DMnYzjYu6WMnyWJq/zahpBBmvBndPjUdRp8Mjmh7FCsUJkq8tAQZmaZTQEWQENk64mxQvQd+mtHUSU5KKA1z22bP7nrxcmBPY+FSh02Drom1MacJsG9f4xICdcVbrrweJpwS3vgthc0OCSlQEdyPgmoLv/O2oKYM3NWwKQ+KquqPV3ugrAZyS2B4l/qtJKFBWk+wx8OHN5n1qYQOSFj+LnQdCSfeDo5miLWkLJh0D8z+x2EWK2mGa59yP8QGtvyU+PJYWvMZqf0MUiOfhSUcWixCCP59fAzHdNZ5br2bzaWmPBmRrnHrECfn1RYufGVCHI8dY7C+xMCpwchOOm9s9vDdHm9UyldKJJ44n5mip7bDm9ZXcHK6gx6xOpf3c3B5vygM1Fos+S66dT2FZn5fz2onDz6KIxIlXloChwN+dTX89/ngzm9pMZBS66RXXND0vMdPMrlQJNUuNlLC338fuJ0QZvhDvA3+/Ca8+qxZ3Rpg8DC4bhqcM7V9ZdhNzQgtXipKWmUqYWOPgdzixsKlIS4vrNgDY3qYpoSuI2DoFJjxV5C1olY2+CztWwU/PApjrjGz6yZFmBisy3C48Hkzf0zBJkBA9lDofaI55yBsrlmI/0T10cVBLE7NFGmaEFzV38FV/R1UeCSagDhb089oRqzGhNh6S9boTnrUhIuhSdzxjb/XAnh1Tw0P92vjaKFIKC82H1h84Qac14b6Q+19rtYeltEVrv0rdOkT5YkqFPUo8dJS3PJ7s9Dipg3+BUx6rJmKPxCaFjiENhDJKdB3gPnzJ6/BmiWhr6ksh3EnwKlnmZkypQR7Owo1bkh6F9i7LYggFJDSuVWnZB3N3KoLhteAfAf89h+Q3AXm/dsULP58k6QBBVvg69pQ1Lg0GPNr6Hls+FNzJsCQs4GzLV/iMqo46NsT/liHMch5Iltdi/EQ2MLY3znO7/GEMBKzfbC9+b5QEomhS6o6eZqEOviA5WUhao+1V9IyrUcYCc18oDF8EJ8EN//LrHu0ZoFpWczuDf1Hta+HHsURiRIvLUViErz3Fbz0b3jnNSguMr/QmRmguwgYmqDrcNQ4GDkOykrhqaesZ+k9arQ5htsFbwdLgXrYeDG11h9bO/84HH8urJwTvM3481tlKkGpLIOF02HTMkBC3xHQtV/g2lIN2b4FbPHmtt2uhaEThtVRVQRzn4Dqa2DQGc2ZvSXcsvlesel6N3o6h5Nt78OCyvdxyaZWqX6OseQ4/ReddPskRS6DJIfm1/pSx8fbPby8MXJhUVcHybBJqtKbChcwbQ6OQOU52jtHT4b3nwBfgN+RpsOwE+DEC8w8Sz4v9B5m+rTYax2Pj7MufBWKaNDOV6sOTkIi3PVHuP1eU4jExsI1p0BhkEgMnw8K9sGvfmv+P78UXn/J2iJ2Wu0NZN0yKC8J3V4IOPHM9lkCwB/DjofBx8KGn5taIzTdrKty/LltMrVDbFkB/7kdXNX1c1w936yFZAVpwIbFMOLEyPJtLH7dLNRoa9loFjvNd8SM1ZIAcGrxTEq8ljzPTna7V+GTXhL1TvR3Hotda/o+lhV4uf2nGtY3KIg4MFnjqeNiOKZz41uaISV/XR55uH//JI0eiYJCu4+fXIEFkATOzgjjeyQN2LEANnxtOmJrOnQ7GoaeA+m9I55vUAwD9q+BinzT2tZ1pLk1mJAC598MHz3d9BpNMwXKBbdAdi/zO6hQtAOUeGkNdB1S08yf4yyEDjas0XPXH2HFUli5NHB7IUzrSdVB+M+DUBGkdlFDNB0uvcla2/aApsO0f8GHT8GPn5lJ6cC8wY6aBJffCzFtGG1UUgD/vs20fDVKty/DC+F2u8z8KjFJpqNvWEhY/SmM+lWY1zWmWBZTLIuxYSNLZOEQjRfmfd5NzeofoLOtZ6P/Z9p7kmnv6bdtHT/s8XDJzOom/isbSw3O+KaK906OZXK3eqG4vthgd2Xk3i7/OSGGMRk2lpZ6OOFnl994Nx3IcGhckmVR0EkDfnwOts+rd5Q3vKalbddCOPE26HlcxHP2y54VsPBF00JXh80JR11kFuE89UqIiYcvXoSyBhFjvYaZ36tsVaBV0b5Q4qWl2bsTlswzF6SkFBh6NOzeGvyaSefU/xwXD+98AW/9F554tGkYtq6bN78kDd5/HhDWn9in/QV6DQjjzbQD7E647B447ybYttp8rz2HQEqntp4ZzPukVrg0s9JOTn9zURtwKqz+JPz+CiIXFiWyhAXeBRykfgHT0RmkDWKENgKttqzBLrf/VP5WiRPJZBwmXkJhSMmv5zYVLnVI4Jo51ey6zIZeu4WTXx3538ImoE+i+X6PTrbz5vBErl5Tjk/Wl5Y0gM5OjW9GJxNvVMGG+WZhS90BnQdAr+ObRoRt/N4ULuC/ptT8/0DmYIhNiXjujdi/FmY93tRa6XXBsrdMv6ph55vbQsefa36vaiohIweye0ZnDgpFlFHipaWoLId/3A0LZoR/7cjj4at3Ycs6SEmHKVfBdTfD5dfAGy/Dmy/D/r1mhtQB/aE0F5w20yxsBSEgpw+cfVn4c2svxCfB8BPaehaNWTW3ecJFaNBjkCleAAafDbt+htK94fWrRfa1LpflfOv9Fg+Nt0d8+FhrrKVG1nCczbQIeJrh8xIrkhgTf94hIWSVL3Z5QqY8qvLBV7lezu1hWl+eWxeZo64u4IJeNtJi6ud4UVYMJ6Y6eG1vNUvLvDiE4PRODi7qbCd21buwfnrjv9P2ebD4NRh7LfQ/xRQPPz5bL1wCYRhmTqDhF0Q0d8D0S8ldDIXbYNu8WuESQPat/MgUyo54857Sf1Tk4yoUrYQSLy2BYcCfb4C1YWTZPYSA357VWIi8/QwcOwkefAFuvM18uWpMB9BLx5nCxXL3GjiccPc/VERAtGlOHRehmduF1zaoh+WIgzMehuXvwdbZ4LPYf9+JEU1hjW8NHjwBw5+3yq0MloNJESno2PER2gF5bNxU8j3bKfcVYdPsZNp7kWnriy5CFAPyw7z91iyKPx0wxcuaIh+zLF7TEF1At3jBw0c33QbKdGrc2/uwcOhFr5gFLP1heGHhS+BIgJ2120IhkaboiJT8TTD7n7W5gYIU3jw0Rw/sWgz9IvvcKBRtgRIvLcGy+dbClP0izXShh7NoFtz3a/h7bSZMZwysmAXVAfKGBKL3ALj3CejZP8L5KQLSZzjk51rftrM5TF8YRwwcdw6cdjWkH5avxREPx14HOUebeV1CYY+LKFzakAY75I6geVsEgm3GNkbro+niGMBO98qgfSZpnUmzZZNmy/Z7fnOJjz8vrWFjiYFdg7O727l3hIMYm3+LTLCIoobUhVB/vtODrUEiOSsk2eHKfg7uGOYgPcaCZahsf2Dh0pBlb0OF1TpVwlrdKb/zOQAzHqkXulYsdkIzhU4YSCkRbfTwI6U0BZdmb7M5KNoeJV5agllfmM6l0ajO2pAVP8HrT0D/4TDmRKiuDO963QZjJijh0lKcdBEs+MJ6+2PPhEt+b1rCQt2E11noV7PBmY+Yi1GYePDgI/Tntbp2u6ifYyy73WsxglhfhscGzs5896JqXtnUeHvq3+vcPL/BzfTT4hjTuemt6cq+dp5bH9r6dFU/c8uozC3DKtY49+x4Bqdqh/xlLLF9PpbKVlgWLph9dY2wptj6r8yFPZyUfNKA+NA+Y9LnRubOwcidA9WFoDsRWWPQep6KiI8wSWIYyJpijJ3fIfcuAJ/LHL/r8Wg9T0PEtHSOZ0V7Q4mXlqC0OPrCpY66/C3JqXDxjeFdaxiQlmFGLr3/P9i+1eznnAvgtHPMzMDNobwMPnoHvv4MCgvMm2J1lRmKffwEuOo3MGR488aINlLWlhSQEJvUvK20HoPMkNJPnrHWvnCvaUELhafGDHENRe8TIaWbtbEPw44dHT2kgIkTpvOpTbMzPv5SFlZ+hJvG/i8aNo6OO5tEPc1vH69tcjURLnV4DDj7uyp2/iqecq/AkNA5VqAJwYBUnSGpGuuKA1sTRnfS6J5obkn1TtIsWV0EMKWnjWHp4W9lUR2excISsamRRxvtWBC+35UtJmABzjqk14Vv2RNQupNDwsjnQu77Cd/+n9FH34FIbbmaCLIqH9/PfwNPZePxd8/Gd2AJ+jH3IOLaa4JKRUugxEtLkJVjRgEdXiE6mpQWw8uPQ3Z3yNtrTSwJAYuWwu/vMq0wPq8ZZjzzG+g3EN76DDpFeAPYsQ0uPwfy8/xnhP34XVPY/N9TcMlVkY0RTaSEld/Aog+hqDZTbFo3GHshjDwzchFz+q8hLRte+WPotmmZ1vp0W9warPRTYsIimtDoo/Vhi7El4NaRRNJHq0/5Hqcnc3LSdRR6drPHswGJJMPWk672AUHN+Y+vDJ500WPAkI8qKak1snSLF9w02MFvBjqYflocJ35ZSa6f8Oc+SYIvT6uP7Lm4t50Hl7lwh1jLJ2TrPH1cbPBGgYjzL9AixhEPk/9kbilGgjeCnDYjLgqZF8jYPr2xcKlDmtXpfateQD/xcUSEzuKh8K1+BTz+MlRLcJfjW/s6tmPubpGxFe2T8O3LitCccVHLCpc6BKZocThNsRSKPqNNAQH1RSDrHIO3b4Gbr45sHj4fXH9JrbUlwKNu3fH774CN6yIbJ1pICd/+G75+Eor21h8v2gPfPGW+Ar0PKxw9GeKSQrfra3FrwGMxsqeZIdpDtaE4cCACbLb0F/1JFk0LNXayd2dE3GmMjDudbo6BQYWL22uQb2F9LWmwO7SnUvLHJS6mfl9Fgh1WTE3gxRNiGJam0TVOMCJd4/UJsSw5P7GRv0xajMbjY03L1uE3OgEk2OCtibF8PDnOeqkBaZiWsLrPR58J1q5zWKl5JGDqc5CaY61PfyR1Cf+anBBWF8OD3DOPwFtRpoCQ+avCH9sCRvk+KNsZvFHJVmTF/hYZX9E+UeKlJeg3NPww5ORUM/w3HKQ0rS73PWFmyg0kYOIS4Pq7YUWQm4vPB8t+htUrwpsDwLxZsHO7NcEmBLz5SvhjRJOdK2D59Nr/+Lkhr/gadkQSKVaLpsHpIYRgTDyMOc1af1az86Y1L5FYgkjgDNsZZIiMRsdt2BiuDWesPrZZ/QPUNENfzTvgY/hHlWwsMbioj4O55ySw5qJEZp2dwLk9/f+Oru7v4K2JsQxKrb/VOTS4oq+dZRckcGZ3i06fZfthwfPw1hXwzlXw3rWmE67NCcPOC36tI96aZTShEzgitADVMfDUMC8Qoa1H1QfBG0JACx1ZtivMsa0hD1gLfjDKc1tkfEX7RG0btRS3PgRZ3eC9FxtnvPUZUO6GmtoiiDE2SHbClF+b7R//XfhjOWPh/qfgzkehvBQSkiB3u5kgLy4BRh4Hu3dC/n3B+9F1+HE2DA/TWXDhPLMukpXaPVLC91/Do0+GN0Y0WfYFQZ0shYBlX0LvoyMf45TLYfNyWLvgsL41c8vu5n9a83cBSMgwXxUhtoW6HxPZXBuQJJI43XY6pbKUElmCjk6myMQuolOsM8mhhR0B1JD91ZKzv61k3rkJdIu39ux1Znc7Z+TY2FMpqfBIuiVoJIZR1JGD2+HbB82kbnWfGXclrPvSDH8+46/gTISVHzbdtuk2yszz8tX9obd0EixuIwaj7yTYuQgOrCOwpaQBSVmht6gsbQXJiPMLhey5Mkg5lQaIUAJLcUShxEtLoWlwyY0w9Vp4/lH44k2ocEP+YRFC1V4oroHcfLjsZjN/y7MPhZczpHOtqTg23nwBDBhuvuqwahXx+nekDE6YK9HBAtO5NzFMS1O02LuRoHOWEg5sad4YNptZyuCn6TD7A9i/w9zeO3oynHJZeOnWhTAzoC58KfD5tF5mRtcokSyS/W4RRYPTc3Sm7458W7XcA8+vd/PoGOv1lYQQ5CRE4MckDZj1d//CQxqmoFzyBky4AwaebuZYqSwAZzJkDqjfLup+jJl4LuDWnoDuzRDLdeg2OOU+WPOpGXkUastxzK9D9xmTDnGZUBUkYkoaaBkt5IxvVZQkqxIGvyTUtlFLY7PDLQ+YhfYOFy4NeeUZeOgeOOtX8OVauPufcN6VwZ/ONQ0GjYAcC4XcevaG+BB1lbxeszJ1uIwea83q0pDPPwx/nGjhseB0EZGIOwzdBuOnwF/egecXwtNz4Mo/RlYnpt/JZsZdaBAKXbsYJ2bBxLs7TNLBF8bHkt6MupE+Ce9vi8LfxwrrvmpcD6gJEnYtMqOOdDtkDzWTBOaMauznMiiIE7jQzEKJVv1nQqHbYcTFcMkr0CPIVt+AU6FbaCurEAKtV5BK5UKDlL6I5J7hzzUE0vBB6Y7QDTU72OOQJduRVflRn4ei/aHES2sghPm4GGpxeetV05lV12Hy+XDLg/CHvwOi6bVa3fbDX6zNISYWLv21eZ0/dB269YDxk6z115CTz4CsLtYXT90GW5tf2C9ivMGjXQCzhlJ7QggYcxWc/TezanTnQeaWxPhb4dx/QnyUo15akDibxqoLEzgrR6dh3rm4MCKVS93NcKi2iqsClr8Tup00zBIOwUjpBhP/YNY8QpivOhHqTIRT/2zRqTcMdDtMuBOOvd4UuHUkd4Pjb4ax11nuSnQZh+h9uHiu/TehK/qI6Bd4lTXFZni0z4LlRXdizL8f3+K/4fvxT3jn/A5jT4gyDIoOjdo2ag0MA2bNsBDBIuDDt+DPj9UfmnCWmSfl5cdhT4MnkP7DTOEy8Chrc1i1DH6a57/+ka6bVpkX3gwsboJhs8HL78Jl55jbQaEQQGwbVX92VdVHWgUjgvT1rUJ6bxj3m7aeRbOJs2m8OclcrItqDOJs4DYEYz+rIL86WJ5fky5xrWBl2vC1WbTQCvmbIGtw8DbdRsFFL8DWOVCw2Uxk2eUoM6eLH78Tn6eCmrINSMOFLaYzzoS+iHATEArNtLD0n2yKMSFMkRSmlU4Igd73XGT2MRh75ptbSLZYROZoRMbw8OcVAumpxLf4cagptnbB4WHU7nKM9W9h5K9CH3mLysR7BKLES2vg81lbMJGwzk8ysuMmw7hTYNt6KCkyfVy692naLhCrV8ClZ4MngKn9uAlm/pUukSU4A2DwMJixCJ5+HN57I3hbrxdOPyd4m5bAVQmvTrPY1l9OCUVLUFf8MAb44rQ4pnxXyf4gD9sacM2AZiZUtMK2+dbbrngXYpNNq1gwnAkw5OygTaT0Ubr3SyoLF2LWrTady3VHKqndL8WZEMG2oxAQkxj+dYd3E5+FPuCiZvcTCrl7dq1waaaFrXANxs4Z6L0sRvYpOgxq26g1sNutZ69dvthM+HY4QkDfIXD0+PCEC8D//Rk83sBVp5cvhtQobDt0zjKjiF54K3AbXYdjT4DhbVC5dv6bULzPWtvq8padi8Iv/ZJ1Vl+YyMld/Fu+dAEDUjSuH2jh+7R9GXzwF3jlRnj3flj1LXgsbBnWYTU5YB2LX7PmTxWCktyPqSxcgClcoG4B97lLKNz2Eu6qEFtURwDG3h9ptnCpRe74GmmE6ZOnaPco8dIa5O0Ht8XoIcMH998OVWHWLQrEnt2w5KfgeSYqK+DLT6IzHsDkM+Gfz9fX7LHZzBfA0cfC82+2vnOp123mb7F6Q4y0MJ6i2eia4P1T4nhwtJN0Z/3nxK7BpX3sfHV6fPCkcqV58Nyv4d17YctCyNsO25fA9H/Bi9dBicVkZklhhi57XabzbjPw1BRQVRQor4kEaVCe932zxugQuKP48OCthvI90etP0S5Qd+jWoDqM/ANSwuKf4Lgh8OwbZk2g5nDAoqXhT3eaETaXXdO88eo4/xKYdBp89oHpnBsXb24VjTi6baJiygvBHcbfof+4lpvLEYghJTvJYwd5lFFFKgn0JoscOkXkb6AJwW1Dnfx2kIM1RQYeQzIwRSfFGaSvskIza/K2xYHblObDe3+EG18JXcCy/ylQuNX6pIUOlYXW2/uhungF5jNloJBqg5Lyzcz2rWGddhAvkmScTKY3R4mWL47YajiTzOR40UJZXo44lHhpDbK7mIt3ONaU8nK4/lcwfS70aUYV6PTQ1WIB0y/nz7+DkmK4+a7Ix2tIcgpc3U6cS8OqFSNg3CUtNpUjjT2ykC9ZQiWNt0yWspVU4pkix5EuIvO3cOiC0RkWnKd3LDctLSGd4iUczDW3lPoET4tP7xNh61zI32BtstIwfVqageEz7xE+BDuSMjgYE4/d8NGnrIBkdzX7Y5P4sO8xGFp94rYavLzFGubL3dzM0WhRdp5tC7Su4zG2fhat3iAhO0p9KdoLHf9T3hFwxsAlV1qrP3QIaTr5vvZC88bu1ReGjrAeRfTEo7D3CEyzndgJMvtYsPoIOP+PkN0MwWiBvCKDPz/v5aw7PJx+q4drHvawYGXzahNZxZAGy+U23pfzeV/OZ7HcjBFhXaQCWcqHLGgiXOooppL3mEelDMPXJFzc1aY1xWo9KiFgm4WU87oNJt8Pg84w84hY6bfHsdbmEGhIRyq7E1J5ZfCJTO81gp+z+jC/ywBeG3gCX+cM5aO+YzACfIZ3UcrHbGzW+G2BlBJZU4ysKcLweTB2z8HYuyD0hZYQZpi3Pcph6Io2R0jZnAp0bU9ZWRnJycmUlpaSlNRGGVutUF4GF59hbqEEcpz1R3IqLPfjwBsOP82Dq6ceqgAbkvMugidebN6Y7ZGNP8LHDwU+n9QZrn0G4lNbdBqzlxo89JLPr/fN8UcJ/m9aaIOolJL9FLOTfCSSbFLpSSZaCHG2Tx7kA37Ee9i2hAC6kk4icfQggwF0w24hXPwLuZgt7A3pSTSOgRwvBoXsLyLmvA4L3g7jAgGjz4HTb7V+iafaDHFe/FrgNkPPhdFXhDGPpuzy7Oc5bbUpUA7/W0oZUnzrCP6PSR3C+iKlRO6Zh7FzBlTXlr7QbNHd4onLQh97jxIv7ZTmrN/t4hP+7LPP0rNnT2JiYhg7diyLFwfZs+6oJCbBB9/AjXdYr2kDZrmA5nLcifDyO9bT8c85Qh0CB54Ap04zfR2EZvooaLULdLehcP0LLS5cSssNHnrZv3ABWLBK8sb04LlF9shCXuV73mEuP7GBn9nEJyzkFWaQL0sDXlcha3iX+U2EC5huzHs4yEZy+ZblvMJ37JUHWSN3slBuZLXcSY1s7HTulT62sM+SC/QyttBiz0lbfgrzAgldwxRS9ljTAjPxboip/R7VCQTNBkOnwKgwi7E2wCcNlsi9vGTbgCE0/yLFgu+QD8k+2n+knJQSY+O7GBverhcuED3h4khC63se+rH3KeFyhNLmlpf333+fq666ihdeeIGxY8fy1FNP8eGHH7Jp0yY6d+4c8voOY3lpiLd2O+jpv0F1kHBMTYPBw+HzWdEZd/5M+LWFHA0OJ2w4gsvLlx+E1d+Zvg+OOBh0InQf3iqOxI+/4eXrBcG/cgmx8NXTTbcpimUF37GcPfh3ZBSAAzu/5mQSRdPqxNPlYjYSfpithsBAoqNxPIMYQz+EEFRJF8/xteV+htCdNBLpSWcyRUrY8wjIi9dDYRgVjZ3xcMcHYfpBNcDwwp4VUJ5nfn5yxvjPoVKRD7nLzCik1O7QZYTf7VuX9PIvFlIcYOutEZJDVSECMY2j6SlaVoQ3F1m8Bd+Sf7TsILEZaN0nIXImIFqoaKSieTRn/W7zv+gTTzzBDTfcwDXXmFEuL7zwAl999RX//e9/uffee9t4di2EzQY33ALjJ8JZ4wO3Mwy48vrojXvcSWZm22CCCaDHEV7gLDEdjo/8Kbk5LNsQ+lmhohqqaiRxMfWrVJms4h3mUkPgkHsJuPGygu2cyJAm57cTpLBeEIy6PCMYzGMdNnRG0QcnduzoeLCWhXYduxEI5rOOrjKNcxlLvAjDChmI7sOsixehwcV/jVy4gGlp6R7E2dfrhoUvwvb5HCrtIQ2IS4cTb4PMxlaf51lqTbhASOEigC40PxldS2PkzjX/FhH6WlmiugBj0wdQuBZ95DQlYI4w2nTbyO12s2zZMk455ZRDxzRN45RTTmHhwoV+r3G5XJSVlTV6dVgGDoHf/9n8+fAnMiFg8llmyHG00HW46/7Q7S79dfTGVFDmK2BV1Q98X/YyVb4KrOSaMQxwSQ/L5TY+lQt5l3lU4w55pUSyHv8O1/62iyJhARvwSh+60BhGz1DraZP5AeyjmPf5Ea/V9PvBGHextXYZveA3L5lipyWZ9zTs+LH2P7J+ga4qgu8fgaJ6oVUgK9kbxW2ePqThEO1/kZYVe1tWuNSPBAfXI3PntMJYitakTcVLYWEhPp+PzMzGyaAyMzM5cOCA32see+wxkpOTD71ycnJaY6otx013wrOvmxFBdXTtDvf/FZ55LcwIJQtcc5OZf8UfQjPzsPzqquiO+Qtmv2crCyo/YL93E15cZHUrCHGFBE3y6LYDPG98xyxWs40DlGM9R40b/2UgYrEQMWMBFx52Y76PsfQnnvCtJxJJEeVsjmAbqwkp2TA5SGHAmASY9rYpXDr1aP54wSjcBrlLAjjGSzNZ5Or6hJDz2R21oeOwczUWa521Nbam25oth8TYPbPlfK4UbUK7cNgNh/vuu4/S0tJDr9zcIyCs9/Rz4dMfYOVOWLoV5q6Aa2+uz0obTYSAF9+Gex9qXBIgNhauuh7e/DQ8h2Ir+HxmtJUvCk/ZHQiXUcmq6hmYS7V545x0Tp1FMdCNVFCRU0V8/yUBRUgoUvDvoDiSMMtKBKEUMx9JDA6SiHwh2kCUMp8ecwFc9jhk9a0/5oyHsRfCnR9CSmj/uaiwY0Hwop7SgN0/g8/c/qshPAfVs+nHbxhFNvX5ZGxojCaLP3ICMR3A6gKgZYXIsRNtqg+CYTHLuaJD0Kaf9E6dOqHrOnl5jffi8/LyyMryny3S6XTidDpbY3qtj9VooOaiaXDDraYVZstG04G4d1+zsnQ02bcHXngKPn4HamogJhYu+JVpbWpOEciDB2DzUnNvpfdwyO4ZrRlHlVzPeg6vj5zaqZwTTl3KjzOOpqH3pUQiENTEGGSemovD7kOL0H+4L/4Tch1NH35iwyEfluYwk9UUyQoySWEfFiv/+qGaKOZ/6TUKrnu+1uohQ2fQbQncFrYFpWHWQNIddCOJFfi3MjckGSdn0I/Rwvzb3kXHzgAtuoyDHd+Auyz49lGdX4wzFdF9InJLpGVMRPutFK+IiDYVLw6Hg9GjRzNz5kymTJkCgGEYzJw5k1tuuaUtp/bLwGaDQUNbpu8dW2HKyVDRYD+/phrefR2++hQ+nmEm0AuHqgp48W7YcFgofY9BcPO/ILUZT9fF+TD/U9i+xkxONmQcHHsWxEUu6Eq8efhbyE44dTlZ3QqZPX0sB/NTkYBPh+IsDwe6exmbUxiWH8nh/MRGdsp8skjBh0EKCQymOzNZFRXhUscKthOPs7bmcWSkEmXBDLVRY35+g3tXwbK3oSQXpM90oB1yDgw4BfQoValOsPAZtMWAw7SOjaEL09kc9PcXi40/Mj5kmQWvNPjZKGShUUgxbpKwM0pL43gtg7hWsshIKS2VgxC2GPQxv8e3/D9QlVcrNIX5d3EkIzJHmcJGsyMyhiM6j0Dmr4r80xuTgtz5HXQZh4iJQhFaRZvTLkKlr776al588UWOOeYYnnrqKT744AM2btzYxBfGHx0yVPqXwMRRsHtn4PMjx8BH31nvz+uFP18ABwPUaopPgkc/h7gIIi1+/hZee6Bp8crYRLjzWeg5OPw+geVVX5Pn3R60jWFoXPPFFXgbfAv/n73zDo+juvrwe2e2r3q3ZNmWe8dgDDa9995L6CHUJKQR+EhIIyQkJIGQhBICoRMg9N6rMcYG995kq/e2fWfu98dIVtsyu1rJNtnXjx5Zu3dm7u7Ozj1zyu9ccvyXlBWmJhG9ZxnZXaP953Iw5cJkC4tkqV8Ln9wTve9Q4WQ45tahVSD14GmC564j6jsuFEMvZt4lOx/6SG7jVTYOHtvtmLuauUwQsRfcDXoHd2vr8ESo/LKjcIM6lWlKdgIvxDxS6siaRejb3zMaICoqonA2yrhjEdkV6B1VyLovQA8j8mcgCqYjhIL0NqBtehnqlrCzl5M1E0oXoFYch7D1N2xl22a0xXckOcteg0qMPxFlwslJ9dxKk1r26FLpc889l8bGRm699Vbq6uqYM2cOb775pinDJc1uSFMj/P7W2IYLwNdfwoa1MNmkWNj7T0c3XAA8HfDSvXD+jaanCsC2tfCvn0V+ztcJf74Wfv9KUkZRgWVMTONFIGjzjmZ8aRPzZlTisIVp7XTQ3O6iJL8TVRm6ybG7Gi0ARWQzmvzhPcjy52DZM7HHNG6AFc8NSWRuJ13t4BwHlSuM8JWqgNMKLpshiOjKh1mn99vkUDEORVN5SW5EqL3GR5vPxtYt4wmPzyJKGhMANdLLH7U1hKJ82gF0/qyt5fdib/JFakPuUuroKx5E1i+BHh+cHkY2LEOr/xqsGRDq9b7K7e+BxYmYdCZy438NDZy+VXChTqh8G237eyizLu+fG5M9HpyF4Gsi8TO7d7zc8irSlokYc3gSrzjN7sIu97wMlbTnZTdi3Wq48BSjuaMZ7vonnHymubE/PAq62mKPsdrh7wn2RPntxVC5JvaYc34ER52f2H6BsAzyYdejhGSASBfbMLDZWUHYEoxYnPJNvjFUEHyHY8mIIKaXEP4O2LoQfG3gyoFxB/Qq4FYvg3dvN7cfqxPO/ZcRMkwGTys8fxtsXxH5eVWBOYfCodeCM6ffU1JKzljWwRuNAYqyPditOu1eG61eByqQYxUsWZDLaIc6aDtvCJ4Qm/mcxpjLuQBOUEo5W01ttZVe9Sn6mkeT3DpewFGg7ncjIqc30VxvXIn+9d+6/xrC0mXLQj3kDoSSzoPZlezx7QHSfAMIheDyc4yqIrM4Eli4PNFl73vnkGDyp65DpYmOwV+YV5Hti0XYmOc6FSv973ZFtwu7OmMcYdWogOhpZdP3B8z3GtxTEICKwhkcMDTDxd8Jn/8T/vMdWPwQrH4JvngYnr0KVr5gvHFrXiOuqlsPIR944pWxR8DbDp89Bf+4NLrhAoZzYeMqsLoGPfVZW5hXG4NoCGrbM9jWlEWr16j404C2sOSuyt5SeV9Icu+nIQ79q599/+jj01CTCf0fWKq3JPrq4qJvfw/T7/Eg4p/c2tY3+/2tFM5C2ft6sA8IowkVLIPf26gEO6Bjm/nxaXY7dnnYKM03hHdfh/oEWwrMP8j82OFYxVtqMXX31pH8RT9bLeTQzIuoDq2jIbQNHY0ctRibrZzVYmnU7fZko6XnfnoUuexFBVU0UUsrCoIJjGI248gSCSw0AIEuaK+BoAc2fQSVn9Pvs+vJV9I1+OopsNiNXJdE7s4TVWDdsBBe+C2EQ/GPI3XoaIT1n8KMI/o99XiNH4ugX95TXzQJ/672c+eUDLxByaWPB1hdJ9G7c2IUq7nXGEqRSKFs24y+/X1k83oIDadIqITGlUipI/pUjgl7Vr9QlDFUg3Ac5fCBe9cCQ0qMT7NrSRsvaVLD4s+N6qVwAroVXo/58nBnppGDEgtbgnfyZrt75xQmtt8BWIWdcba9GGfrFRB7SX4Rc5s9NWQ0nyk4sVFOIUXCSBCdyRBCFYEuWPKYIbWfSNO+Zc8mdhxXHrgTSBxu2Ar//fXgJO9YCAU2LR5kvDQG9aiGSw/tYYkuJQ98Fu41XACkINBhxZYZinvOVIihV3bp295B3/Ds8Ev79x7RMEy6jRcpdbQlf06JZotwR5YUSLNnkA4bpUkNiS62QsBzT5off4QJCfjDzkpsDlkmSyYPOjWx/ZoglKA42Z7CXCYyV0zcabgMiaAX3rgVNn+UeLfhoMdohmhW62XOOYlZjIv/m9h8AJBG36MBjHYoWOIcutAq0HR46qtwr+HSTf3X5oyuo5TI2llmkW2bDcMFRshwAZwFCKVXGVpu/xDC5tWmIyIUKJiFcOzezSvTxCZtvKRJDfsfmJjXRVFg62bz44+5CDJiXGzc2XBSgk0sX/+3uXEHnpLYfk1QiInFfQ+KHQlgNPk4RYr0UgDWvgHtQ+iBU7qXuW0nHm78JMKGzxPzuuyc05RBD11S5ojpeVGBb4920uyBjgj9G+u/KsBT74x5upyklDF1iKXSWuU7Q9o+cQTKmP5eKr1m4RB3qYA1A3XarmnMmiZ1pI2XNKnhyONhVJn5XkxCQEYC5cfODPjZYzB68uDnRk+Cnz8Bjhj1pJH46l1z48yGlxJgAVNjD5ASeyiIIxTcI4wYiYnXlCirX2FIFSVj5sG+PX26Irg23IVwxE1w4DWJx+m0JFo3KCrMHtxXbJ8sK5eW2iM6Ly0CyhwK3xvrxB4lyK+HVNY+PZG6pYXo4f57KcbB9epkzlLHJD7fgTStNjnQzHspoDhOi4D8aYjyw/o/FmgzOYcI81AsiLKDUOffgnAOc4l+mmEnnfOSJjVYLPCv/xil0u1t8Rf8cBhOOj32mIHklcCtT8K21bDhK+OxyfvAuBlJTdlItDRBKACW1DQ17MEmLBwop/EZEaqduo2VQ7etI8vv473x02l1Z6YsEWYoiri9+zBqpnQkKgrHsjdjRQr7B2lBCCWWgNk7OQVyyiGvAvLHQ8kMWPcWNK43qlJGz4Epx0LGEHKZSibCjtXmvUJCgVNvAndOxKfvm5HJaIfK3ZU+OjXj0xHACQU27pmWQYFNARvsVSpYWSsHhY70oMr298vY8fEo7r5IMqtEpRA76sBu9Ukipb6zH1NMFCuicBbkTERWvg/+CMKAihWxzw2oeZOQ3tPQt7yKbF4HgW6JBdUOihW8jehrn0IU7QVdNciwz2QYUKDM/YFRTSQUyJmEcOaC1d0vBJVmzyZtvKRJHVOmw9tfwDOPwXNPRA8LqSrsuwDm7p/cccbNSN5g6UvxGGiJ01dGUQ2vzzCwQEzFLR18LFfiJ2QYJ1KSGfCxoHIj+X6j+eFxm1fx+uS9aHe4k69K7YMLB5Mp5Wtiq//GIo8MyikgnyymU45dpHhRaIygOGsKYVQNHXBVr7GXXwEHXp2yqQGw72mwfWX8cYoKkw+ABedGDBn1oArBLya6+UmFi8/bQgR0yexMyyBtl6sOsnLtM5GNCFXA7BKVo0bZU68eG2jHlMmbUYoy+yq0L34PgShVenoI4a2DvEkIVyHqzMsA0GoWIVc9DHoItACEupDVjcjqT4zthGok78ZDCETuJER+aj2BnmAtDZ1f0O7fhETHbR1FYeY8chxT0mq9u4C08ZImteTlw9U3GD8fvwc3XGl4YixW4y5V0+Cgw+HuB3d9Sc1ZN8Bv4sS+Zx04rFOYLcYxqcPL9qon8NrsZPl8OCIkpx62eQ2vzdiP8BB8Jrm4WcA0JlOKRahMlKN4naV4iJBIEYcOvBwl5iQ9l7gEPMltV7aXoZSbNy6l0xnE1INhzvGw7I2dRifQW4Uz/2w4+KKEK+BcquDI/Oh5Q4dPUrn1OCu3vdXrNRQCNB2mlwj+fvYwGC5gvqlh5hho3wIdW2MO0ze9jCg7aOdcpbcBufrfgIweJjVjuABIHdmyFlGQur5tjV1L2dHWv51JV3AHXc07KHDPpTznmLQBM8KkjZc0w8chR8Lna+GtV2HDGrA74OgTYGoKvCapoHwyHHAyLHwl8vOuTLjiN8M+DYEgIxQkIxTdLe/SQlzcVsZHuWG2Up9wg0WBYBKlTBflADTINpaxNSnDBSCExjPyUxYwdXh6E2Um2B6kfB7sd+nQQkGJIASc8AMYMwsWvwB1m4zHKvYxDJeKfYbt0OfPtXDkZJXnl4fZ0ixxWeGYaSrzxykow7WA2jIhYzR0VcUcppTMQ29aFb+UOtiOvvF51MlnIn3NaJtfTW1ulz/5TucD6QrWDDJc+tLkWUqmfSy5rhTnfKWJSdp4STO82O1wypmAyTYAI82lv4BRFfDGv8HbLbglFKOr9HduTzwJOAms7jEmXOKCrIxJnC5yANgia3meRaaPIZFMoxyPDPAqi9lBlCaFCbCDJnbwCSfJ/Zgiyoa8v37kjTVyVlpi38GTPRqOuBGyIpQBBzzQWmu0jcgbnXpPnxAw62jjR+qAGDFvYlGm4OqDRi5/QwiBMv549BX/jDYCMsoQeVOQLWswE9+U294ivONj0IZY+hwJe+qaUFY2vxx3TEPXl2njZYRJGy9p0hx7sfHjaYeAH3IKjFyFEUK1uHHl7Yu3eTGR8woEzpzZWGw5Ox8ZL0ZhlRZTejECmEwZuWTwOB/STByxP5PI7rm+yVIqZDE2keLLyYLvwJu/6K7sifC+zDjZCBEN/Ky87fDBv2Dlu71VQZkFcNjlMPvo1M6xB7N6MnswSsk8I4l204t9PCvd6d+uItR9vosQApFVgTQb4hkOwwVhhK9SgCdYR0CLr7DtDVan5HhpzJM2XtKk6cGdbfzsArLLTiEcaCLYtZneeiDjt9VZRk75YM/VcezDKyyOuV8BzGAsR7EXG6mhidTLuYfQWEcVsxmX2h0XTIATboOlT0DN8t7Hi6fD3AugMELZvK8D/v19aK2hn8HT2QSv/AFq1sFx303tPP+HUMafgCiei171MbKrBmFxIIrnIormILrbK4jC2YbnI2CiH9kwIbe/C5MTFK2MwI7WN0yOTOe7jDRp4yXN7k04aOQT6BoUjjPaBHwDURQbBROuxN++Bk/zYrRQK6o1G1fePJzZM3cuDH2ZIspQ5f68wzI89DaltKJSQQljKGQCJWR2N0BcK6tSUiY9aO4ImofBKAKMxNujbwFvK/hawZEN7hgaHZ//Z7Dh0pelL8OkBTBh3+GY7f8Ewl2MOuXs6M8rKupe16AtvoPUn21mkMiqT5CTzujXEylRAuE2vCFz/doc1hHKtUqzk7Txkmb3RNdg4dPwxX+NDsIAqgVmHglHXQ2O4Slf3pUIoeLMmYUzZ5bpbSaKUiZSSrPsoIlOcnBT3J0XMxAfgWFZSiRgHe5LiSvX+ImFrsHSV4m7YL79d7jm4cjPdTRB1SojeXT0DMhOoXbN/xAiZzyi/DDkjg92zQTCPmSgE9myBr32Cwh5Ee5ilLKDIXeSqcogbyiOjEIfRmUdPJTZpkmCtPGSZvdDSnjtz7Di7f6Pa2FY/jZUrjAWlpp1xoJVPAH2Ox3KZ+368utdRL7IIp/YTS5zyaCOtp25KqlCIplMaUr3mRQBr9EPKR4tVUZ4ydnn/fJ74I27YO1HfapeBExeACf+CFwmG4im2YlScRxa9afdfamSOOdU5xByYgT6kjvBW09P+FV2bker/QJRegDKjIvjemUUzOW92dU8cpyTkpxnmmT55meZpdnzqF472HDZiYS2Wlj1rrEItdXC+k/hsR/Bs7dGbHyXxmA241JuuAgEFRRTFMXbM6LYHJjOPfD2ycfQQvDkT2HtxwPKdSVsXASP/whCyZWUf9OR3kb0hq/RG1ciw/3fI+HIRdn7uiSS3wVkjUUZfwJJ5ZIIBawu8DX2zLL7l1G6LWsWIivjtwZx28sRJgyYSYXpPkm7grTnJc3ux7I3jAteoo3vNi6Cd+6F478/PPNKAl3z4W1Ziq91Gbrux+oYhTt/PraM8SMualVGPjMZwyq2p2yfYyjkZOL0qBkpVCsUVUCDCeVgV07v/9d8BLXrI4+TOjRugxXvwNyTUzHLbwTS14y25jFoXtP7oGpDjDkKZeIpO70aSv505JRzkWufSGTvKBNOQbiKYGOi3buF8ROKLXKob3sbMeZIRAzDyqI4KMzYh4auJUTzHBW498ZmSXvldgVpz0ua3Y+2uuQ69oJh+Hh3XZVDX8KBRurX3kl79UsEvZWE/fX42lbQtPk+2qpeQI5ww0UhBMeyDwcyLWX7PJ652FLdGmAoHGWiDcDkA/onfq94K06ps4Dlbw55at8UZKAdbfHvoXld/ye0IHLr6+irH+33sLA4zO9ctaPMugKlcBbCXYwonkt874vo/fzsWYjSBcRd2oId3SGl2JRmH0GOoyckJPr9znJMYHTOMJXep4lL2vOSZvfDlU3S7QN1DbZ9DdMPS/GkEkNKnaYtD6GHuwY8Y7iuvc2fY3OOwl2wYETnJYRgAVNpkx5Wp8ADs9vd/VTsDbOPiR52tNgNvZe+dDbHabAojTFpAMNrQbCTnnN5ILJmIXLMkYgsQ81ZZI83t2OLC2XBz1CcvYrNouxgZNvmyN2kFRuU7Idw5BjVeBmliIJZyK1vIM1cPkxo0ShCpSL/TLqCO2jxrCCodWBVM8l3zSLDPjbdEmAXsttde9KkYeaRDKnEUjPZLRqgegfcdzccvT9MHwUT82H2GPjVTVATWwo9FoHODWiBJqJd4AE6Gz4cce9LDwcznQwSuCOOwqes3WWvISon/RiOuBKsA17fqMlwyV1QOLb/41mF8T0vmcPQAmEPREqJrP4strEnFPSahb1/ugqhYGZ8IT/Nj6x8b+dxtPXPon91V3S9GD0INQuROz5AFO+DUjTHCAPljI/f7Vt1gMtcCwohBJn2MYzNO4lJhRcwLu9kMh3j0obLLibteUmz+zFxPyieCPWbktu+JIJ4GRhNIT94GxZ/Bu3tsHYlrF4xeJynCx59AJ5/Cp56FaabL13uIdC5kXjeIy3Ygh5qR+2jnDtSZAgnF8rDeJ8VbKQm6f2sYBszGEMZMbRXRhohYME5MO802L7CqELKGw3FUTwAex0LW7+KsUMJe58wHDPd85BhCMep6JJyUG8hdcalaAt/ETsXRerI6s+Qk89Er/4MWflOzxMxDqZD2Ie24p+o839mKPzmTTUME28jkW8eBKL8UIQavQFmmt2ftOclze6HosK37uxfymqW8lmD76wBNq6DI+bCVRfCIw/Ac09ENlz60tUJ114Cepy7uAgY3oj4HolUV/8kQqZwcqrYn8s5knIKknJ2CQQriNN/KIU0dErW1uk0dpmYrMUG4/eFaYdEN1wAph4Co6dH9gwIxSjFn3lk8pP+JiEshtci5hgB9v7fXWHPgoJZxM1f0fxoX/8Nue5p83OSOnTugI5t3YdXUOdcA1Yn/Ze47mPnTqKx8CTeWhvm861hgtpu5jlMY4q05yXN7sPWrfDEE1BfD2VlcOJP4I3bIBTE1MrqzIZTfjL48dYWuOAUaO++GwzH7we0kx3b4LMP4eAjzG8DKBaniVECRR2+xo/hQAtasBXF4sTiGBXVzZ3hbePIuo20+rewtnAUW/KKCKvmLg0SSQsD83ris6Q9xL07fHzaGkIVghMKbVxd7mSiK3L1x6oanT9/EOLzbYYhKYADxiv86HAr00qGeA+mWuC838Fbf4NV7/fmQgjFMHyO+57R3DGN4dkoOwC548PooRmpo5QeMHhbexZSiPjdo/tWMJmfGbJ9GyK7wvgroxT1gF+i7/gQWbPI8Ba5itjkPpbvfzKD7W/r9HhlLEqI8+eq3HS0dfi6cqdJOWnjJc2uR9PgBz+Av/0NFMX40XW4RcItN8A0O2z5sne8Mwt8new0aFSr0XDvkEshI4IK6zOPGQZMvDh4JBQFln+VsPFizo0h0YLNKM5Ric8rBiFfHW3VLxHs6g27WewFZI06YZB6r79jHc1bDLVZFzpza7Yxp7aSusxcAo5clpeU4RexjT0Hibnf76708uP1HiwCwt1v09+2+/jHdh/PzsnixML+hsJXOzQueyJIuM/HJ4FFW3XO3x7g0W/ZmV02RAPG7oJTbjRyZapWG0com5bOdYmAMu5YtNrFhkEw6DslEMX7ILLHDd6uZB7atmj6TUNFwoAWGsKejTrxVJh4KgAbGnTOejDAQEdLWIfHvtSo7ZDcc1baSN1TSBsvaXY9v/iFYbhIaRgyWp8qgN/8Gf70J/jeU4Z0uysbckcZSbn+LrC54t8Vv/pCcoYLGHOyJhMbVzBXMZXaO72Qr47Gjfcg9f5Jy+FAEy3bHiV3zHm48uYCIPUQLZVPMjAvQJWSso4W6GjD5y5iZZYSM7w1jdGm5/dpa5AfrzfyHsJ9dqlJYxbnLu9gw0F5lDoMD4yUkp+9FiKsgz5gCpo0nCS/eD3IC1cOPfkYMIzfqQelZl/fUIQjF3W/n6Ktegja+4QMhYoYfShi/PHoLesQKJA1FmExvp8iaywU7gWNK0h9zyOByJ8ec8QN/x1suPTl3fU6a+o0ppeMXEf5NMmTNl7S7Fo6OgzjJJYr+be/heuug7I+d8GqFdxxet304OlMfn5SwmFHJbyZPWMCnXEu0IrqxuJIbUO39ppXkHp0leG2qhdw5sxGKFZ87auQMeXXdSqqlrFh2n4ERHiQASMQ5JLBZMpMz++vlT5UIFKRqgRCOjxY5efWiUY4bXmNZGtz9PdRl7CuwciDGXL4KI1phLsYy/43IzurkJ07QLFCdgVyyxvoH99sJPYCqHZE+eGGcJ1iQZ39bfTVjyLrvox9gMRmY3S2dkZPGm/olGxtib+nf3wS5m9nD7/xIqVknezgPb2ObdKDDYV9lTwOV0rIFb03SyHNg6b7sapuVCVFBvo3hPS3PU1iaBosXQoffwy15jquxuStt8AfR3q9pQU+/TT5Y0yeBmqSF6QDDoUpse/oImFzj8PiLCXWV0zXvHTUvjPIS5IsWrCNQOeGmGOkHsDXvgqAsL8e4sifu0I+TmxSyNKNC6ro/gcwilzO5SAswvx7+05zMKLh0oMOfNjS+35UtpjzmP3tk1A68XIXIDJHo5QugLzp6Ev+hKz+uNdwAdACyG1voS2/Hyl1hGpHnX0lytTzUnDw7u9W7iSUGRfHHLq12dx5tL11+M8hKSX/0Su5Q1vDV7KFJgLU4OMVvZqbw1+zSe/EE6hmY+OTrKy9mzX197O85i9sbX6RQLg1/gH+R0h7XtKY58EH4Ve/gqpu/RNFgVNOgbvvhjFjkttnl8lkz46O5PYPcOHl8M7riW83aQr89V9JHVIIQf64S2jadC9aqC3KKElXw3v4O9ZQOOlalHhVHHEIRz1Of7SgcQEUio1YOjQ92Go+4PgaqM8uoqt4LnZnGWMpYpQw6fnqZmFrkK4EhZOz7ObCau9v0Pnus0H+cY4NVRnZpEspJVVtEk8QSrMFWQ7j+J6gZOEWjWXVOjlOwV5lCvuOUb5xSaHS24T+xW9jlEFLaFyObFqNKOzOuXIOweOo2CC7AuHIRYzaH5E/LW6TRbfN3HueMQIpL4tkE2/qxo1f32+fBALo3BlezQmtX1Mc7sDa59lW31o6AluYUngpDmve8E90NydtvKQxx+9/Dzff3P8xXYdXX4UvvoAlS6A0ic7C5eXmxnUOIfRz0OFwzrfgmceNMs5YISpFhYoJ8J3vwmnngiX5r4jFnkd2+Rm0bHko5riwv5b26lfIHXN20scCkNLcBbon/OPInkFH7RumthFASXsDtL9B5qjjySqekvD8zloW3wAVwOH5vZfsBRUKGXboCsTf/8ebdd7boHPMVPOeoDaf5LMtGv4QTCpUmFUqEhIfe2+9xl8/DrGhwXhPLQocO02QbRf8Z5mONsA2HJUluONUK/PGfDPyKqTU0b66O24vIVDQqz9F6TZeRO5kUG2gJdhINWcS6uwrEI7EFu9pJQKnFXxxnJyX7Df8S+KbWk3UbDgJ+IXk+fw5WPUws3w1zO/aigUdkGh6gKq2t5lYmALP1R5O2nhJE5/aWvjZzyI/Fw5DYyP8+tdw332J79ttslT4ssvgzTfhxhthzpzEjiEE/PYumD4bHvw7VFUaj5eVw+HHGMZKRiYcfizkp7a6xN++GjOJu97WpWSXnohicQ3hWHF0a3Zi3KWq1lxTcxtIZ+0buHJmYbGbv3v+tDVIYyj+cRQBV5T1eqAcVsG1B1n4w3vxy9sVAc98HTZlvIR1yZ/eD/HEEo1QH2/QlCLB70+xMbU4fkT9v8vC/Oy1UL+U67AOr62OrvFT2yG54okgT1xiZ1bpnh+1l02rTPUIAr1Pl2cQFjti7DHILa/G3EpMPhthdYPUEDkTEBlJ3CABqiK46kALd30Y/TwqyiAhwzcZglKnkjgif92EFAtfu8pptGRwatsKFIzzqiOwhWC443++IWTaeEkTn0cfje2tCIfhkUeM8JE9Qb+r1WRTP12HZ5+F556DV16BY4+FzZvh7bchGIR582DBAsNQiYSiwEXfNkJIDXXG6ykeZTw+jOhhL6aMA6kR9FXjyJwUf2ykzaWOt2WJ2UkB4GtbZm5uEfA0f0F26Ummx7/WYO4O+4Bs685Kox4u3d+CPwx//Si2AaNLqDSZ23DrayFeXKENevUbGyUXPRrguSvsjM2Lfm50+iW/ecu4jU/0HQzrcPdHIR48f88vy5VNq4zcEzPVfLb+i60y4ST0UJehGTPQiHbko8y6HCU3ue9DJL5zgIXGLskTSwbHLkuz4b9X2Hc7yX8pBDvseWx0FDLF37Dz8UC4JW287OoJ/C/SENJoCesUWRXyLIlZ+lJKGroFL4osysh82bZti24U9OD3Q1OTIS6XCLNnQ3a2Idcfj3DYmMcppxjhps2bjb+FMIybmTPhmWdgWoyuyYoCJcndvSWDxZaHWe+GGELZdKBzPVIzd0dncxsKxCFvFYYXJvEy8lCfC6kZrCZtxFmZg78PQgiuOcjKm2vDbIhz2DY/BMMSmyX6e/llpcYLKyIn3+jSCC3c/1mY20+OXiL/2mqNYAJah32RwMItOs0eSb57ZBfLdp9kyXYdTcKMEkFZzhCN9wS6vyul/ZuQCqGgTrsAOeZw9JrPkf5WgqqOr2AMSkY52Y5xQ5vbAIQQ/OxYG99eoHPvp2E2NkoybHDpfJUDKkZmKbQJhXG4qcRj2ugVUmeVs7Sf8aIqe77hO1TSxssIstQT4I7adj7rMu5CFeCYbAc3j8pmoiO2B0KXkkebPdzX0MmOoHHBKLepXF2YycUFbprDOn5dZ5RVxZJqb0JOTn/tlWhkZw9+rLkZHnrICPmEw3DAAXDVVTBunPH8G28kls8ipeFp2by59+8er9DatXDQQbBiReJG1DDhyt+PrsaP4o4TworVZV4vZSDe1mWYMpKEFXuPdyeBKqGBSC1IZ8OHCKFiz5yC1VEUc/yCHCsQqyzb4Lox0ZWJz9zLyu/eiZ204AkY5a43HD74+xQIS377VojnlsU+lzUJr67W+OUJEpsa2bjY3ipRFfoJ5yWCBFq9I2e8BMOSP7wX4tmvNYI9AsLAIRMVfn2CjaLM5OYhsschqz+JPzCjDFG8T+R9uEfRVjqZqra30aQfgk3QshSBlTz3DKTU0GUQu5qPzZJFu28DAa0dq+Iizz2bPNd0FGHSgwuUZCn86oRd19foOLWU+7SNpsdLodCm9n4vrGoWTmvJcExtj2LPD7ruIXza6eeMjY0s6up1n+vAO+1+TtzQwPoYmWRSSn6yo5VbqtqoCvZeeKuCGrdUt1GxvJo5q2uZv7ae8StquGRLE56BmYJDYbTJRfWyy/pXD332GYwfDz/9Kbz/vlFe/cc/wsSJ8Pjj0NYG552XVO+giGia4cG5++7U7C8FWB1FZBQeGmeUwFUwf0jVRlLzY8a7k1Vy3E5vnSN7Gsl4XQCCns101LxOe/UrNKz7I81bHkIPRzdOjimwMSpOxce+WRYmu6PfT52xl0pBnBQpCTy5NEww3P+9kFLyg+eD/Hf54FBRJEIaHPZXP/d8FKLdN3iLTEd8lftYKAIKMkbGcJFScsPzQZ5c2mu4gPFefbpZ58JHA7R6Q7R619BU9wZtde8SDraZ2rcomdfd6yjGa3GXoO77I4QS2cBo9a6lsvVlw3DZOTOQhGj2LKPFu4o233rquxayo+1NOgJbCISb6QpWsb31NdbVP0zYpNdxd2B/kc9eiVTqSYmzj5xCadahu114a1eQNl6GkbCUSCnRpeSH21vRGSzOpQE+XXJLVfT6/Y86AzzdYnw5+14ve/7f13utAe92+Jm/pg5fqgyYL00KSj3/PBxzjOEZaWyEI480Spz7XuV7FHQvuQS+8534Gi+Jomnw73+ndp9DJKv0RDJLjiPa182eOZnsUUPrWqza86Puvweh2MksPqT3uBkTsThGxd0uOr2Jqf6OtTRuuhcZJfdBFYIX9snGHeVQ5Q6Fd+ZG8Nz1IcMuOHlmfG9RZwC2tfS3LJZV63ywUR+k0huLVi/c91mYcx4O0Ozpv+Fx09SYaq2xEMCRkxVynCOzAC3aZrz2SMaWJqGmXef+19/Fsegecla8QMaKZ9A/+imdy/+EHoxtFAiLA2XO1YYXL1K5cu4U1AW/QNgyIm4vpaS6/b04ryDaG9197oWb2dbycpx9DB9S6rT61lHd9j7V7R/SGajsbswaGSEE9TK+F7Iv0/x1CFRGZx9FvjvxLvffRNLGS4ppCWv8vradWStrGLu8mikra7hiazPVIS3qPa4GfO4Jsi0wOIjeEda5aUfiwkQtms6Pk9huED6fkUdiBl2Hzz+Hp5+GCy+EQIz6ViGM5NvhoHX3EnISQpBVciSjZv2azJJjsTpHY7EX4ciaTl7FZeSPvxyhDC2C687fn9heFEFmcf/+TEIoFIy/Aou9YOeY/r8TI+yvpXHDPeha5M99bpaVZQfm8d0xTnIshjxemV3h95NcrD4wjwwTiTFZDkGUSE4/Bt6YvrRCQ03iaqdLqG6T/Pat/gnHFfkKJ89U4qaCRcJphRsOMx/mGCovrNBivme6hLe3zsYS7r21UqTEUb+ewOJfI8Ox69SV/OmoC36GKD0ALE6j83RmOcr0i1Hn3oBQohucnmA1QW0IGk5ATwWOP9Q8xP0kjidYw6rav7G1+XkauhZT37mIjY1PsK7hIYLhyK9ri95FHeZu2hQJ2VJylG0as0q/R1Hmfqmc/h5NOuclhTSENE7d2EB1UNvpYfHoknc7zJ2o2wJhxtl7P5J/1Hdwe21H0l1AXm9PzLqPyAsvGAaMWRTF6FMUz1tjJocmWcxqx4wwimonq+QoskrMtxuQUsPfvgZ/53qQOjZXOc7cvQeFmKyOYjKKDqer4YMIexFYnaNwFwzu2aPasima+kP87avxta1E6kEsjmLDGJI6XY2fEg61YrHm4G39CqnHXshCvipatj1OwYQrIj4/zqny56kZ/Hlq5DvxeMyvULk7TtVRvgsq8vuv1s1emXR0UpPw1jqdpi7ZL9Rz24k2FBHi5ZUaQhihoLAOGTaYVSpYtG1wR6gZowS3n2RjfMHI3TfWd8o4XiJBcyhzkMkqAIu3BW37O1jGx64sExmlqDMuhjhKtwMJ66kL93QGKnFYo7cISDWBcDsbG59Elz1VZ70nmC/UwMamJ5hWfCWK6L/MvqJXmT7GWOHmOusUChKt4vwfIG28JMkmf4itgTDZqsJctw1VCG6pau1nuPRg9pqZ0ef26NVWL7+tHdodSVBCSEqsQ4mPbtliCLWFTZZW6DpsiC1RP6woipEQHA1dh65OcDjBtuuS9qKha36Cnq1IXUNYnLRtfwYt2NuUxdvyJe01r5JXcQmOzMn9ts0adTwWWy4dde+i973rEyo293ik7jeEwQYghIozZzbOnNmDnsspP33n/z3NX5h6DYHOdQS9O7C5Um9E7lUqmDlKsLZeDhKA6+GS/S1YBqjsjsoSKApRt4mHLmFjo05BRq8XwWYxNGGuP0TnnXUaXQHDaDp6qordImjqkny2JczmJklJlmBBhUpFfmqNlkbpp0p6saEwSWQRCgpeXa2xqlbHqsLBE1QK3RJVENOAsYowIV3Bqgx+g7QdH/YzXqSUtPnW0di1BG+oHiFUcpxTKMqYh9OamHKuTY0dKtydaez6sttwiSw3Fwi30upd2y/ME5Q6y6Q5z/BJopSzLGNTM9lvIGnjJUFWeYP8X1UbS729buQii8L5eS7eaPcn7SWxCXiksYtcVWGCw8pvatpSMt8hf8C5uYl5SYQwjAJFSU0irs1m5NCYQVVh6lS49trBz3W0wwN/hScfhvY2Y36HHwvf/ynMGLxojzRSanTUvIGnaSFSxq6okXqQ5s0PUjT1R1gdxTsfF0LgyJ5FZ8NH9Ks8kmE8TZ/ha1tG4aTr+oSJetHDXkKBBqPqyTkqoty6asvpZ0hFR8HXtnJYjBchBPecZeeSxwNsb5U7X2XP4nzyTIXL5w8+60+fbeGxL4fm7bNF+TKNzlG4bP7g96sgQ3Dq7OEJDzVKP49oW1gleyUGbLrKjs+L2PZZEapiFN4/tVSjJDO24QLg0e38aPPl3DXxXyiid7AAlEAHMuxDWJxIKdne+jrN3uXsPMckNHuW0+JZwfj8s8h2TjT9OpzWIhyWQvzhxviD45BhT75aLxlavKuJnSQvaPWt6W+8YC5hHGC+klrBzG8a6ZyXBFjrC3Hapka+9vZfTBvCOnc3dA2pyXtQwvNtPg5ZV88p6+upCg194S+3qkPPSj/zzMS3mTdvaMfsobDQvOFitcJFF8Enn0BmZv/n2lrhzKPh3rsMwwUMw+q9N+CUw+DXN6eu4ikJpJS0VD5JV+PHcQ2XPlvRVvXizr+0YBtdDR/TtOkf3QbGwLNRooe9tGx7st+jWriLlsqnqF39a5o2/p3GDXdRt/q3dDV+Nijp0J0/H7P5MEb10/BQkiV48Uo7vz7ByryxClOKBEdPVfjXBTbuOCVyb6NpJQpnzkm+NDzDBrNG7R6Xy1YZ5DfhVazpY7gABBWNogNqGXtENZreW8bd2IWJPCGFD9pm8X7r4GRQAWjvf5/wF7+jY8dL3YYLDCwfkOhsaX6esG4+zCyEINsxVCE6hQzbGJzW2OX6qUaP+1015Pz7YpEC1eR36BM5dIPum0za85IAt9e0E9RlksWl5lkarwGHSW4uTYFLNi/PUM01UxVkscCkSXDbbfB6Eo0QBzJrFnz0UXzPz7PPwhFHGHONxK9vgi2bom//yP2w9Au44lo46nhwmWxZkCKCnm3428xK+/fZrmszuh6mo/oVPM2fE79UWifk20HQW43NVYYe9tK48e9ogRb6Bjf1cAft1S+ihdrJLu2tgnIXHIC39WvC/njdxGVCrQPMUuXXeLDKz4ctQQSCw/Os/O4sB6UOlS1eje1+jZVdklkZkY32Xx1vJaTByyuT88C0+aAoM/644eY1vZouQhGvQ0JAyb5N1H9dgL/VyIsyWxWloPNs4wHsl7WRl5r2493WvfDqdqa5dnBu0WfMYBvu9q0UFOfRlB9Z3VUSptmzguLM/U2/Hl/YTHuB6NjULMblnzro8Q7/Vhq6luANVgMKOc6JFCYR2oqG3ZKLLxRr7goOq+E9qZQeXtaqWCrNeC4NvtSbOV8dN7RJfoNJGy8maQhpfNCZfFhopPl+UQan5ibfJ2cn77xjvpz5qKOMNgE5ObDXXrB8edxN+iGEEc6R0tCDMWO4AJxxRnSZ//Z2eNlEVdOqZfCD7xiGy09uhYuvTGjqQ8GQ9U9G6VbSXvUi3hZzuSg9hHxV2FxldDV+jBZoJprR09XwAa68eVgdxsVeUe0UTrqGxo3/IOyvi34AoeDMiyxI1hddlyz6WLJ6BTgccPRJUBKl388rDQHOW96BJnvlBha2hfj9Vi9T3Cqr+rSrnuJW+d0kNycX9U9yVBXBHafYGJsb5J6PNRSB6dJpTxB++EKQxy/etYmTupR8ojfEPFOkDgUzW6j6JDElaR2FTb4STl55C61hN0ZQTrDRO4oXmhZwVembXF/2BqPqW+jIdBK0RQqJCTzBmoSOq+nmri8ClWznFIJaG6FwBxbVRb5rNvnuvQYpzta0f0hd50L6hk+bPCto8iynIv90cp1TE5pjJAoz5rK9NdZNmk6Be2/W6u38SVuLnuDqERz22+Q9m7TxYpKGkPlY5e7ACTkpMFwA6k3eFf3970auia7D6acbKreJ4HTCCScY0v5XXGEo8G7dGj9ZuKQkdn+il55JTE3M64Ff/RTefAW+fR0cepSRSzOMaKE2khWLS9RwAUCoSCnxNC0iXsy+o+5NMvIXYHWPQVFsKKqTwonX0LDhbrRg64DtjYUiZ/TpqJbY3quli3T+clv/avqXn4UJk3V+8Udwuno/042eMOcu7yAs+x9NxzA++houABs8Gmcu6+CxWZmcO2qw8N+1B9s4YLzOk0tCLN0hqWmPf35IYOkOnTV1OtNLdl34yIdGwMS5Ys9KxnsraQllIhHIPhkFGsb5f3/NcUx01nJs3jLyWjupK47s6RQJZiM4LAV4gtVxx5VmH0px5vy449p9m7oNFxh8xsC25hdxj7oOmzo0N1q+axat3rV0BrYR6XtUlLEfDmsJ94WXojG48iwWAigXiV/D1wR83N/RQrOuUahauDIzj+n25MUvd2d2jyDuHkCeZeTfqqEcsTLZxisDKTV599aT5/Laa/Dyy4kZDIoC++4L//ynkSB8wQVGz6OtW2MbLqoau7IIjHBQMnzxKVx5PhyzP1RuTW4fJlGtWST3aSeTzySM5o8yjK554oyV+NtW0LT5fupW/ZqO2jeRUkOxuCia/H3cBQsQfWTZrc4y8iou69acic7aVTq//3lkGaDNG+DGa+iXb3PvDj865hsg9kjnfXdtF4EorpU5ZQp/ONXOe9c7KDRZta0IWFw5jCX+MVhcqfGD5wOcfm8QPRz7c5cSQp7k7kvDWHYaKwNR0Hm49kgE4PBHM44kWY6KhI5Z4J4Td4xAJd9lLrG+oWsxsb4bEkmzZ5m5ycWak1CZUHA2JZkHoiq98v02NYcxOcdTln0kK2Qr7YQSvvGVwFGK+RYAIV3nkvrtXNRYxacBL2tDAT72e7iocQdXNOwgNBQ56N2UtOfFJKU2C/PdNhZ7Rs6Zd0aOk42BMMuTyIHJSUaRKxL7mRBFEgImTDD+/8ADhlGRSIWSrsNJJ8GUKUZzx769iqKhqlBRAd//fux9D1UfYUclXHgKvL1o2HJhXLlzzXeE7keiFySBM3cfVGu2YRwIC0hzRq7UA3TWv0c40Ezu2AtQLC5yRp9OVumJaMF2FNWGajWXY3X/n2M/X1cDixdK9j/QWIBebwwkpWbbGpa81hjkjOLY58AZe1n458KwqRDSSK8BUkrufD/MQ4vCqApousCyJpfCmS0RBW0BFBWaVkfJ/4qBwyII6dHLyXUU1njH4NFs6BGSokFgUVzkuiI3RpVS0hnYhj/chCpsZDkmYlXduO1lFLr3pdET7TsgmFBwNhZ1sCfCE6imoWsJXcEdCATZjol0BaqI/d2QdAV2xHjePIqwUJp9CKOyDuwW21OwqVk7c66qdG9SAeGDRCH7CPOf4XVN1awKRdZgWhb086OmGv5auHv0e0sVac9LAtxcmo3CyL1pb3X4aAlrzHFY+HaBm7NyHKbutXNVhf0zUhSb/+yz+GOkhA+6xdE2b07McFEUI1fmr3+Flpb+q0Nfr4urz4VLVeGss4y55cbpETJjL/NziYSmQW21ubyZJLFlTMCeOZVklW3jY+zXnjmJnPIzjEeEwJW7N4mezb62ZQQ923b+rSg2rI5C04ZLMKBTbWLdePHp3v8nK8OvAHdu9XLOsnZu2tDFek9kQ+2ieRbyTdiluoR9yoc3hDiQ11ZrPLTImHePUVGzqBgtqBCpE4OU0LQ6F29j9AaXkXBY4bBJ5s4FiaAjy8XA89WiOJlUeH7EJoldge2srvs7m5qeoqrtHSpbX2Nl7T3saH0bKXVG5xxNec6xqINCJYIc51Rsas6gfdZ3fsH6xkdo9a0lpHUQ1Npp9HyFxIxBntqruBAqdksudkt2v2RxG2pCtxhOFC5SKrhcnWC6UrQ2HGJpMHbe0GcBL01airzxuwlp4yUB9nXbeWJCAWW2/hcwm2BYjJpOHXaEdJb7wzzY5OHjrqCpL8L1RRlYUtW4K5bEf6RxBQWxc1D6oqpw8slGQ8fq6uhGjxAwapRRtn3EEXD99UZFU5GJ0sjTzzUE6YaCEPDGS0PbR8zdC/IrLsaVN4+BZ5FqH1r3WEV148rdh/wJ3yF//LdRlF6Ruoyiw7vbEiRyrih4W0z2uopAq8lii84+VcAH5VqxJHE668CXHWFeaAhy1zYfMz9r5WcbuwaVgOe7BU9dYqcoRvhIFTCjRDC7dGQb4j20KMxAJ0egzc6aJyfhbeqfy6BrgrqlBWx5Y0zCxynNgv3HKjFF/AQ6FY563O4sSid+j8KMfXHbRpNpr2B0zjHMKLk6YrmyN1jHxsanCGoDu8frNHqWsL31TYQQ5LlmYrMMzEMxBPHWNfwLT6A3L6YzsL1PTyS933gzmAlt6VKjw7+VVu9avMG6mP2KorGPkht3Ri5UzhFjuF2dw73W/TlSLUFJ4Pr9VGebqXEve9rjD9qDSIeNEuSgTAcLp5WwsCtAZTBMlqpwRKaDprDOv5u6eLzJgzfFvuWevTWH4zsfK2wqVxdHLmNMijlzzI3be2/jd4/Wihl0HV56CbZvj52YK6Xh0amsNP7/0UdG5+if/hR+97vBjWz6kpUNf3kArr/UOF4yn42U4OmKP24ICMVK7pizyRp1PIGuTSDDWJyltFY+HX/jiCgI1UHh5OsjitIBWB2FFEy8mpZtj3drw/QRtouKTtiUUF1kskxW72fl9P7/2jFOHq81aURHoccsvmOrj3KHylXl/Q3ashyFt69zcPFjAVbUyH7vhAAKM+DuM20j2s3XH5KsrY/8efianKz69xTcJT5chT70sEL71kzC/uQu6VubwRuCLAd0BSJXYUkEF435Csu8H2K151Nu0rCu6fikWzo/8mtp9i6jOHM+DV1fRCk9lugyxJbm/zJz1PUIodDY+SXJVegZuG2xBe0au76ipuMjtD6aNQ5LIWNzT8BtNx9+KRQO5ot8vpDNUb9ZF6jjOEhJXqOmXZrzdLfvQi2r4SDteUkCRQgOynRwYX4GJ+e4cKsKY+0WflGWQ5GJ5nLJEusUFRiN7u4am3isOyYTJxrdoWNV3IwZYyjbgtGQcZJJ0akeQ+Lrr82JxIXDvV2pAe64A+65J/52x5wIz70Nx59qGEkAzm63t5nFSLXAlBnxx6UA1ZqBK3cOrrx9CftqTWiqdG9n6/u5CxzZMyia/L2ohksPNlc5RVNvJGvUCdgzJxrdgWMikJqf1h3P0Vb9MoGurQndkTpdCqUmhFDPvKD3//OyrfxhshHXMdOUMR6/3+JFizBnu0Xw+MV2fn+Klb1HCwozYFKh4EdHWHjxSgdlOSN7uYz/rgo8dS4aV+bTvDY3acOlh2e/1vj72TZslv7vsyqM7+Zpkzs458xTEU7z/YM03U+HfyPxXk2zdwVNcZJoQ3oX7X5Dr6kzUEmyhgsItrW8GLUDen3nYna0vdnPcAHwh5vY0Pg43qC572QPl6sT2FsY4W2luxZLdP+crYwZkuECMM1mLkVglu2bVXUkZDK+sN2Ijo4OsrOzaW9vJysrhR4HEzQEw/ykqo0PO/yEMSxBmwD/ML+jxRaFDk3i6/PRldtU7izP5aDMYThBKysND0xbW/Qx3/8+3HWX8f877oCbbjK/fyGSz4QsLoaqql6jJB5SGsaPxQJLFsFtt8DKr+Nv99IHMHOI+TMJ0rDhHkLeHcRfxhRKZtyClDpS86JYs+OWKvcQDjTSvOURwoF6jDO4p1YnHj0LuY7NXUF+xaUoFnOlnatX6PzyR9GfHz0W/vLgYEPhF5u6+N0W36DZ9dx/m/Eb9fD1glxmZu7+jufTH/SzoUGa1qMZCoqA1f/npLpN54klGm+uDRMIw+QihQv3tXDkZCVhz1Mg3M7qur/HHZdpraAzFL+qrzjzAMqyD2N5zV8GGReJMj7/THKcU/o9pul+VtTcjYx6qyjItI9hUuGFCR9vm+xikd6ER4YpFA4OVgrJFUPPTfRLnUOqN8e8uXUg+KRsQkLhqJFgKOt32vOSJNv8YfZfW8e73YYLGBfQ4TZcAOrDOidkO7h/XB5/LM/lmQkFLJxWMjyGC8Do0fHzWO6+G2q6xalefNGcR6OHodjP9fXw1VfmxwvRa+jsOx9efA/eWGj8f9DY7td83Y9G1HAJerbTtPlfhLzbMbMc29zjCHq2oagOrM5S04aLobB7L+FAjwx5IgXJOj13vkHPNpq3PmLaAzNjtsJPfw2R5CemzIA//GPw45+3hSIaLj3kWwVT3eYvZ/6RsAZSwOXzLSNiuAA4u/Nsy3IUbjzKyvvfdfLZD5w8fKGdo6Yk12rEqpgzaP1as6lxobCHrsAO7GoOQ0twV+jwDzaWWn3rYhguAJLOQCXBvo1PTTJOZHCeOo4L1QqOVkrIYWiNYfUwbP8Utr+mcH19ccyxv8gr3u0Ml6Gy+9967Kact7mB4C68/r3Q5uPOMbnYzCbHDoWFC41KoHj8/Ofwr39BQ8PI1pT6hnYHxuSp8MTLRpuAh+8zqot6Hr/6BjjlrCFP0Sz+jvU0b/kXiZRBBz1baPFsQQgrGcWHk1l8ZMTGigPxNH+BHu5K6FiRkQQ9Wwh6K7G7xxHyN9LV+DG+1mVIGcJiLyCj4EBcefO6E4Rh3wUKj74kWfSJZO0qcDnhmJMhvzDyvP+01YsqIBxhqjrQHJKMcZj7LtgVmOwe2aqhZDlphsqqGp1Hv9TidoYeiNrdRbvnd7yxJ81I/XsSv/+PQdikym5HYCMtvgSVu6My+E0JaV2YyaUJ6V3YSMxT8IXexOtaDZUY+kolODhWKeVQpShhw+KDX8DCP0J456Uvi3OmuPnsnhqqp/W+l4WKys9yiznIObItT0aCtPGSBOt9QXakoHHiUNCBp1u8XFxgUmFrKJgxXAAWLzZ+jxsH27aNTLPDnk7SQ8VigSuug8uugZYmI88lJzcxD9IQkXqYlm2Pk6wxIWWIzrq3kXqY7NLj4473tn6d9LEGI/C3rQKp07T5n4ZGffcCEPbX01b1PN62FRSMv2KnAaMoggMOFRxwaOw9Syl5vSkY0XDpIZHUzYtKHWTtAtHJZBBCcNPRVg6bpPLAZyEWVZr7vO4+08qS7To17ZJsp+CE6Qq/fyfE1ubBBpAiwKrAJfunfjnoCJgTeDQb8kuk6WNsdNy2wV3PrYobM2eSMc48/9W284pe3c9XVIefR/QtbJadXJFAafTzF8HKxwc/7l+vMu/Ecn61OIRvSoBxFjul1uhdzWuWwPbPwF0AM84FZQ+zBvaw6e4evNNh7i4hRxHYFEGHpg9LOOnjDv/IGC/jx5sbp2lw441GKMes4TKUfBeLxehrVBzbZZoQigIFI9udtgdP82KkyTvQWHQ1fEBG4YHdyr3R0TXvkI/Vi0TX/bRsfQSkRqSlKNi1iYYNf8WRNRVX3r5YHebeZ0lkj0tfdKDUrrDZq+HRoi+EORbBkvYQsz5rYVaGUXV0SK415VVEmpS83RTk1cYgPl0yO9PCxaUO8pJI6BdCsKBCZXOT5IvK+GqtY3PhmKkWjum26aWUCCF46EKV6581qqlUxTAYwjrkOI1Kqor81Bt00ZJiB2K1ZBEIN5nZ49AmBIBAVRwRxfRyXFPZ0fZ2zJyXDFs5NovJsjlgi97JK7rhzY00+09lI3vLPOb2EaVr2wZfPwQtG8GeA+OPhrwJ4G+NbLj0oIdg6dVWvr0outGy9X147jzw9mla/cIlsPflcPIDpl/WLidtvCSB1eR1rk2XoEv2clhY7k+9QNDnngCalKjD7R2YNcvonBevQePatbB+fWIel2QNF1WFsrLeJOFvAN7WpSnak8TXupyMooNjjrLYCwmGOkiV90UPd8U1iML+Wrr89UbTx/wF5Iw+LW6ISxGCmRkqq7u0qPfECrB/tpUfj3Nx8lft+PXB989WAe1hybJOY2Ha5NV4tj7IdeUO/jI1I2UGTI1f48Sv2lnVpWERxrv7eE2An2308PDMTM4uGd6qj0v2s6BLyUsrNB77Msz6BolFgUMnKdx0tLGofbRJJ6TBzFGCI6eo2IZQxhUIt9Hk+RpPsBoFC1mOCeS5Z2FRHLhs5m4sRmUeRFXb24RlKg3qyCjCwoT8s1HE4OXPojgZlXUINR0fRNhSIBCUZR+e0PE+0OtjegYV4D29jrmKYbx8cju8/zMj5U5KY8Ol95k/XvUX4GkAd4R7g+2fwqNHMegrLzX46p/gaYTzXjB/rF3JnuE73c04M8FuzWYMl2QuHW2a5EtPMIktEyQcNh8+GYlQUU4O/OAHsGSJIV4Xi1AIOjpGZl5DJOxLrAQzFr72NXHHZBQsIHVho55+RGYuKcZn4W3+nM66d03t+7tjnTGd+YqAy8ocHJJnY/VBedw03sXMDJWJToXzS2wU2wS67P9qe7w5f9/h5+HqoXu8wPC4HL+0nXUebecxtO7jBnX41opOFrYl0zQR9hurxP207BY4eZbKjS8F+b9XQ6zvrlQKavD+Bp1vPRqkpl1yw2FWfnKkleOnW4ZkuDR7lrO67l7qOxfRFdhOR2ALVe3vsLruXrzBWpzWIlzW2P3RrEoGua5pTCw8HyXJ6huBgtNqTnemIv9MMuzR6/WLM+czOvso1AFzsVtymVh4QUI6LwCV0hPz3NWBHdLIg1n+KLx/CyC7HZhJXrY6a43voz7g5vCFi4n5lV//IjRvTO6YI03aeEmCAquFua6hZYoPJNklpDk8Ao3i/vjHoSfFHnjg0PNHVq6Ezk5objbmVBBDw2TFCjj/fKOtQHY2FBbCLbfELvfexUgzYlPCnCJu0LMJX8f62INUJ4jUncdmq5z60tX4EboW3wC/pNTB2cXGXPtetCzCeDcenJFJqcNIOB3tUPnVRDdfH5DH2oPzOaPYQX0wViAA/rTNl5SC6kDeaAyyxqNFDHNJDCPrj1uT8y5MLlLYt1yJqnWjCLhgroW31+m8ttpY9fpWKmm68fePXwwx63c+pv/WxwF/8fGHd0MEowhgSqkT0jxo+uDPqCtQRWXra0Qqr9d0PxubnkbTA1Tkn4olStWRImxMLrrEaFdhK2ZmyTWUZh2G01qM3ZJHjnMa4/POJt45L5E4rYUxx/QQCLfS0LmYmvYPaez6alCysBCCosz9mFX6fSbkn83Y3FOYXHgx04uvItOeuHqxPUqjy77YUJASPr6NlHQJ+aNax/zqTcyr3sQ5dZU839VOW7WkzUQK0ns3G78DnbD47/DQwXDvbCPUtPWDke/vFY102ChJnp5YwKFr66jZxYm7o23D/BGGw4Zuy1BQFCOcFO+sF8IYG6lNwK23wsyZ8Y+l6/CnPxk6M329LS0txut4/nmjJ1JeisX8UoDFnt+nbDkydncFQrXhb18dd38dNa/gzJoS8Tlv20patz0adVtn7r74O1YjNXNGq9U5GlfuPgm3DpB6kKBnM46syM38elCE4LHZWRxV7eee7T5Wd4dkji+w8cNxLg7MjR7j/7A1hFVAKMrpJ4ENXo0PW0Icnj80Y+7lxiAq0QUlwxJeawwS1iWWiM0NY/On021c9FiAHa3Gi+kxiHQJC8YpfP8wC+c9HIibANtjq7R64eEvwjy3LMzz37YzuluIT9OD1HcuotGzdKeeSqZ9HCWZB5LpGAtAQ+cXRE+1lWi6jxbvKgoz5jKt+NvUdy6m2bMMTfpRhZ0C994UZe6PVe01ei2qi5KsAyjJOqDf3nJ8U2jzrY/6qkRII299E0r9FkKZDjomliCjNKetanvb2AaBRKeq7R3Kso+gKHNev3GKsJDtNCm4GYN9lXw26Z1RPw8F2E8poGWTkeMyNCT+Ao1Pcjp3noNbwkF+29bA4kVhnMQXGGzbZnhf/rUAfH0q2BtXw+r/wD5Xwkn3EbUx6EiRNl6SxKUofDGthIeaPdxb30mdCen+VCKAiXYLs53RL9op4a23oH2IPTGkNBe2ycyEyZONcFBfhIA1a+DLL2HevMjbgtFf6ayz4NVXIz+vabBxI9x8M9x/v/n5jxDuggNor47dQymr9ESEYjFlvIT9kaTWDXdya+WTMbf1tS5BseaZNF4E2WUnYXOPx+IY1X1c898HqZsLo6hCcPloJ5ePdqJLQ8LfTJ6KWY/KMUvbGeNQeGhmJofmJWfEdIb1mCohYCy/IZncxbcoU/D8t+28vFLjxRVhWr1Qnis4e28LR01RUASsb5AJe3I7A3DhowHeu96BIMTGxsfxhurpayx0BirpDGxjXN6p5Llm0BHYsvN5TQo+6yrns84xhKTCVGczJ2RvoN2/mcKMuVjVDEbnHMHonCN2JhADEArAsvegrREqZsD4WQD4Q800dH1Jq28tUoawqTkI1O5E2v6vrmDJFka/vw7F7yeT7kRkp40dx+1F66zBFUU928udvzWq2t9BVezku2cn+M7F5yClkFf1KjyEB30rBGBF4UilpE/Zc7IYr2jNTU39zsGed2thYQdHmjBeQl64f47xu9/euyf/1T+haBbs/92hzndopI2XCOi6zpMtXt5oN0Sxjs92cmGeC2WApoqiKHy7MJMzcl3MWVUb96KVKnqaQN4+Omf4e62sX28kxybSKToSs2cbPY+i7cdiMUqsl0ZIWpUSnnvO+FFV2G8/+N734Nxz+4eibroJXnst9jw0DR59FO680zCWEmHb1/D+g9C8w7jtKJsGR10NhWMT208U3Pn7421dFlWczl14MDZXd7xd2EDGD7f0Wyi68bYsBRk/D0sPdZfIC7U7AB8ZoTqw2AsQQlAw/gqaNt8f14PUD2WwoVAb0HivOURQl8zNtrLXADXcRHQxDsy18o8d5nJatvt1jlrSzk/HObltcuKVfGZ0WKyAHpb9NPjrOiRPfxXmw40aQQ32Hq1wwVwLM0YNvr112wTnz7Vw/tzBl28pjUqiZO6lGjrh4006U4s/H2S4dO8dgMqWV8lyjN9ZSVQbzOD724+nMpiD2r08v94+mb/Vz+PPFWuYOCC6K4Qwbmae+Qt8+Azofc4tVya+865m3ejKfoaKP9wMSFThQJN9dEy+qqL89WW9++7+rfqCVLzwJVIVtE030Y8CqGn/iDzXTFMaSYngFhZusszgzvBaWgnuDHtalrjJfaKYqU15fD3GyoxzQHWANoT0q2COjmd85JsB74Qw4VwNS2vsMFbT2vjH+fxO2O+6Xet9SbcHGMCXXQHO29w4qLTZIeDJ8QXsP0DFdlsgzFeeIP9u6uRrbyjpbhuxsAsI9JnPLKeVX5blMD9j6NLScXnwQbjyyqHtQ1UNDZj5840w1FBPuZ7y6u98B+67z/h7xw6jp5LZLtjLlsFeCajmvnonLH8r8nPHfhf2PcX8vmKga0E6697C0/wFUjdei2rNIaPoMNwFB+w0ROrX/Ymwvy72zoSFsr1+N+jhlsr/4GtdEmGDZFFwFx5ITpnxHkg9jLdtBZ3176KZMGKE6qJo8vex2PPwapLvru3kiZpAv5uB/bMtPDIriwmuxIXUgrpkwsctNAbje0X68sisTC4YlVhl0PnLO3iuPv45OGO5m3PnWLjhMCsrqnWu/k+QoNabo9IjLPeTIy1cPj8x7+p1zwT4aJOekKBdD2fPUThz73/Eld4fnX00rb61tAVqOW/TmdSGMtEGpFAKJKqAhfvnsnfWgNfwz1vgy8Hfp54pbz1jP9pmRjY6ijL2J8NejhoWZPz8OwjvwG7VvfsKZrtY/b1jTefbTS68OGYy71AIS52lsoXVvnaaLinG998MFItE6gKhGIq59hwItCV/DIkEAZ8/Wk3TYYM/wwl/z2HG7wtIRWLN97dCzrih7WMo63fa89KH2mCYMzc1RrzA+SWcvbmJz6eXUGazUB/S+OH2Fj7sHFq3WzPiTG9PLiKEoDGsUWJVmewY5lBRX045Ba65JnrHZzNYrYYBc8EF8NhjxoWkJ4zU0026rAxqa82Fl3qMnwcegAkTDAXgl19OzChyOuOP6WHZm9ENF4C37oGKuZCfWBVCJBTVRnbZyWSNOs7wXghLt1ej/8KQVXIMLTFyVgBcOXOiHiO16HibF+PInIan+XOC3mp0zQO6ue+G1Px01r9HTvlZnL2snXebB98ELOkIc+jiVpYsyKPEntjtnk0RvLRPFscsaacrHFv8vS8/Xd/FeSX2hLw8dsWcYJ4vBI9/qbFsh8bGJgho/U/fHkXcP74XZlqxwoIK80bbFQssfLAxuSpEX0gz0TNIwR9upChjHi80LqcqFFnzRHYvkH/Z5uPR2X2uWbVbIxou0Hs9HPvKUtqmlxkJPQNo6FqMECpFm/1RDZeefdnbvbirWvCUxw+XVJPFQo8f3dfEOKuNo50ZOFOoYG4RCvuLAhp/VEBVdzmyHjZeX09IZiiGC3Tn8UjJvGtG8cbaLf2eszeqTHgod2gH6INJCZ9hI11t1Idbq9tiXtg04OdVbXRoOqdvbOCTJAyXnq9itiL4blEGN5ZEtzZV4OgsBxOdNqY5rRyS6RhZwwWgqAiuu25olUJ+v9HY8bHHjL913TBaiovhnHPg9dehujrxcmZFMUJFiRouEyea73wN8Mlj8ce8e6/5/ZlAKFaszlKsjqKIbmxnzixs7ujigYrqJqvstIjPuQsOStU0dyL1AM1bHsDfvtoIN5k0XAx0vK1f8X6zn7cjGC5ghGOagpK7K720eCSeBHtz7JNlZeWBudw83kW+SaGmuqBkcXtiRvvBudbYhosEm0dBkUbp9opa8Iein76qgH9/kdgc9ilXuf1kK6oSce2PyfQSc0uCIqzkOKfyhW9vlBivOCzh+foBcZB3noi5bwGoIY3MzZFztkBS3/k5tdUvm5qrxdP/XNSB1RTzNHvxBHvzIeP5GwfwS47lEa/kia5WftVazzG1W3jf12XqGGbpqoev/zW8C79AYO1SKXrPqPBy1KoUv+Nm1q2F2JtUUuF1sWdBduKFVykl7XnpwwcmlHM/6vTzZLOH7UEtqfJmAZya4+Tv44w7ASklLZrOPxu7dlYp9Pye67bx17G7QVXMnXcaein33ju0kE9f40RK8HrhV78Ce5Lhr2S1W37+88SMsY6G+GOq4uuqpJqCiVfTXv0inubFfXJYBPbMSeSO/RaqJfL7anUUdifWpk5XppckPxMZ5rEaL5Yo/YvA+E7ctcnHi88aXoj54xSuOtDC/HHmvBKj7Cq/mOimzKFwzRpzi1JrgtWE55U4uHmDh/awjPxOCMit638DEusbpUn4ojLx9/S02cb78tyyMGvqdASSjzfLmLkwVgVO38tOQ+dYuoKxmoLqZDunGCFMpRid2F6eoIS6ji8oydrfeKBhh6nX4GjqoHNSNO0WSSDLnAcxmN3rZW3GyV85mBqyUdARgNZnMe9tagFeKflpcy33F45mH3sCntoYbH7LCA+NBOPvzWHs49mUvOdGyNTmRs65fNe3E0gbL30wczMXkPBksydpXRYdeLXNx21hnVyL0WL+l2U5nJvn5qlmD9uDYXJUhdNzXRycmZjLetiwWOCGGwwF3Q8iKU8mgaYZ2jF33gn33GOULpvtoTQUfvc7uPji1O93F/hQhRDkjD6d7LJTCfkbQA9hcRShqPGNwcJJ11G/7k/oodYRmGkvrbqDx31784RvL5p0N1p3weoktQXVEb8NQLiPnbK4UueLbUHuONXKyTPNX8rOLLbzvbVdUUun+1LRnWPzdUeIJ2oD1Ad0yhwKF5c6mJ4x+Jhui+DFfbI5cWkbPq1PybQEBGTXWchoSeyyKyVoukRN0I1SkiW4/pBeQ2lZlcZFjwUjGjBCwB9Ps5HjFKjiQDY2VUbZq8BtKyWjuy/QrEwrbzSFYmjoSMba2qjpeI9c11TslmzIMndDprlin8edFYUEMx1YO/0RfQkS8Bdm4SvJAaAdO7dxFF0Y+9VNBB4k8M+OZu4tTE0eTChVrZlMUPiF4XkRqRCO6YOrAI75Y0p3mRTphN0+TFxehS+eFAmQoUDnENeqtyYXMTPFQnfDxubNRolyR8fQq44G4nZDV5fhDbntttTuuy/Z2fDmm0bScKL8+UzwdcQeM3oGXHJXUlPbVUgp8bUuo7Ph/RjJvwKhWLFnTsbfvmpIx6vWMjmr9ULq9IzufIi+F1WJGXe2GhRULO8vhmdT4aPvO8hxmr9I/36Lh59vii4WJ4DJLoXD82280xRks083pP6lsdCHJVxT7uCuqRkRbzCq/BoP7PDzj01+PGGJ3auQ3WDF2aEmtZgowJFTFL69wMrsst5FV0rJYtnM21otW+lCQTBT5HCcMoqpyuBclDavzi/fCPHxZh1fyAgrHTRe4dqDrezVZ7/NnlVsb30NvxBstRfgU6xkaj5mSAdT8s/CohoLY6VPY/InLTH8bZIbSz7jzLx1lGQeQGn2obB1NfzukpivVwpY8ZOT0Byxr5GO9Y1Me+ZTQ++mz1ImjTp6Nn7rILrGFbKKIv7GQWgmBOMi8WLJWMotQ79eVy0y9FP2ZC75CMYdkpp9DWX9ThsvfThiXR3rTUj5ZynQMUTj5bNpJYyz7yGOr7PPhhdeSL3h0kM4DMEgTJsGldHu+JKkqMgoq77mmuSF6T55DD6OnRzLBX+Air2T2/9uQMCzjeYt/0JqfnqNCIlQHOSPvwybeywdtW/S1fipqTLrSJzTej5LQ2WDqlJMIyGv2kZebf9FRAA3HW3l4v3Mf5+klFy7posHI7QF6H318RPqfznBxS0ToisLv7IqzI0vRdexERipW7puTmVbAH89y8ZRU1SklDymbeV9Wd9vnj0Jw5co4zlcjd5bSNMlioAG/LTIIJnCShlOhDCSPl/RKnlZryUsQEiJFAI3Khep45mv9NY//2O7h++v86Kg9/FmGObpAvcO/jjmbSwCcpxTGJ9/hvH07ZfAtshaRRJonjeV7cdPj/ledCp2ns3bhwlrNnHe8y8wqqE3vLujrJT6Y2bQUV7Ic8ziC8YylFyP8RYrTxaPxTpET7iUcN9ehuDbrk54TQoFFvwAjrkzNbvb7aqNtm3bxm9+8xvef/996urqKC0t5Vvf+ha33HILNlvvhWfFihVcd911fPnllxQWFvLd736XG2+8cTimZIoyi0IcQXVgaIaLAkx1WBlrS+4OYMRpbk7ecJkxA9ati71taalRifTAA+YNF1XtvtqbuNx/8YWhHzMUDvoWfPwYMZeXTx/bo40Xu3scJdNvwdf6NYEuQ3zMljEeV+4+KKpRLpxdeiIZhYdQt/b3EEEuPhYbwvksDkUSDDOJBEtAkN0wOGFdVWBTY2JfSiEE987I5IxiG99f18VGb+/2DsXoQxS5N3Z//rTNxw/HuXBG0ew/dqrKPxeG2dIkB5UuqwKynXDz0VZueTWEpsfXiZHAT14M8skNDtZbW3lf1u98vIeeV/KovoXpShbFInK+xnY8PBHeyiZ683/KcHKuOpYq6eV5WbtzvZfdi7YHjfu0jVhRdjYSvKbcBd7neaRpNl95jT5GJdYuzs1bzTl5q7AICSj9ewXd+CD85TrY+NWgeYkDTyHvWz9Fetewo/1Non0KH2ROxqPYWD5rJstnzmDsjiqyOjtozcmlprSUnLCFZS122nEw1CTVLeEQH/u6ONKVoDZUH4Ie+PIf4GvdQw0XAB1qB39ku4RhMV7WrVuHruvcf//9TJw4kVWrVnHllVfi8Xi4807DZOvo6OCYY47hqKOO4r777mPlypVcfvnl5OTk8J3vfGc4phWTsJQs9SbXMC0RdOAno7KGX1wuVVRXJ2e4CAFnngm//nX0MYoCV11lKOfecEPs/U2ZAhUVxv/3288wjM49N/Y2RUUwNgUCcivfJe4ytn0l+LvAkbiw2e6CojpwFyzAXRDdr60FWxM2XAC+DCaWM+BSYKc9IcHdqlJYaUfVBn9vJJJt1hDHLfGyrDOMXRGcVmTj+jFOJrmNS1xdQGdJewhVwIIcKzlWw0NwdIGdNQfZ2erVaAjqfNke4gfrPabn2alJPm0NcXRB5JCCzSJ4+EI7P3w+yOLtOkp3LyZNwtg8wT1n2RhfoDC7TOGppRpPLgkTjPN184fh5ZUaO+bUxizLFhgdjc9Txw16bqXeyt3aegaWHdTg48/aOixxFvuHtc3sTQ6KYuTtHV+YyfyM1/HrCmGp4FZCA3LidXKcU3v/tFjhJw9AQxXyjYcItm7Dl2en9aB5uAtnky9AJ0S0712nYmebPb838V4IKsf0N46/6nLiwUYqqmsU4A1vZ9LGi78dHjkM6lfswYYLgABranKXh8ywGC/HHXccxx133M6/x48fz/r167n33nt3Gi9PPPEEwWCQhx56CJvNxowZM1i2bBl//vOfd4nx8n6Hn3Y99RE0BaN6KAzYheD20Tkck72bfPpmSDbUIiV861tGUu4fI2R3qaphkNxwA1x/fa/eSyRUFQ47zBCk67v/3/7WaNYYzQPzu98NvRkkwNqPzI3buAhmHTX04+3GyCRCRpoUPOLbJ6Ft1h2Ux8cNIX7ycgi7V8ESihxqkkhqywM8RRi1pSdBVvJAlZ9/Vft5fFYmz9YF+W99r+idXYEryhzcMTkDR7fHpMKlUuFS+dM2rymdlr7441w38t2CRy6ys65eZ+FWDU2HOaONJos9NzFjchUu21+YKos2WgDo1MuuuN2Kt8j+VVWb9E6e17ezRkbO4ep5JeE4xnoXYR7Wt3CFMhGA4swFtPnW4VA0Bnd2EjitxWQ5Bpf2d2QG2HKUE11O7H6kjtb2Oqra30VVogsENlvcMb/bIU2hM5gawwWM97JNTz5s/t7/Qf3KPdxw6WZyavQ4h8yI6by0t7eT12ch/PzzzznkkEP6hZGOPfZY1q9fT2tr9AqIQCBAR0dHv59UsCUQTjKVKzpZiuDHJVl8pyiTO8pzWD5zFOfmJ955d5cyejQccIDhJTGLqsJJJxlaKnfcYVQTlZb2Pm+3w6WXwqefQlYWfPxxbBE8TYOPBhgQQhg9jCZO7P0beud5001w2WXm5xwLs2lh34QrUwwCXVtoqXw64e0+DI5ng2au468A9suyMMqhUr1Jxd1uiWq4AHTmh+koMs6dvktLWBqhn3OWd/YzXAACOty3w8+Zy9rRB3y2XVqUEucYzIhQdTQQTZc0dBo5JtkOwdhcZZD31ax2jZSgOkIETczU0ucSv0pv43faatZGMVwS5RPZSKs0NFRctmImFJyzMzQkdjYxAbetlIkF5w5+vYEaNjX/B11G8njraHr0hGo1jnHlD6dGz6T3eDAmSsKuT9d51dPBve3NPN7ZSm24/+sJdMKyh2J22NgjECpkFMPsC3f1TAxGJGN006ZN3HPPPTu9LgB1dXVU9IQBuikuLt75XG5uZCXA3/3ud/zqV79K+RwzFJFyaf8OXXJSjpMJIy0sl2p++1s48sheWf6B9Dzek3k4Z47RP6jnueuvNxJmV60y5PunTDGqf3qwmDgN1QimZXk5rFhh9Dx65hmjgeT06UbbgL1TmH8y+QDYvDj+uAn7pe6YuxlB7w6aNj+QlIH2jG8mKrqpRF0J3DjeqGRp8fapHkHiy9JoLwoRcOkIHTJaLXTlhBGyu7okwr4gcodnHXi7OcRbTUGOL+zNxZiZYeGDllDckm0wclaOzLMyPk7LgsWVGje+FKS+0/CaSAnizRBHT1G4dH8L00oU7BZBUabAphI3bIRFo+aAtaauV3sL4zqqSck/tU3oJN60MRaf602coBrK0lmO8cwq/R6t3nX4QnUIYSHbMQm3rSximHx76+tJH3dUsB2bHiYYRWxEpPRVGufQ6e4sNocCbAkFcQqFuXYnH/q7uL21Aa+UWDDOq7vamzjDncVPcoqwCkHzeggPoV/RLkUY/YukBhklcNHbYNtNIuMJGS833XQTd9xxR8wxa9euZerU3thmdXU1xx13HGeffTZXDrVHDnDzzTfzwx/+cOffHR0dlJcPIRGwm2OynfxfVWyF3WRQ95Tcllgcdhi8+CJcfjk0NfUmzKoqnHqq8buyEgoLDQ2V004zWgL0RVWj9xI6/ngjJBQrbHT88ZGfcziM8NS3vpXkizPB3ifAO/+AcIxcj1GTwZ0zfHPYxXTUvNFtuCS+KNTqWaYrjH470YVXkxz5ZRsr7WE80wWZzSpBh6SzKNyvorrVERrSzbUKPFTt72e8XDHawV2V8cU4VAGFVoW/T4+dA7G6VufbTwV3yv33RJikhLfW6by1LkimHS6Ya+HaQyycMkvl+eUasSJRs4+txa/GDy+5sXCAYni8VshW2kltTp8Adsj+3hFFWMl3zwJmxdw2GG7HFzYh/hgFCzr7eLezyD0+4jnQGUitDMWRDjd/am9iZbDXCrEDfbV7+34iz3s6EAhuzi0i4W4cAlQb6KFd58wVVljwQ2jfBqoVJhwH08+CKLqXu4SEjJcf/ehHXHrppTHHjB/fG9esqanh8MMP54ADDuCBBx7oN66kpIT6+v7yzz1/l5REU1UEu
gitextract_yufqghio/
├── .github/
│ └── ISSUE_TEMPLATE/
│ ├── bug_report.yml
│ └── config.yml
├── .gitignore
├── LICENSE
├── README.md
├── deploy/
│ ├── .dockerignore
│ ├── Dockerfile
│ └── entrypoint.sh
├── pyproject.toml
├── src/
│ └── mistral_inference/
│ ├── __init__.py
│ ├── args.py
│ ├── cache.py
│ ├── generate.py
│ ├── lora.py
│ ├── main.py
│ ├── mamba.py
│ ├── model.py
│ ├── moe.py
│ ├── rope.py
│ ├── transformer.py
│ ├── transformer_layers.py
│ └── vision_encoder.py
├── tests/
│ └── test_generate.py
└── tutorials/
├── classifier.ipynb
└── getting_started.ipynb
SYMBOL INDEX (125 symbols across 13 files)
FILE: src/mistral_inference/args.py
class VisionEncoderArgs (line 13) | class VisionEncoderArgs:
class TransformerArgs (line 30) | class TransformerArgs(Serializable):
method __post_init__ (line 54) | def __post_init__(self) -> None:
class MambaArgs (line 63) | class MambaArgs(Serializable):
method __post_init__ (line 75) | def __post_init__(self) -> None:
FILE: src/mistral_inference/cache.py
function get_cache_sizes (line 13) | def get_cache_sizes(n_layers: int, max_seq_len: int, sliding_window: Opt...
class CacheInputMetadata (line 28) | class CacheInputMetadata:
function interleave_list (line 54) | def interleave_list(l1: List[torch.Tensor], l2: List[torch.Tensor]) -> L...
function unrotate (line 59) | def unrotate(cache: torch.Tensor, seqlen: int) -> torch.Tensor:
class CacheView (line 70) | class CacheView:
method __init__ (line 71) | def __init__(
method update (line 83) | def update(self, xk: torch.Tensor, xv: torch.Tensor) -> None:
method interleave_kv (line 94) | def interleave_kv(self, xk: torch.Tensor, xv: torch.Tensor) -> Tuple[t...
method max_seq_len (line 120) | def max_seq_len(self) -> int:
method key (line 124) | def key(self) -> torch.Tensor:
method value (line 128) | def value(self) -> torch.Tensor:
method prefill (line 132) | def prefill(self) -> bool:
method mask (line 136) | def mask(self) -> AttentionBias:
class BufferCache (line 140) | class BufferCache:
method __init__ (line 146) | def __init__(
method get_view (line 172) | def get_view(self, layer_id: int, metadata: CacheInputMetadata) -> Cac...
method reset (line 176) | def reset(self) -> None:
method init_kvseqlens (line 179) | def init_kvseqlens(self, batch_size: int) -> None:
method device (line 183) | def device(self) -> torch.device:
method to (line 186) | def to(self, device: torch.device, dtype: torch.dtype) -> "BufferCache":
method update_seqlens (line 193) | def update_seqlens(self, seqlens: List[int]) -> None:
method get_input_metadata (line 197) | def get_input_metadata(self, seqlens: List[int]) -> List[CacheInputMet...
method _get_input_metadata_layer (line 225) | def _get_input_metadata_layer(self, cache_size: int, seqlens: List[int...
FILE: src/mistral_inference/generate.py
function generate_mamba (line 12) | def generate_mamba(
function generate (line 44) | def generate(
function sample (line 151) | def sample(logits: torch.Tensor, temperature: float, top_p: float) -> to...
function sample_top_p (line 161) | def sample_top_p(probs: torch.Tensor, p: float) -> torch.Tensor:
FILE: src/mistral_inference/lora.py
class LoraArgs (line 13) | class LoraArgs(Serializable):
method __post_init__ (line 17) | def __post_init__(self) -> None:
class LoRALinear (line 22) | class LoRALinear(nn.Module):
method __init__ (line 35) | def __init__(
method forward (line 71) | def forward(self, x: torch.Tensor) -> torch.Tensor:
method _load_from_state_dict (line 76) | def _load_from_state_dict(self, state_dict: Dict[str, Any], prefix: st...
class LoRALoaderMixin (line 92) | class LoRALoaderMixin:
method load_lora (line 93) | def load_lora(self, lora_path: Union[Path, str], scaling: float = 2.0)...
method _load_lora_state_dict (line 103) | def _load_lora_state_dict(self, lora_state_dict: Dict[str, torch.Tenso...
FILE: src/mistral_inference/main.py
function is_torchrun (line 36) | def is_torchrun() -> bool:
function load_tokenizer (line 41) | def load_tokenizer(model_path: Path) -> MistralTokenizer:
function get_model_cls (line 60) | def get_model_cls(model_path: str) -> Union[Type[Mamba], Type[Transforme...
function pad_and_convert_to_tensor (line 67) | def pad_and_convert_to_tensor(list_of_lists: List[List[int]], pad_id: in...
function _get_multimodal_input (line 77) | def _get_multimodal_input() -> Tuple[UserMessage, bool]:
function interactive (line 102) | def interactive(
function demo (line 203) | def demo(
function mistral_chat (line 268) | def mistral_chat() -> None:
function mistral_demo (line 272) | def mistral_demo() -> None:
FILE: src/mistral_inference/mamba.py
class Mamba (line 23) | class Mamba(ModelBase, nn.Module):
method __init__ (line 24) | def __init__(self, args: MambaArgs):
method dtype (line 46) | def dtype(self) -> torch.dtype:
method device (line 50) | def device(self) -> torch.device:
method forward (line 53) | def forward(
method from_folder (line 64) | def from_folder(
FILE: src/mistral_inference/model.py
class ModelBase (line 11) | class ModelBase(nn.Module, ABC):
method __init__ (line 12) | def __init__(self) -> None:
method dtype (line 17) | def dtype(self) -> torch.dtype:
method device (line 22) | def device(self) -> torch.device:
method forward (line 26) | def forward(
method from_folder (line 36) | def from_folder(
FILE: src/mistral_inference/moe.py
class MoeArgs (line 11) | class MoeArgs(Serializable):
class MoeLayer (line 16) | class MoeLayer(nn.Module):
method __init__ (line 17) | def __init__(self, experts: List[nn.Module], gate: nn.Module, moe_args...
method forward (line 24) | def forward(self, inputs: torch.Tensor) -> torch.Tensor:
FILE: src/mistral_inference/rope.py
function precompute_freqs_cis (line 6) | def precompute_freqs_cis(dim: int, end: int, theta: float) -> torch.Tensor:
function apply_rotary_emb (line 13) | def apply_rotary_emb(
function precompute_freqs_cis_2d (line 26) | def precompute_freqs_cis_2d(
FILE: src/mistral_inference/transformer.py
class SimpleInputMetadata (line 22) | class SimpleInputMetadata:
method from_seqlens (line 27) | def from_seqlens(seqlens: List[int], device: torch.device) -> "SimpleI...
class Transformer (line 33) | class Transformer(ModelBase, LoRALoaderMixin):
method __init__ (line 34) | def __init__(
method dtype (line 101) | def dtype(self) -> torch.dtype:
method device (line 105) | def device(self) -> torch.device:
method freqs_cis (line 109) | def freqs_cis(self) -> torch.Tensor:
method embed_vision_language_features (line 122) | def embed_vision_language_features(self, input_ids: torch.Tensor, imag...
method forward_partial (line 163) | def forward_partial(
method forward (line 221) | def forward(
method load_state_dict (line 244) | def load_state_dict(self, state_dict: Mapping[str, Any], strict: bool ...
method from_folder (line 298) | def from_folder(
FILE: src/mistral_inference/transformer_layers.py
function repeat_kv (line 16) | def repeat_kv(keys: torch.Tensor, values: torch.Tensor, repeats: int, di...
function maybe_lora (line 22) | def maybe_lora(
class Attention (line 31) | class Attention(nn.Module):
method __init__ (line 32) | def __init__(
method forward (line 56) | def forward(
class FeedForward (line 96) | class FeedForward(nn.Module):
method __init__ (line 97) | def __init__(self, dim: int, hidden_dim: int, lora: Optional[LoraArgs]...
method forward (line 105) | def forward(self, x: torch.Tensor) -> torch.Tensor:
class RMSNorm (line 109) | class RMSNorm(torch.nn.Module):
method __init__ (line 110) | def __init__(self, dim: int, eps: float = 1e-6):
method _norm (line 115) | def _norm(self, x: torch.Tensor) -> torch.Tensor:
method forward (line 118) | def forward(self, x: torch.Tensor) -> torch.Tensor:
class TransformerBlock (line 123) | class TransformerBlock(nn.Module):
method __init__ (line 124) | def __init__(
method forward (line 158) | def forward(
FILE: src/mistral_inference/vision_encoder.py
function position_meshgrid (line 12) | def position_meshgrid(
class VisionTransformer (line 31) | class VisionTransformer(nn.Module):
method __init__ (line 32) | def __init__(self, args: VisionEncoderArgs):
method max_patches_per_side (line 50) | def max_patches_per_side(self) -> int:
method device (line 54) | def device(self) -> torch.device:
method freqs_cis (line 58) | def freqs_cis(self) -> torch.Tensor:
method forward (line 72) | def forward(
class VisionLanguageAdapter (line 105) | class VisionLanguageAdapter(nn.Module):
method __init__ (line 106) | def __init__(self, in_dim: int, out_dim: int, bias: bool = True):
method forward (line 116) | def forward(self, x: torch.Tensor) -> torch.Tensor:
class VisionTransformerBlocks (line 120) | class VisionTransformerBlocks(nn.Module):
method __init__ (line 121) | def __init__(self, args: VisionEncoderArgs):
method forward (line 136) | def forward(
class PatchMerger (line 147) | class PatchMerger(nn.Module):
method __init__ (line 152) | def __init__(
method forward (line 166) | def forward(self, x: torch.Tensor, image_sizes: list[tuple[int, int]])...
method permute (line 180) | def permute(
function get_sub_grids (line 206) | def get_sub_grids(
FILE: tests/test_generate.py
class DebugTokenizer (line 12) | class DebugTokenizer:
method bos_id (line 14) | def bos_id(self) -> int:
method eos_id (line 18) | def eos_id(self) -> int:
method pad_id (line 22) | def pad_id(self) -> int:
method encode (line 25) | def encode(self, s: str, bos: bool = True) -> List[int]:
method decode (line 32) | def decode(self, t: List[int]) -> str:
function test_generation_transformer (line 36) | def test_generation_transformer() -> None:
function test_generation_pixtral (line 72) | def test_generation_pixtral() -> None:
function test_generation_pixtral_patch_merger (line 121) | def test_generation_pixtral_patch_merger() -> None:
function test_generation_mamba (line 174) | def test_generation_mamba() -> None:
function test_chunks_transformer (line 199) | def test_chunks_transformer() -> None:
Condensed preview — 25 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (245K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/bug_report.yml",
"chars": 1738,
"preview": "name: Bug report related to mistral-inference\ndescription: Submit a bug report that's related to mistral-inference\ntitle"
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"chars": 271,
"preview": "blank_issues_enabled: false\ncontact_links:\n - name: Documentation\n url: https://docs.mistral.ai\n about: Developer"
},
{
"path": ".gitignore",
"chars": 3078,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packagi"
},
{
"path": "LICENSE",
"chars": 11357,
"preview": " Apache License\n Version 2.0, January 2004\n "
},
{
"path": "README.md",
"chars": 17119,
"preview": "# Mistral Inference\n<a target=\"_blank\" href=\"https://colab.research.google.com/github/mistralai/mistral-inference/blob/m"
},
{
"path": "deploy/.dockerignore",
"chars": 17,
"preview": "*\n!entrypoint.sh\n"
},
{
"path": "deploy/Dockerfile",
"chars": 1061,
"preview": "FROM --platform=amd64 nvcr.io/nvidia/cuda:12.1.0-devel-ubuntu22.04 as base\n\nWORKDIR /workspace\n\nRUN apt update && \\\n "
},
{
"path": "deploy/entrypoint.sh",
"chars": 393,
"preview": "#!/bin/bash\n\nif [[ ! -z \"${HF_TOKEN}\" ]]; then\n echo \"The HF_TOKEN environment variable is set, logging to Hugging Fa"
},
{
"path": "pyproject.toml",
"chars": 1159,
"preview": "[tool.poetry]\nname = \"mistral_inference\"\nversion = \"1.6.0\"\ndescription = \"\"\nauthors = [\"bam4d <bam4d@mistral.ai>\"]\nreadm"
},
{
"path": "src/mistral_inference/__init__.py",
"chars": 22,
"preview": "__version__ = \"1.6.0\"\n"
},
{
"path": "src/mistral_inference/args.py",
"chars": 2131,
"preview": "from dataclasses import dataclass\nfrom typing import List, Optional\n\nfrom simple_parsing.helpers import Serializable\n\nfr"
},
{
"path": "src/mistral_inference/cache.py",
"chars": 10285,
"preview": "from dataclasses import dataclass\nfrom typing import List, Optional, Tuple\n\nimport torch\nfrom xformers.ops.fmha.attn_bia"
},
{
"path": "src/mistral_inference/generate.py",
"chars": 5676,
"preview": "from typing import List, Optional, Tuple\n\nimport numpy as np\nimport torch\n\nfrom mistral_inference.cache import BufferCac"
},
{
"path": "src/mistral_inference/lora.py",
"chars": 5842,
"preview": "import logging\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any, Dict, NamedTuple, Unio"
},
{
"path": "src/mistral_inference/main.py",
"chars": 9365,
"preview": "import json\nimport logging\nimport os\nimport warnings\nfrom pathlib import Path\nfrom typing import List, Optional, Tuple, "
},
{
"path": "src/mistral_inference/mamba.py",
"chars": 2662,
"preview": "import json\nfrom pathlib import Path\nfrom typing import List, Optional, Union\n\nimport safetensors\nimport torch\nimport to"
},
{
"path": "src/mistral_inference/model.py",
"chars": 990,
"preview": "from abc import ABC, abstractmethod\nfrom pathlib import Path\nfrom typing import List, Optional, Union\n\nimport torch\nimpo"
},
{
"path": "src/mistral_inference/moe.py",
"chars": 1095,
"preview": "import dataclasses\nfrom typing import List\n\nimport torch\nimport torch.nn.functional as F\nfrom simple_parsing.helpers imp"
},
{
"path": "src/mistral_inference/rope.py",
"chars": 1646,
"preview": "from typing import Tuple\n\nimport torch\n\n\ndef precompute_freqs_cis(dim: int, end: int, theta: float) -> torch.Tensor:\n "
},
{
"path": "src/mistral_inference/transformer.py",
"chars": 13923,
"preview": "import json\nimport logging\nimport math\nfrom dataclasses import dataclass\nfrom pathlib import Path\nfrom typing import Any"
},
{
"path": "src/mistral_inference/transformer_layers.py",
"chars": 5752,
"preview": "from functools import partial\nfrom typing import Optional, Tuple, Type, Union\n\nimport torch\nfrom torch import nn\nfrom xf"
},
{
"path": "src/mistral_inference/vision_encoder.py",
"chars": 7869,
"preview": "from typing import List, Optional\n\nimport torch\nimport torch.nn as nn\nfrom xformers.ops.fmha.attn_bias import BlockDiago"
},
{
"path": "tests/test_generate.py",
"chars": 7167,
"preview": "from typing import List\n\nimport numpy as np\nimport torch\nfrom mistral_inference.args import VisionEncoderArgs\nfrom mistr"
},
{
"path": "tutorials/classifier.ipynb",
"chars": 119755,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"id\": \"0f47fc4b-7cdc-48ce-b42c-e0b73d902ece\",\n \"metadata\": {},\n \"so"
},
{
"path": "tutorials/getting_started.ipynb",
"chars": 7194,
"preview": "{\n \"cells\": [\n {\n \"cell_type\": \"markdown\",\n \"metadata\": {},\n \"source\": [\n \"# Getting Started with `mistral-inf"
}
]
About this extraction
This page contains the full source code of the mistralai/mistral-inference GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 25 files (232.0 KB), approximately 104.8k tokens, and a symbol index with 125 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.